Colorectal cancer exhibits dynamic cellular and genetic heterogeneity during progression from precursor lesions toward malignancy. Analysis of spatial multi-omic data from 31 human colorectal specimens enabled phylogeographic mapping of tumor evolution that revealed individualized progression trajectories and accompanying microenvironmental and clonal alterations. Phylogeographic mapping ordered genetic events, classified tumors by their evolutionary dynamics, and placed clonal regions along global pseudotemporal progression trajectories encompassing the chromosomal instability (CIN+) and hypermutated (HM) pathways. Integrated single-cell and spatial transcriptomic data revealed recurring epithelial programs and infiltrating immune states along progression pseudotime. We discovered an immune exclusion signature (IEX), consisting of extracellular matrix regulators DDR1, TGFBI, PAK4, and DPEP1, that charts with CIN+ tumor progression, is associated with reduced cytotoxic cell infiltration, and shows prognostic value in independent cohorts. This spatial multi-omic atlas provides insights into colorectal tumor-microenvironment co-evolution, serving as a resource for stratification and targeted treatments.