The evolutionary dynamics of tumor initiation remain undetermined, and the interplay between neoplastic cells and the immune system is hypothesized to be critical in transformation. Colorectal cancer (CRC) presents a unique opportunity to study the transition to malignancy as pre-cancers (adenomas) and early stage cancers are frequently detected and surgically removed. Here, we demonstrate a key role for the immune response in tumor initiation by studying tumor-immune eco-evolutionary dynamics from pre-cancer to carcinoma using a computational model, ecological analysis of digital pathology data, and multi-region exome sequencing and neoantigen prediction in a total of 62 patient samples. Modelling indicates there are several potential routes to malignancy, each of which uniquely sculpts tumor ecology and intra-tumor antigenic heterogeneity (aITH). In patient samples, the immune microenvironment was characterized using the spatial distribution of 17 markers across registered whole-slide images, and patterns of intra-lesion aITH measured using multi-region exome sequencing and neoantigen prediction. The patient data were best described by a model whereby immunogenic adenomas do not progress to CRC because they are under immune control, and progression initially proceeds in adenomas with low immunogenicity followed by the gradual construction of an immunosuppressive niche depleted in CD8+ cytotoxic T cells. There was little evidence for immune blockade (PD-L1 expression) in tumor initiation or progression. These results suggest that re-engineering the immunosuppressive niche may prove to be an effective immunotherapy in CRC.