The global food system is an important contributor to greenhouse gas emissions that lead to climate change. Animal agriculture is responsible for a large share of the food-system emissions, both directly and through the production of animal feed. Limiting global warming to the goals set forth by the international community will not be possible without rapid phasing out of a substantial share of animal-source food. We show that the rapid adoption of alternatives to animal-source foods, such as plant-only diet, or plant-based, cultured, or fermentation-derived analogs to animal products, can be consistent with climate goals. Importantly, the longer the delay in the adoption of alternatives, the larger the share of diet they will have to represent in 2050 for the food system to stay within its carbon budget.