Even though physical activity and sedentary behavior are moderately heritable, little is known about the mechanisms that influence these traits. Here, we combine data for up to 674,980 individuals from 51 studies in a trans-ancestry meta-analysis of genome-wide association studies for self-reported moderate-to-vigorous intensity physical activity during leisure time (MVPA); leisure screen time (LST); sedentary commuting; and sedentary behavior at work. We identify 99 loci that associate with at least one trait. Loci associated with LST are enriched for genes whose expression in skeletal muscle is altered by resistance training. Molecular dynamics simulations suggest that the Glu to Ala substitution encoded by rs2229456 (ACTN3) – associated with more MVPA – disrupts salt bridge interactions and makes the alpha actinin 3 filaments more flexible. In isolated type IIA muscle fibers, the Ala-encoding allele is associated with lower maximal force and power during an isometric contraction, suggesting protection from exercise-induced muscle damage. Finally, Mendelian Randomization analyses show that the causal effect of LST on BMI is 2-3 times larger than the effect of body mass index (BMI) on LST, and that beneficial effects of LST and MVPA on several risk factors and diseases are mediated or confounded by BMI. Taken together, our results provide mechanistic insights into the regulation of MVPA and into the role of LST and MVPA in disease prevention. These insights may facilitate the development of tailored physical activity interventions.