Identification of the structural connections between neurons is a prerequisite to understanding brain function. We developed a pipeline to systematically map brain-wide monosynaptic inputs to specific neuronal populations using Cre-driver mouse lines and the recombinant rabies tracing system. We first improved the rabies virus tracing strategy to accurately identify starter cells and to efficiently quantify presynaptic inputs. We then mapped brain-wide presynaptic inputs to different excitatory and inhibitory neuron subclasses in the primary visual cortex and seven higher visual areas. Our results reveal quantitative target-, layer- and cell-class-specific differences in the retrograde connectomes, despite similar global input patterns to different neuronal populations in the same anatomical area. The retrograde connectivity we define is consistent with the presence of the ventral and dorsal visual information processing streams and reveals further subnetworks within the dorsal stream. The hierarchical organization of the entire visual cortex can be derived from intracortical feedforward and feedback pathways mediated by upper- and lower-layer input neurons, respectively. This study expands our knowledge of the brain-wide inputs regulating visual areas and demonstrates that our improved rabies virus tracing strategy can be used to scale up the effort in dissecting connectivity of genetically defined cell populations in the whole mouse brain.