Normal brain aging is accompanied by patterns of functional and structural change. Alzheimer's disease (AD), a representative neurodegenerative disease, has been linked to accelerated brain aging at respective age ranges. Here, we developed a deep learning-based brain age prediction model using fluorodeoxyglucose (FDG) PET and structural MRI and tested how the brain age gap relates to degenerative cognitive syndromes including mild cognitive impairment, AD, frontotemporal dementia, and Lewy body dementia. Occlusion analysis, performed to facilitate interpretation of the model, revealed that the model learns an age- and modality-specific pattern of brain aging. The elevated brain age gap in dementia cohorts was highly correlated with the cognitive impairment and AD biomarker. However, regions generating brain age gaps were different for each diagnosis group of which the AD continuum showed similar patterns to normal aging in the CU.