The identification of Alzheimer's disease (AD) using structural magnetic resonance imaging (sMRI) has been studied based on the subtle morphological changes in the brain. One of the typical approaches is a deep learning-based patch-level feature representation. For this approach, however, the predetermined patches before learning the diagnostic model can limit classification performance. To mitigate this problem, we propose the BrainBagNet with a position-based gate (PG), which applies position information of brain images represented through the 3D coordinates. Our proposed method represents the patch-level class evidence based on both MR scan and position information for image-level prediction. To validate our proposed method, we conducted experiments comparing our methods with state-of-the-art methods and evaluating the robust performances of two publicly available datasets: 1) ADNI and 2) AIBL. Furthermore, our proposed method outperformed classification performance in both AD diagnosis and mild cognitive impairment conversion prediction tasks over the comparative methods and analyzed our results from diverse perspectives. Based on our experimental results, we believe that the proposed method has the potential to provide new insights and perspectives in deep-learning based patch-level feature representation studies. Code is available at: github.com/ku-milab/PG-BrainBagNet.