Brain tumors are lumps of aberrant tissue that can develop into cancer and have a significant negative influence on a person's health. MRI scans of the brain can reveal them. Segmentation and classification are two elements in these approaches that are extremely crucial. As opposed to anatomical organ segmentation, tumor segmentation is much more difficult due to the variety in size, location, and shape of tumors. For this reason, it is imperative to build reliable, precise, and effective deep learning-based methods. Recent deep learning techniques for classifying and segmenting brain tumors produced encouraging results. These approaches, however, have heavy-weight architectures by nature, necessitating more storage and costly training procedures because of the enormous number of training parameters they must be fed. It is crucial to investigate transportable deep learning models without compromising classification precision. In this research, we provide compact deep neural network models using the pre-trained Attentiveness MobileNetV2 models along with the attention module. The four phases of the proposed system are preliminary processing, division, extracting and categorizing features, and severity classification. Anisotropic diffusion processing as well as data enhancement methods are used initially. The tumor region is then segmented using the proposed modified dimensional U-Net (3D-M-U-Net). Finally, the extraction and classification of features are implemented using the Compact MobileNetV2 framework. Here, the high-level tumor-based information is initially recovered from the convolution features. The important semantic information is then captured using an attention module. Once high-level tumor-based data as well as fascinating semantic information have been combined in the convolutional and focused modules, fully linked layers as well as the layer of softmax are utilized to categorize tumours into either benign or dangerous. Finally, Support Vector Machine (SVM) is used to categorize tumors into moderate, severe, and mild phases. The suggested approach was tested on the high-quality brain cancer images available in the Brats-2020 as well as Brats-2019 datasets. In regards to precision, recall, accuracy, F-Score, Dice Similarities Coefficient (DSC), as well as Structural Similarity Indicator Matrix (SSIM), the suggested model outperforms existing traditional and hybrid models. It was also the most effective and productive method tested. The suggested model has a 99.9% accuracy, a 99.9% precision, and a 99.8% recall across both datasets.