The research addresses the issue of image variability in practical settings by introducing a deep learning-based system for allergy prediction and Indian dish recognition. Food plays a big part in supporting a healthy lifestyle, which is becoming more and more popular globally. Recognizing the different types of food and the allergies that are in it is crucial. With the help of convolutional neural networks and visual transformers, the model can predict possible allergies and identify foods with accuracy, which is important for managing diet and raising awareness of allergens. Its uses are extensive in the food service, medical, and nutrition domains, providing workable answers for safer and better-informed nutritional selections in Indian food. Extensive analysis validates the model's effectiveness and potential influence on many sectors. In this study, various CNN architectures, including ResNet50, VGG16, VGG19, and a custom CNN, were employed for Indian Dish Recognition and Allergy Prediction. To enhance model accuracy, techniques such as image augmentation were integrated. While certain models showed promising results, reaching a maximum accuracy of 81%, others did not meet the desired expectations. To improve performance, Vision Transformers (ViTs) were employed, resulting in a remarkable accuracy of 92%.