Cardiovascular diseases (CVD) stand as prevalent and severe health concerns, significantly impacting individuals. The potential for early diagnosis to prevent or relieve CVDs, thereby reducing mortality rates, underscores its critical role. In this effort, adopting machine learning models to identify risk factors emerges as a promising strategy. Additionally, feature selection methods prove invaluable in identifying crucial attributes, contributing to the reduction of diagnostic expenditures. The analysis in this work was consolidated and improved by using a dataset from Cleveland, Long Beach, VA, Switzerland, Hungarian, and Stat log. In our proposed Method, a hybrid Differential Entropy-based information gain and LASSO algorithm are employed for feature selection. The proposed hybrid model, when combined with machine learning techniques like the Random Forest approach, minimizes data dimensions, improve classification performance, and enhances the efficiency of identifying and training feature sets. Finally, the proposed model produces enhanced performance metrics, encompassing accuracy, precision, and recall.