Background The present study was conducted to examine the operation status of the Competency-Based Curriculum (CBC), which has become the main curriculum in the medical schools in Korea, and prepare valid evaluation criteria consented by experts to make improvements. The evaluation criteria were derived based on a model built by combining the Context, Input, Process and Product (CIPP) evaluation model and the Kirkpatrick evaluation model, which are representative educational evaluation models. Methods Firstly, literature survey was performed and a semi-structured interview was conducted with 5 experts to develop a draft of the evaluation criteria. To verify the validity of the draft of the developed evaluation criteria, two surveys based on the modified Delphi methodology were conducted with a panel consisting of 20 experts. Results Based on the literature survey and the expert interview, a draft of the evaluation criteria was derived, including 5 evaluation areas, 18 evaluation items and 58 evaluation indicators. Two Delphi surveys were conducted to validate the evaluation criteria. The evaluation criteria that showed a relatively low content validity ratio (CVR) were corrected and complemented by reflecting the experts’ opinions to finally derive 5 evaluation areas, 16 evaluation items and 51 evaluation indicators. Conclusions The significance of the present study is that an evaluation model and its evaluation criteria suitable for the curriculums of the medical schools in Korea were developed with the consensus of the experts. The preparation of valid evaluation criteria will open up the possibility of improving the evaluation in medical schools, contributing to the improvement of the educational quality and the continued quality improvement of medical education.