In oil and gas drilling, timely and accurate identification of formation lithology is an important guarantee of drilling safety. Aiming at the problems of inaccurate identification of lithology in drilling by traditional methods, and low efficiency due to the fact that even modern instruments cannot respond to lithology in real time. the Categorical Boost (CatBoost) model was applied to lithology identification in this study. However, since CatBoost uses more hyperparameters in its modeling, it is difficult to optimize model prediction by manually tuning the parameters. Therefore, the introduction of Kernel Principal Component Analysis (KPCA) extracts fewer and more important features from the original data, eliminates the redundant information contained therein, and combines with Bayesian Optimization (BO) algorithm to optimize the hyperparameters during the training process, thus improving the prediction performance of CatBoost. Two experiments were designed to verify the recognition ability of the model, and the final test results of the model showed that the KPCA-BO-CatBoost model proposed in this paper had the best overall performance, and the lithology recognition accuracy reached more than 90%. The model was effective in identifying the formation lithology, realized real-time lithology identification by combining the parameters of logging while drilling, improved the efficiency and accuracy of lithology identification, and was of great significance in guiding the subsequent drilling work.