The lives of millions of people in the East African region are highly affected by severe floods and persistent droughts. Early warnings at least a couple of seasons ahead would help the mitigation measures. However, most prediction systems using dynamical models are seen to perform poorly at long lead times. In this study, we propose a statistical deep-learning based approach using convolutional neural network (CNN) to predict extreme floods and droughts during the short rains season (October-December). The proposed CNN accurately captures the extreme floods and droughts two to three seasons ahead. Diagnosing the model’s skills using heatmaps, the extreme floods and droughts are found to linked with Indian Ocean Dipole at shorter leads and with western and southern Indian Ocean sea surface temperature anomalies, at longer leads. Though a few exceptions poorly predicted, the superior skills of the CNN-based predictions on longer leads are advantageous in organizing mitigation measures.