Electronic circular dichroism (ECD) spectra contain key information about molecular chirality by discriminating the absolute configurations of chiral molecules, which is crucial in asymmetric organic synthesis and the drug industry. However, existing predictive approaches lack the consideration of ECD spectra due to the data scarcity, and the interpretability to achieve trust-worthy prediction. Here, we establish a large-scale dataset for Chiral Molecular ECD spectra~(CMCDS) and propose the ECDFormer for accurate and interpretable ECD spectra prediction. ECDFormer decomposes ECD spectra into peak entities, employs the QFormer architecture to learn peak properties, and renders peaks into spectra. Compared to spectra sequence prediction methods, our decoupled peak prediction approach significantly enhances both accuracy and efficiency, improving the peak symbol accuracy from 37.3% to 72.7% and dramatically decreasing the time cost from 2-10 CPU hours to 1.5 seconds. More significantly, ECDFormer demonstrated its ability to capture molecular orbital information directly from spectral data using the explainable peak-decoupling approach, showcasing its potential to uncover hidden correlations between molecular substructures and spectral features. Furthermore, ECDFormer proved to be equally proficient at predicting various types of spectra of complex natural products, highlighting its substantial generalization capabilities.