Chrysanthemum morifolium Ramat(hereinafter referred to as Chrysanthemum) is one of the most beloved and economically valuable Chinese herbal crops, which contains abundant medicinal ingredients and wide application prospects. Therefore, identifying the classification and origin of Chrysanthemum is important for producers, consumers, and market regulators. The existing Chrysanthemum classification methods mostly rely on visual subjective identification, are time-consuming, and always need high equipment costs. A novel method is proposed to accurately identify the Chrysanthemum classification in a swift, non-invasive, and non-contact way. The proposed method is based on the fusion of deep visual features of both the front and back sides. Firstly, the different Chrysanthemums images are collected and labeled with origins and classifications. Secondly, the background area with less available information is removed by image preprocessing. Thirdly, a two-stream feature extraction network is designed with two inputs which are the preprocessed front and back Chrysanthemum images. Meanwhile, the incorporation of single-path residual connections and cross-path residual connections is employed to extend the receptive field of the network and fully fusion the features from both the front and back sides. Experimental results demonstrate the proposed method outperforms existing methods, which gain higher accuracy and stability. The proposed method presents an effective and dependable solution for identifying the classification and origin of Chrysanthemum. Code and data are available at https://github.com/dart-into/CCMIFB.