Despite significant progress in existing methods for predicting drug-target binding affinity, there is still room for improvement in better utilizing molecular sequences and designing feature fusion strategies. Addressing these two points, we propose a novel computational model, Secondary Sequence and Cross-attention Block based Drug-Target binding Affinity prediction (SSCBDTA). The model is composed of sequence encoding, feature extraction, modal fusion and a decoder, with three innovations: (i) applying the byte pair encoding algorithm to process vast unlabeled data for obtaining molecular secondary sequences; (ii) extracting features from two perspectives: the primary and secondary sequences of molecules; (iii) combining cross-attention and criss-cross attention to fuse the extracted features of drugs and proteins. In two benchmark datasets, SSCBDTA outperforms ten state-of-the-art models on nearly all evaluation metrics. By conducting four different ablation experiments, we separately validated the effectiveness of molecular secondary sequences and multiple cross-attention in improving the prediction accuracy and stability of SSCBDTA. We also utilized SSCBDTA to predict binding affinities between 3,137 FDA-approved drugs and 6 SARS-CoV-2 replication-related proteins, identifying a number of promising molecules that could be further developed as anti-COVID drugs.