The 3D-QSAR models were established in this study based on comparative molecular field analysis (CoMFA) and comparative molecular similarity index analysis (CoMSIA), the optimal CoMFA model established gave \({\text{Q}}^{2}\)= 0.671, \({\text{R}}^{2}\)= 0.925 and \({\text{R}}_{\text{p}\text{r}\text{e}\text{d} }^{2}\)= 0.868, and the best CoMSIA/SEA model gave \({\text{Q}}^{2}\)= 0.627, \({\text{R}}^{2}\)= 0.775, and \({\text{R}}_{\text{p}\text{r}\text{e}\text{d} }^{2}\)= 0.962. The predictive ability of the developed models was evaluated by external and internal validation. In this study, steric, electrostatic, and hydrogen bond acceptor fields played a key role in the anti-cancer activity. Molecular docking results theoretically revealed the importance of residues ARG164 and THR45 in the active site of the TrxR enzyme. Based on these results, we designed several new inhibitors, and their inhibitory activities were predicted by the best model (CoMFA). In addition, these new inhibitors were analyzed for their ADMET properties and their similarity to drugs. These results will be of great help for the optimization of new anti-cancer drug discovery.