The single-pin track system is widely used in light-tracked vehicles due to its simple structure and high reliability. This paper aims to establish a nonlinear three-dimensional tooth engagement search process to simulate the meshing process between the single-pin track shoe and the sprocket, thereby improving the accuracy of multibody system dynamics simulations. Based on the actual geometric structure, the contact surfaces between the sprocket teeth and the track pin are discretized, and a body-fixed coordinate system is introduced for each contact surface to facilitate the geometric contact criteria and contact force calculation. Through simulation, the effect of pre-tension on tooth engagement is analyzed, revealing the dynamics of the track chain as a function of pre-tension variation. Simulation results, validated by commercial software and experimental tests, indicate that the proposed 3D meshing model can effectively simulate the impact of high-frequency excitations on tracked vehicles. Furthermore, the study provides suggestions for optimizing the design of the track shoe and sprocket to improve the service life of the meshing system.