In this study, a 42-degree-of-freedom (42-DOF) translation-torsion coupling dynamics model of the wind turbine’s compound gear transmission system considering time-varying meshing friction, timevarying meshing stiffness, meshing damping, meshing error and backlash is proposed. Considering the different meshing between internal and external teeth of planetary gear, the time-varying meshing stiffness is calculated by using the cantilever beam theory. An improved mesh friction model takes account into the mixing of elastohydrodynamic lubrication (EHL) and boundary lubrication to calculate the time-varying mesh friction. The bifurcation diagram is used to analyze the bifurcation and chaos characteristics of the system under the excitation frequency as bifurcation parameter. Meanwhile, the dynamic characteristics of the gear system are identified from the time domain diagrams, phase diagrams, Poincare maps and amplitude-frequency spectrums of the gear system. The results show that the system has complex bifurcation and chaotic behaviors including periodic, quasi-periodic, chaotic motion. The bifurcation characteristics of the system become complicated and the chaotic region increases considering the effects of friction in the high frequency region.