The vibration signals of a planetary gear set exhibit rich spectra that can reflect its operating states. In previous phenomenological models, the sidebands are mostly explored under the condition of constant speed. However, the vibration signal is not only affected by internal nonlinear factors such as velocity fluctuations, fault excitation and manufacturing errors, but also by transmission path effects that cause the vibration signal to be further modulated, making it difficult to recognize fault characteristics. To solve this problem, this paper characterizes the actual rotational speed fluctuation through Fourier series, deduces the excitation force model by comprehensively considering the manufacturing error and fault excitation, and establishes a vibration signal model of the planetary gear set by considering the influence of the transmission path effect. Simulation results show that manufacturing errors and sun gear faults cause the meshing frequency to be modulated by the sun gear rotational frequency. Moreover, the vibration signal model considering speed fluctuation reflects the fault characteristic frequency, and the harmonics of the fault characteristic frequency are modulated by the planetary carrier rotational frequency. The spectral distribution pattern is consistent with the frequency distribution of the experimental signal, verifying the accuracy of the model.