In order to study the influence of local nonlinear factors such as bearing clearance and Hertzian contact on the nonlinear behavior of the sprag clutch-rotor system during high-speed steady-state engaging, a nonlinear force model of rolling bearing considering radial clearance and Hertzian contact is established. Regarding the inner and outer rotors as substructures, the fixed interface modal synthesis method is used to reduce the dimension of the substructure model, and the nonlinear dimension reduction model of the sprag clutch-flexible rotor system (SC-FRS) is obtained. To improve computational efficiency, the Newmark method combined with the fixed interface synthesis method is presented to calculate the system response. The response results of the reduced model with a different number of dominant modes and the unreduced model are compared, and the number of reserved dominant modes of the inner and outer rotors is selected based on the calculation efficiency and accuracy. The effect of the radial clearance of different bearings on the nonlinear vibration response and the dynamic load of the inter-shaft bearing is discussed. It is found that the sensitivity of different radial clearances to the system's amplitude-frequency response is different, and the amplitude jump and the frequency hysteresis appear at the resonance peak.