To study the overall vibration characteristics of the sprag clutch-flexible rotor system (SC-FRS) under high-speed operating conditions, a finite element model of SC-FRS considering rotor flexibility and bearing support stiffness is established based on the proposed calculation method of the stiffness matrix. According to this model, the natural frequency and mode shape of the system are calculated, and the correctness of the model is verified by comparing it with the calculation results of ANSYS software. Under the action of unbalance, the bending-torsion coupled vibration and the dynamic load of the inter-shaft bearings are analyzed, and it is found that the resonant peak in the torsional direction has the same resonance frequency as that in the bending direction. A test rig for the sprag clutch-rotor system is built, and the axis trajectory and critical speed are tested. The test results show that the finite element model of SC-FRS can accurately describe the vibration characteristics of the system.