In this paper, for the phenomenon of self-excited oscillation caused by nonlinear factors in a large aircraft wheel brake control system, the nonlinear dynamic behavior of the pressure servo valve-controlled cylinder system (PSVCS) is studied,and the influence law of the key parameters on the nonlinear self-excitation behavior is obtained. On this basis, the stability of the PSVCS is analyzed both in time domain and frequency domain, and it is proved in principle that the PSVCS is a stable self-closed loop control system. Firstly, the nonlinear dynamics model of the PSVCS is established in this paper. Secondly, using the method of phase plane analysis, the nonlinear dynamic behavior of the PSVCS and the influence law of key parameters on the system are studied. Thirdly, the nonlinear system of the PSVCS is transformed into a segmented local linear system, and the stability of the prestage and the power stage are analyzed and studied respectively. Finally, through a performance test platform, which is used to simulate the load of PSVCS, the theoretical analysis results of this paper are verified experimentally under a variety of working conditions. The final experimental results show that both the nonlinear dynamic model established in this paper and the influence law of the key parameters obtained by the phase plane analysis on the nonlinear self-excited oscillation behavior are correct, and the relevant conclusions can provide a reference for the design of the braking system control system.