This paper focus on developing an optimal controller for the strict-feedback nonlinear systems with or without asymmetric time-varying full state constraints. A novel nonlinear state-dependent transformation function is presented, by which the strict-feedback nonlinear systems with state constraints is transformed into a new strict-feedback where the state constraints is implicit in. Optimized backstepping technique is utilized to develop the optimal controller for the new strict-feedback system to track the desired reference signal without the feasibility conditions. Reinforcement learning (RL) is exploited to implement the optimal control in every step, where identifier, critic and action network are used to estimate the unknown system dynamics and generate the control output, respectively. It is theoretically proved that all the signals in the close loop system are bounded and the proposed optimal controller can track the desired signal with or without time-varying asymmetric full state constraints. Two simulation examples are presented demonstrating the efficacy of the proposed scheme.