The nonlinear dynamic equation with two masses in relative rotation is established, which contains a kind of generalized nonlinear elastic force. Using variable gradient method, the Lyapunov function is gotten, and the stability of the relative rotation nonlinear autonomous system is studied. The approximate solution of the associated harmonic response of the relative rotation nonlinear non-autonomous system under two forcing excitations of different frequencies is obtained by the method of perturbation.