For solving stability of motion in a nonlinear relative rotation shafting, a sort of periodic variable coefficient linear ordinary differential equations was studied. Movement stabilization criterion of nonlinear relative rotation system was obtained by transferring the fundamental matrix of the solution of the ordinary differential equation to Jordan canonical. Using the Lyapunov function method, the stabile balance position of relative rotation motion of driving system in a variety of mechanical equipments was studied, and the analytic expression of the stable region was obtained. The method supplies theoretical laws, similarity analog criterion and technique means of selecting working parameter of stable working interval of a type of mechanical driving systems. The result of the research can be used for analysis and evaluation of the torsional vibration stability of the main driving system of heavy and complex rotary machines.