In the relativistic klystron driven by intense relativistic electron beam, due to the influences of intense current and high electric field, especially the high Q value for the intermediate cavity, the nonlinear interaction between the intermediate cavity and the electron beam is very strong. It will significantly affect the performance of the device. According to the Maxwell equations and one-dimensional motion equation of electron, the self-consistent equation of the beam-wave interaction is obtained in the intermediate cavity. Based on these equations, the influences of the modulation depth and the modulation frequency on the amplitude and phase of gap voltage are studied respectively. The voltage amplitude obtained by the self-consistent equation is close to the voltage amplitude of particle in cell simulation, especially under the higher modulation depth compared with that obtained from the equivalent circuit model of conventional klystron. Meanwhile, the bandwidth of device becomes wide with the increase of the modulation depth. Finally, an S-band high-gain RKA is designed, and the corresponding experiments are done on the LTD accelerator. The output power with 1.1 GW is obtained, the gain is 49 dB.