The traditional cylindrical resonant cavity loaded in gyrotron is substituted by the photonic-band-gap resonant cavity （PBGC） which is constructed by triangular arrays of metal posts, and the TE operating modes in PBGC are analyzed and calculated. Under the consideration of the cavity structure features, a research method for the mode theory on PBGC is provided. The single-mode operation in the cavity is found, and the distribution characteristics of the azimuthally inhomogeneous electromagnetic modes are reported. The research results show that the resonant frequency in the cavity can be excited without the limitation of transverse dimension. The study provides a physical theoretical fundamental of which reveals the possibility of changing the operating status of gyrotron essentially. This possibility will play an important role in the following aspects: enhancing the power capacity of cavity, carrying out the beam-wave interaction between high order electromagnetic single-mode and high electronics cyclotron harmonic wave efficiently, and reducing the gyrotron operating magnetic field.