The Lipkin-Meshkov-Glick (LMG) model originally describes a Fermionic many-body system in nuclear physics. However, in recent years, it has been widely found in condensed matter physics, quantum information systems, and quantum optics, and it is of wider and wider interest. Previous studies on this model mainly focused on the physics under the thermal dynamical limit, such as quantum phase transitions and quantum entanglement. There are also some researches about LMG model with finite size in some special limits, but the finite-size effect on energy spectrum is not very clear yet. This is the main motivation of this work. In this paper, the exact diagonalization method and the quantum perturbation theory are used to calculate and analyze the energy-level structure of the LMG model at a finite N. To solve it, we first map this model into the angular-momentum space to obtain a reduced LMG model. By this mapping, the dimension of Hilbert space is reduced to N+1 from 2N. The exact solution of its energy levels can be obtained easily in the U(1) limit where the total spin is conserved. We find that the levels are woven into a fishing-net structure in the U(1) limit. While away from the U(1) limit, the crossings between even and odd levels will open a gap, and the system's energy levels will be grouped into pairs with an odd and an even level, forming some bound states, called doublet states, and the parity of each doublet state will oscillate as the Zeeman field increases. This work gives the values of the critical Zeeman field for the parity crossings. These critical values shift as the interacting parameters and disappear at zero in the Z2 limit. In the Z2 limit, the system energy levels form splittings near the zero Zeeman field. In this article, we analytically calculate the relationship between these energy gaps and the Zeeman field. For odd and even number N, the parity of each state has a different behavior. Specifically, the ground state and the doublet excited states of the system with odd N will suffer a parity reversion at zero Zeeman field, while the states with even N will not. By tuning the interacting parameters, we also study the crossover from the U(1) limit to the Z2 limit. The parity oscillation we find in this system is a very important physical phenomenon, which also exists in some other systems like optical cavity quantum electrodynamics and magnetic molecule system.