To study the application of memristor in chaotic system, we employ the smooth continuous nonlinear flux-controlled memristor model and feedback control technique to design a hyperchaotic system based on the simplified Lorenz system. By using memristor as a positive feedback of the simplified Lorenz system, the dimensionless mathematical model is derived. The differences between the memristor-based chaotic system and ordinary chaotic system are then further studied. Firstly, the stable equilibrium and unstable equilibrium point sets of the system are analyzed theoretically, and it is found that the system has infinite equilibrium points including stable and unstable equilibrium points. The stable and unstable ranges of the system with different parameters are also determined. Theoretical analysis shows that the system has the same symmetry as the simplified Lorenz system. Thus the system has rich dynamical behaviors, such as limit cycle, chaotic attractor, and hyper-chaotic attractor. Secondly, by the methods of bifurcation diagram, Lyapunov exponent spectrum, Poincar section, and Spectral Entropy algorithm, the dynamical behaviors of the system are analyzed in detail. By calculating the Lyapunov exponent spectrum, the dynamical behaviors are studied and they change with system parameters and the initial conditions of memristor respectively. The maximum positive Lyapunov exponent of the memristor-based Lorenz hyperchaotic system is higher than that of the simplified Lorenz system, which indicates the memristor-based Lorenz hyperchaotic system is more complex. Further, we find all the complex dynamical behaviors to be coexisting with the infinite equilibrium sets, which is quite different from those of many ordinary hyper-chaotic systems. Meanwhile, we observe the attractors coexisting and state transition phenomenon in this system, caused by changing the initial conditions of the memristor. State transition phenomenon is then further studied by means of phase portraits and spectral entropy algorithm for the first time. Finally, by using operational amplifiers, diodes and other discrete components, we design an equivalent circuit of the smooth continuous nonlinear flux-controlled memristor model, and the equivalent circuit is used to design and realize the analog electronic circuit of the memristor-based Lorenz hyper-chaotic system. By using an analog oscilloscope, the phase portraits of hyper-chaotic attractor are observed clearly. The state transition phenomenon can also be seen using the oscilloscope. It is found that the circuit experimental results are in agreement with those of the theoretical analysis and numerical simulation. It verifies that the system is physically realizable, and lays a strong foundation for its applications in engineering. Next, we will try to investigate the chaotic secure communication based on this hyper-chaotic system.