A nanoscale memristor can replace the nonlinear part of a chaotic system, which can greatly reduce the physical size of the chaotic system. More importantly, it can enhance the complexity of the chaotic system and the randomness of signals. In this paper, a new memristor-based chaotic system is designed based on a new three-dimensional autonomous chaotic system. In order to study the complex dynamic characteristics of the memristive system, the chaotic system is investigated by the theoretical derivation, numerical simulation, stabilization of equilibrium points, and Lyapunov exponent spectrum. The influences of different parameters on the phase diagram and the stability of equilibrium point of this system are also discussed in detail. It is interesting that when system parameters a and c take different values, the location and stability of the equilibrium point of the system will be changed, then two scrolls of the system will be overturned at a different angle, and it will produce a different degree of aliasing between the two scrolls. Parameter b has a large variable range, when it is changed, and the system will transform into three kinds of classical chaotic systems defined by Vaněček and Celikovsk. These indicate that the memristor-based chaotic system has a lot of valuable dynamic behaviors, so it has applications in the field of secure communication, information processing etc. Field programmable gate array (FPGA) technology has a large capacity and high reliability, which is widely used in modern digital signal processing. And with the development of FPGA technology, applying FPGA technology to realizing the chaotic systems has gradually become a hot topic. Moreover, the improved Newton iteration method is used to design a square root operator of memristor in this paper by using verilog hardware description language (verilog HDL) which only needs three times iteration to reach the required accuracy. The results of FPGA hardware are consistent with the numerical simulation results. It breaks through the previous bottleneck that the chaotic system based on titanium dioxide memristor can only be simulated in computer, which is of great significance for further studing of memristor, and provides a reference for further research on the memristor-based chaotic system and applications in secure communication and information processing.