In the marine environment, when the line spectra of underwater target radiated signal are unknown or the continuous spectra are weak, it is extremely hard to accurately detect the underwater weak target. The line spectrum based method commonly requires spectrum information for detection, and the continuous spectrum based energy method is hard to achieve accurate detection in long distance. In this paper, an underwater radiated noise detection method based on generalized Duffing oscillator detection system is proposed. Firstly, a generalized Duffing oscillator detection system for non-periodic and non-stationary input signal is proposed through deducing the traditional Duffing oscillator detection system that is perturbed by periodic signal. And the proposed generalized Duffing oscillator detection system is able to detect the signals of targets without needing prior information. Secondly, when the target radiated signal (non-periodic and non-stationary signal) is input into the generalized Duffing oscillator, a special form of output phase space (a special state of motion) is discovered and the differences in output phase space among different input signals (periodic stationary signals, nonperiodic non-stationary signals and the target radiated signals) are analyzed. It is found that the special phase space has different form from the output phase spaces of other kinds of signals; accordingly the underwater targets can be detected through the representation of the difference between special phase space and ordinary phase space. Thirdly, a discrete distribution sequence calculation method based on phase space is proposed for the precise and efficient judgment of system motion. The proposed calculation method defines a similar-grid function, based on which, the distribution sequence calculation method of output phase space is deduced, and the distribution sequences of different kinds of output phase spaces are calculated. The method realizes an embedded expression of system output by using the statistical complexity, therefore achieving the embedded underwater target detection when the line spectra of underwater target radiated signal are unknown or the continuous spectra are weak. The analysis result indicates that the method is of low-computation. Finally, the experimental results in the sea are described and the lowest signalto-noise ratio (SNR) of the method is calculated to be -9.133 dB. Simulation and experimental results have shown that the proposed method can detect target successfully in a lower SNR than traditional detection method, and the real-time performance can meet the demand for underwater detection. The method in this paper provides new ideas and ways of thinking for underwater target detection, and has very important reference value for low SNR long-distance target detection under real condition.