Navigation ground verification is an essential part of X-ray pulsar navigation (XPNAV) research. Aiming at the need of real and continuous pulsar signals for navigation algorithm verification, and to avoid the difficulties and high costs of X-ray modulation and detection, we propose an XPNAV ground verification system based on visible light source. In this system, the pulsar signal model at the solar system barycenter and the orbit information are used to establish the real-time photon arrival rate function at a spacecraft, and then the rate function is digitized and converted into voltage signal by the designed hardware system to drive a linear light source. After the processes of light attenuation, signal detection and pulse discrimination are experienced, the real-time photon time of arrivals (TOAs) at a spacecraft can be achieved. These photon TOAs contain characteristics of the pulsar profiles and frequency, the time propagation effect in the solar system, and cosmic X-ray background. The system uses semi-physical devices to modulate and attenuate visible light, and judges whether the spacecraft can observe the navigation pulsar according to the real position, thereby realizing the simulation of X-ray propagation in space. At present, the detection method of pulsar observation with single detector include detection of single pulsar, time division detection of multiple pulsars, and simultaneous detection of multiple pulsars. The system has four channels, each of which has three output modes mentioned above, and can support the verification of multiple navigation modes. This system consists of signal simulator and controller, single photon generator and detector, single photon screening and time tagging, and navigation algorithm verification. This paper presents the testing results of the system characteristics, the authenticity of the simulated photon arrival time series and the navigation verification. Monte Carlo experiments show that the recording accuracy of photon arrival time is 10 ns and the delays of the four channels are (11 ± 2), (15 ± 4), (14 ± 3), and (16 ± 4) \textμ\rms, respectively. The multi-physical properties of simulated photon arrival time series are introduced in detail, including photon flux, shape of observation profile, pulsar frequency characteristics and Doppler shift. The position and velocity errors of autonomous navigation algorithm test are 13.587 km and 14.277 m·s^–1, respectively, with an orbital altitude 26610 km and within 10 h. The ground verification system adopts master-slave control mode, the master computer mainly implements parameter setting and navigation algorithm verification, and the slave computer mainly carry out pulsar signal simulation. The communication based on TCP/IP protocol is applied to realize parameter transmission and real-time control between the master and slave computers in navigation verification process. The results of performance and functional test show that the system is available to accomplish the simulation of photon TOAs of X-ray pulsars at a spacecraft in real time and implement the ground verification of XPNAV.