Quantum noise is an important factor influencing the quality of the images obtained by X-ray phase contrast imaging because it induces a random fluctuation on the intensity. In this paper, a method is reported to analyze the noise characteristics of hard X-ray differential phase contrast imaging, and the relationship between the quantum noise and images is determined by numerical simulation. The results show that the mean square deviations of the refractive and scattering images are inversely proportional to the visibility of moir fringe. That is to say, the higher visibility of moir fringe leads to the lower mean square deviations of the refractive and scattering images. Therefore, in the case of ensuring an enough X-ray detection efficiency, the improvement in the visibility of moir fringe will be beneficial to obtaining high-quality phase contrast images and the effective reduction of X-ray exposure dose.