In this paper, a new method of encrypting a color image based on θ modulation is proposed by using the tricolor principle and computer-generated hologram (CGH) technology. The encryption process includes the θ-modulated three primary color components and the coding of computer-generated hologram, which is implemented in a Fresnel diffraction and spatial filtering system. Firstly, the color image modulated by the first random phase key is divided into three encryption channels by red laser, green laser, blue laser, and tricolor filters. Each channel is introduced by a transmissive amplitude-type sinusoidal grating with different directions, which is used to separate the three primary color components in the spatial spectrum plane. Secondly, the modulation results of tricolor components are superimposed together to form a compound image, and the phase truncation of the superposition result is performed to achieve the asymmetric encryption. Finally, the amplitude of the compound image is modulated by the second random phase key and is encoded into a binary real-value gray-color CGH by Roman-type coding method. Therefore, the gray-color information of the original image is completely hidden in the encrypted CGH, which is more general and deceptive in the storage and transmission process. Decryption is an inverse process of the encryption. Firstly, the encrypted CGH is placed on the input plane of the spatial filtering and Fresnel diffraction system. Secondly, the demodulation of CGH phase key and the spatial filtering based on optical filter are performed. Finally, the color plaintext image is obtained by using the correct Fresnel diffraction. The simulation results show the validity and feasibility of the proposed method. In addition, the anti-noise attack and anti-shearing attack performance of this color image encryption method are investigated. Compared with results from the three presented methods reported in the literature, our investigated results demonstrate that this method has good robustness to noise attack and shearing attack, and has obvious advantages when the attack noise density is larger. Due to the characteristics of high security and anti-noise, we believe that this color image encryption method promises to have important applications in the information transmission and multi-user authentication.