A new type of diffraction grating called the compound modulated diffraction grating is proposed, which incorporates both surface relief and periodic refractive index modulation. Based on the angular spectrum representation and strict modal theory, the propagation of linearly polarized Gaussian beam through the compound-modulated diffraction grating whose characteristic size is comparable with the wavelength is studied. It is shown that the propagation of beam in each layer is different and the intensity fluctuation frequency in the compound-modulated grating is smaller than that in the conventional relief grating at the same transmission depth. Using the compound modulated grating model, the influence of surface fluctuation of the sub-wavelength_volume phase holographic grating and waist width of the incident beam on the diffraction efficiency is analyzed.