Compared with traditional X-ray imaging systems based on polycapillary X-ray optics or a pinhole, coded aperture imaging system has the advantages in simple structure, high sensitivity, and strong expandability, which make it possess the potential applications in X-ray fluorescence imaging. In this work, a new coded aperture X-ray imaging system based on a novel imaging model which decomposes the mask projections into a superposition of two separable functions is designed and proposed for high-resolution X-ray imaging. The performance of the system is demonstrated by using the Geant4 package. To reduce the computational complexity of calibration and image reconstruction, a separable mask with 90 × 90 pixels is used. The mask is designed by selecting the central part of the original rank 463 modified uniformly redundant arrays. The mask is made of platinum foil with a pixel pitch of 25 microns. To study the effect of mask thickness on system performance, the mask thickness is varied from 25 to 200 microns. The active area of the Si detector employed in the system is 2 mm × 2 mm, divided into 80 × 80 pixels, each with a size of 25 μm × 25 μm. The field of view of the system is equal to the area of the detector, which is 2 mm × 2 mm. The detector is parallel to and center-aligned with the mask with a fixed distance of 2.0 mm. The images are reconstructed by using the fast iterative shrinkage-thresholding algorithm. The high-quality reconstructed images of different energy line sources and complex shaped objects are obtained. The simulation and analysis results indicate that for the near-field imaging, unlike imaging systems based on the conventional convolution model, the system has the performance that is not affected by the aperture collimation effect. The spatial resolution of the imaging system is about 65 microns. The calibrated matrices used have an important influence on the image quality. The quality of the reconstructed image is affected by the energy of X-rays used during calibration and the energy of X-rays emitted from the object; the smaller the difference between these two energy values, the higher the quality of the reconstructed images will be. The three-dimensional reconstruction results show that the system can correctly estimate the distance between the object and the system from a single two-dimensional projection. The axial spatial resolution of the system is about 1.1 mm.