In recent years, with the continuous development of computer technology, it has brought convenience to people to obtain image information. However, at the same time, the falsification and theft of image information have also emerged, so information security has received increasing attention. When images are used for medicine, military, court, and other purposes, it is necessary to ensure the authenticity and integrity of the image content. Fragile watermarks are used to verify the authenticity and integrity of image content due to their sensitivity to tampering. The watermark information is embedded in the image and integrated with the image. When it is necessary to detect the authenticity and integrity of image information, the extracted watermark can be used to determine whether the image is reliable and complete. Therefore, we propose an optical color fragile watermarking system based on pixel-free expansion visual cryptography. On the one hand, encoding watermark images by using pixel-free expansion visual cryptography avoids pixel expansion issues caused by visual cryptography, allowing for the selection of color host images with the same pixel size as the watermark image in the future, greatly reducing the network bandwidth and storage space occupied during transmission. On the other hand, phase recovery algorithm is used to process the encoded watermark image to obtain phase information for embedding into the host image, further improving the security of the watermark image in an optical way. The feasibility and imperceptibility of the proposed optical color fragile watermark are verified through computer simulation, and its good fragility is verified through a series of simulation attack experiments. It can sensitively detect image tampering in the face of common attacks such as noise pollution, rotation, motion blur processing, filtering, etc.