Security analysis of optical encryption system based on double random phase encoding indicates that the system can be classified as a linear symmetric block-cipher cryptosystem, which may lead to a great vulnerability. Under the ciphertext-only attack (COA), an opponent can attack such a cryptosystem only on the basis of estimated support of wave function in the object plane with iterative phase retrieval methods, and subsequently deduce the phase keys in the Fourier plane easily. The ciphertext-only attack (COA) requires much less resources than other types of attacks. Estimated support of wave function in the object plane could have some translations relative to the true support, so retrieved wave function could also have translations in both the amplitude and the phase, leading to a translation of retrieved plaintext relative to original plaintext. However, attackers can take this translation as a priori knowledge to traverse estimated support in the object plane until finding the best estimated keys, which bring about the best decryption quality.