By expanding the density matrix of the open system in terms of Gell-mann matrix in a three-level system, we parameterize coefficients of expansion by some azimuthal angles and find an identity mapping of the density matrices onto interior points of the unit Poincaré sphere. Thus, the relations between the points on the unit Poincaré sphere and wave functions are extended to connect the interior points in the sphere with the nonunit vector rays corresponding to an open system in complex Hilbert space. Thus,the geometric phases for the open system are proposed to be observed by the nonunit vector rays,where the geometric phase of the pure state is the limiting case of our definition. The results show that this geometric phase merely with duplicate three-dimensional Hilbert projection space geometry structure related, has nothing to do with the open system concrete evolution way; and it depends on population inversion and is a slippy and single-value curve of Bloch radius. Therefore, the mixed state of open system retains indeed a memory of its motion in the form of a geometric phase factor.