Theory of chaotic signal transmission in optical fiber and physical mechanism of chaotic signal acting on optical fiber transmission medium are theoretically studied. A physical model of nonlinear evolution of chaotic signal propagation in optical fiber is presented by coupling chaotic laser system with optical fiber channel. Action of self-phase modulation of optical fiber on propagation and evolution of chaotic signal is analyzed in detail. Self-phase modulation of fiber does not affect the pulse shape of chaotic signal, however, its product of nonlinear phase shift can widen the chaotic signal spectrum. Self-phase modulation does not affect the power distribution and field intensity distribution of chaotic pulse signal. However, it can affect the power spectrum distribution, the transformation of optical field of chaotic signal and field's slow-variying part. A frequency chipped model of nonlinear evolution of chaotic signal in fiber is presented. The evolution and characteristic of the phase, spectrum, field and field's slow-varying part in fiber are numerically simulated.