In a passive target tracking system, the position and velocity of a target can be estimated based on time difference of arrival (TDOA) and frequency difference of arrival (FDOA) received by different stations. But TDOA and FDOA equations are nonlinear, which make the target tracking become a nonlinear estimation problem. To solve the nonlinear estimation problem, the most extensive research algorithms are those of extended Kalman filter (EKF), particle filter (PF), unscented Kalman filter (UKF), quadrature Kalman filter (QKF), and cubature Kalman filter (CKF). But the existing algorithms all come up with shortcoming in some way. EKF only retains the first order of the nonlinear function by Taylor series expansion, which will bring large error. PF has to face the degeneracy phenomenon and the problem of large computational complexity. The standard UKF is easy to become divergence in a high dimensional state estimation. QKF is sensitive to the dimension of state, and the calculation is of exponential growth with the growth of dimension. Although CKF can effectively improve the shortcomings, the discarded error is proportional to the state dimension, which may be large in high dimensional state. In view of the above problems, this paper presents an orthogonal cubature Kalman filter (OCKF) algorithm. This algorithm reduces the sampling error by introducing special orthogonal matrix to change the method of cubature sampling based on CKF. It eliminates the dimension impact on the sampling error. In the absence of additional computation, it effectively improves the tracking precision. Simulation results show that, based on the TDOA and FDOA, compared with the EKF and CKF algorithms, OCKF algorithm can improve the tracking performance significantly.