Target tracking has been introduced as a key point in the physical applications, such as passive sonar and chaotic communication etc. It is typically a nonlinear filtering problem to estimate the position and the velocity of a target from noise-corrupted measurements. Some approaches have been proposed for the problem, such as the extended Kalman filter, the unscented Kalman filter, and the cubature Kalman filter (CKF). However, they are effective only in the Gaussian and white assumption for the measurements. Actually, the measurements are easily polluted by the measurement outliers in practice. The measurement outliers may lead to inaccurate performance due to non-symmetrical or non-Gaussian property. In order to cope with the measurement outliers in nonlinear target tracking system, a robust filtering algorithm called the M-estimation based robust cubature Kalman filter (MR-CKF) is proposed for the target tracking problem. Firstly, the nonlinear measurement equation is transformed into an equivalently linear form according to the orthogonal vector, and then the Gaussian extremal function of the target tracking can be obtained by the constrained total least square (CTLS) criterion. By employing the Huber's robust score function, the Gaussian extremal function is further rendered into a robust extremal function, thus the generalized M-estimation can be introduced to the CKF without linearization approximation. The only difference between the Gaussian extremal function and the robust extremal function is the weight matrix, implying that the CKF solution framework does not change and the virtues of both the CKF and M-estimation can be fully utilized such as derivative-free, high accuracy and robust performance. Furthermore, an improved Huber equivalent weight function is designed for the MR-CKF based on the Mahalanobis distance. The outliers' judge threshold is determined according to the confidence level of Chi-square distribution and improper empirical value of the Huber's method can be avoided. In addition, the improved Huber weight function reduces weights of small outliers and removes large outliers, and this is more robust and reasonable than the Huber's method. Moreover, the statistical information of outliers is also not required. Theoretical analysis and numerical results show that the proposed filtering algorithm can improve the accuracy and robustness than the conventional robust algorithms.