Detecting homoclinic orbits is a key problem in nonlinear dynamical systems, especially in the study of bifurcation and chaos. In this paper, we propose a new method to solve the problem with trajectory optimization. By defining a distance between a saddle point and its near trajectories, the problem becomes a common problem in unconstrained nonlinear optimization to minimize the distance. A subdivision algorithm is also proposed in this paper to improve the integrity of results. By applying the algorithm to the Lorenz system, the Shimizu-Morioka system and the hyperchaotic Lorenz system, we successfully find many homoclinic orbits with the corresponding parameters, which suggests that the method is effective.