Silicon photonic switch is recognized as a cost-effective optical switching technology because it has many applications in long-haul telecommunication networks, short-reach data center and high-performance computing. In this paper, the research progress of various silicon photonic switch technologies is reviewed systematically. Firstly, the principles of three kinds of switch technologies including Mach-Zehnder interferometer (thermo-optic and carrier-injection types), micro-ring resonator (thermo-optic and carrier-injection types) and micro-electro-mechanical-system actuated waveguide coupler (electrostatic actuated type) are introduced. The switch technologies with the state-of-the-art insertion loss, crosstalk, switch time, footprint and power consumption are summarized and compared. Then the recent demonstrations of large-port silicon photonic matrix based on the above switch technologies are discussed. In this paper, we also investigate the key technologies such as topological architecture, passive components and optoelectronic packaging, which affect the performance of large-port optical switch matrix. Specifically, we study the scalability of various topologies, low-loss/broadband waveguide components, high-density optical/electrical packaging and control interface to improve the overall performance of the silicon photonic switch matrix. Finally, we discuss the critical technical challenges that might hamper the commercialization of silicon photonic switches and envision their future.