An amorphous mixing layer (3.5–4.0 nm in thickness) containing silicon (Si), oxygen (O), molybdenum (Mo) atoms, named α-SiO_x(Mo), is usually formed by evaporating molybdenum trioxide (MoO₃) powder on an n-type Si substrate. In order to investigate the process of adsorption, diffusion and nucleation of MoO₃ in the evaporation process and ascertain the formation mechanism of α-SiO_x(Mo) on a atomic scale, the first principle calculation is used and all the results are obtained by using the Vienna ab initio simulation package. The possible adsorption model of MoO₃ on the Si (100) and the defect formation energy for substitutional defects and vacancy defects in α-SiO₂ and α-MoO₃ are calculated by the density functional theory. The results show that an amorphous layer is formed between MoO₃ film and Si (100) substrate according to ab initio molecular dynamics at 1500 K, which are in good agreement with experimental observations. The O and Mo atoms diffuse into Si substrate and form the bonds of Si—O or Si—O—Mo, and finally, form an α-SiO_x(Mo) layer. The adsorption site of MoO₃ on the reconstructed Si (100) surface, where the two oxygen atoms of MoO₃ bond with two silicon atoms of Si (100) surface, is the most stable and the adsorption energy is －5.36 eV, accompanied by the electrons transport from Si to O. After the adsorption of MoO₃ on the Si substrate, the structure of MoO₃ is changed. Two Mo—O bond lengths of MoO₃ are 1.95 Å and 1.94 Å, respectively, elongated by 0.22 Å and 0.21 Å compared with the those before the adsorption of MoO₃ on Si substrate, while the last bond length of MoO₃ is little changed. The defect formation energy value of neutral oxygen vacancy in α-SiO₂ is 5.11 eV and the defect formation energy values of neutral oxygen vacancy in α-MoO₃ are 0.96 eV, 1.96 eV and 3.19 eV, respectively. So it is easier to form oxygen vacancy in MoO₃ than in SiO₂, which implies that the oxygen atoms will migrate from MoO₃ to SiO₂ and forms a 3.5–4.0-nm-thick α-SiO_x(Mo) layer. As for the substitutional defects in MoO₃ and SiO₂, Mo substitutional defects are most likely to form in SiO₂ in a large range of Mo chemical potential. So based on our obtained results, the forming process of the amorphous mixing layer may be as follows: the O atoms from MoO₃ bond with Si atoms first and form the SiO_x. Then, part of Mo atoms are likely to replace Si atoms in SiO_x. Finally, the ultra-thin buffer layer containing Si, O, Mo atoms is formed at the interface of MoO₃/Si. This work simulates the reaction of MoO₃/Si interface and makes clear the interfacial geometry. It is good for us to further understand the process of adsorption and diffusion of atoms during evaporating, and it also provides a theoretical explanation for the experimental phenomenon and conduces to obtaining better interface passivation and high conversion efficiency of solar cell.