Silicon photonics has witnessed rapid development in recent years for its fabrication compatibility with the cost-effective CMOS technology. The advancement of relevant simulation tools, however, is at a relatively slow pace. The high index contrast of the usual silicon waveguide that has imposed new challenges to the convergence and accuracy of the solution technique, the growing intricacy in solitary component design, and the increased complexity of their integration, are the impelling factors that motivate us to improve the computer-aided design, modeling, simulation, and optimization methods. The theme of the thesis is on the frequency domain simulation methods supported by the complex mode theory. The complex mode theory is introduced to the simulation domain truncated by the perfectly matching layers (PMLs) enclosed in the perfectly reflected boundaries (PRBs), wherein the discrete complex modes as eigen solutions can represent the continuous radiation fields, thus yields a unified approach for handling both guided (discrete) and radiation (continuous) waves. In this thesis, theoretical investigations have been conducted along a few different lines aiming at improving the efficiency and accuracy in complex mode expansion. Properties of high-order complex Berenger modes are firstly addressed through asymptotic solutions, and it is found that as the mode order increases, the symmetry of the cladding and substrate in the simulation domain, instead of the guiding schemes, plays a more and more decisive role regarding mode classification and modal field distribution. A weighed optical path method is then proposed to unify the high-order Berenger modes, and to enhance the symmetry of high order modes’ field distributions in the asymmetric structures, leading to the improvement in convergence speed and stability in the mode expansion. Next, an improved mode-matching method (MMM) is proposed based on an error-minimizing method instead of the conventional approach relying on the unreliable modal orthogonal property. The newly proposed method is significantly more robust as numerical errors usually jeopardize the modal orthogonality. This claim is exemplified by simulation results on silicon channel waveguide facet, bending waveguide, and silicon-germanium photo-detector waveguide. As a direct application of the improved complex mode theory, a hybrid plasmonic-photonic nano-ribbon waveguide is proposed, standing as a combination of the silicon slot and surface plasmon polariton (SPP) waveguides, is proposed and analyzed. We have found that the fundamental mode is featured at low loss as in optical waveguide as well as high confinement as in plasmonic structure. Simulations have shown that millimeter range propagation can be sustained with strong confinement. We have further studied such waveguide with an extra layer of phase changing material incorporated, attempting to realize the efficient electro-optical phase and/or loss modulation. Finally, an optical switch design is proposed by taking the full advantage of the aforementioned structure. / Thesis / Doctor of Philosophy (PhD)