In this thesis, we proposed general forms of eigen-mode analysis for multilayer optical waveguides. This study discussed the periodic structure in transverse direction and used the slowly varying envelope approximation to approximate the wave function. Firstly, we presented a general method for analyzing the multilayer nonlinear optical waveguide structure by using modal theory. The nonlinear optical waveguide is a medium whose refractive index changes with the electric field intensity. The general method can also be degenerated into some other special cases for analyzing multilayer nonlinear optical waveguide. Secondly, a general method for analyzing the multilayer optical waveguides with photonic metamaterials characterized by simultaneously negative dielectric permittivity and magnetic permeability was studied. The research pointed out explicitly that the three-layer planar waveguide with photonic metamaterials could support forbidden regions. The complete set of modes of all possible solutions for the TE wave in photonic metamaterials optical waveguide was found. The transverse electric field distributions and dispersion relations in multilayer optical waveguides can be obtained by using these general forms. Finally, we used the general forms to design an all-optical mode converter which composed of a pair of multibranch optical waveguides. The analytical and numerical results show excellent agreement.