The objective of this thesis is to study of optical effects, arising in the form of non-uniform waveguide structure, complicated refractive index profiles or due to pulse propagation in dense wavelength division multiplexing (DWDM) optical communication systems. These effects are important and critically influence the performance of DWDM optical systems. A comprehensive survey of current literature on optical effects due to nonuniform optical structure and nonlinear optical effects is first done, showing their advantages and disadvantage in optical communication systems. A survey on methods of optical waveguide analysis is also done. The main contribution has been made to three main aspects of the problem : Accurate analysis of uniform/non-uniform optical waveguides with arbitary refractive index profiles Pulse propagation and distortion in DWDM Raman amplification systems. Use of non-uniform FBG to compensate for pulse distortion We study several existing analytical techniques developed so far for analyzing the mode of non-uniform optical waveguide structures. Later, we verify the analytical results by finite element method (FEM). The convergence study is also carried out. A new computational technique is proposed modifying the finite element method to analyze complex refractive index profiles required for the analysis, namely single mode step index profile, multi clad fiber, W -profile, chirp profile etc. An accuracy of 10−4 in the calculation of propagation constant/eigen-value is demonstrated. Dispersion characteristics of optical fibers w.r.t. different profile parameters is evaluated. A modification to scalar BPM is proposed and applied to study the effects of inhomogeneities along the propagation direction. The applicability and accuracy of the method is tested using integrated optic waveguide devices, namely, graded index slab waveguide. The proposed BPM uses Fourier decomposition of the transverse field. Coupled mode theory (CMT) of optical waveguides in non-homogeneous optical medium is applied to study the interaction of lightwaves propagation together such as in a DWDM system. The BPM results is verified by CMT. The inhomogeneous waveguide theory is extended to study pulse propagation in DWDM optical communication system. Nonlinear optical effects are an important aspects of DWDM systems with fiber Raman amplifier. Finite difference time domain (FDTD) method is necessary to study these nonlinear optical effects as other conventional methods are not suitable here. Here, we discuss DWDM optical communication systems due to nonlinearity in the form of SRS effect. In case of FRA, we study the various kinds of fiber profile design parameters, for the purpose to achieve and extend the flat gain bandwidth over the EDFA window. We also propose and study, a new bi-directional optical fiber transmission scheme with various constraints, using Raman amplification process with and without pump depletion. Our scheme, provides an advantage like high SNR, low pump induced noise, for long-haul communication link. We find that, there is a quite significant crosstalk and power coupling among the dense DWDM channels but earlier discussed BPM fails to account for possible interference effects among the channels. To reduce the harmful nonlinear optical effects like four wave mixing (FWM), we need to deploy a high chromatic dispersion fiber, which will ultimately lead to high pulse walk-off rate among the DWDM channels; hence for high bit rate long haul systems, walk-off effect can not be ignored. Application of FDTD provided an improved insight into the effect of GVD on stimulated Raman scattering crosstalk than different modulation techniques and line codes. It is shown through analysis that pulse walk-off phenomena may distort the data asymmetrically; especially for case of wide-band DWDM transmission system. Hence, the pulse walk-off effect should be considered in future systems containing optical amplifier. It is shown, that large walk-off rate may reduce the crosstalk among DWDM channels but tends to increase the asymmetric pulse distortion. Data may lose due to high walk-off effect. We also investigate channel addition/removal process in DWDM fiber Raman amplifier. We also demonstrate that the pulse walk-off effect tends to lead significantly to positive chirp for higher frequency channels. This feature can be exploited to overcome the chromatic dispersion effects in DWDM transmission systems. Pulse walk-off induced chirp, can be compensated by using the nonuniform fiber Bragg grating (NUFBG). The CMT due to periodic perturbation of the circular cylindrical waveguide structures is applied here. Here, we discuss the function of fiber Bragg grating as a transmission versus reflecting grating filter. We also discuss, FBG application to gain flattening of an EDFA window as well as how the group velocity dispersion (GVD) will be affected with bandwidth and coupling coefficient. We develop a new analytical technique to estimate the bandwidth of FBG based optical system. Finally, we investigate the dispersion compensation properties, pulse distortion, peak reflectivity analysis in uniform/non-uniform FBG due to an uniform/non-uniform incoming signal. More complicated refractive index profile can significantly reduce the GVD as well as side lobes intensity. Dispersion characteristic due to an arbitrary refractive index profile is discussed in details for the case of non-uniform FBG. Thus, we concluded that wide band DWDM optical communication system need to closely take into account various inhomogeneities and nonlinearities of optical fibers w.r.t. wave and pulse propagation.