This thesis studies and mathematically models nonlinear interactions among channels of modern high bit rate (amplitude/) phase modulated optical systems. First, phase modulated analogue systems are studied and a differential receiving method is suggested with experimental validation. The main focus of the rest of the thesis is on digital advanced modulation format systems. Cross-talk due to fiber Kerr nonlinearity in two-format hybrid systems as well as 16-QAM systems is mathematically modelled and verified by simulation for different system parameters. A comparative study of differential receivers and coherent receivers is also given for hybrid systems. The model is based on mathematically proven assumptions and provides an intuitive analytical understanding of nonlinear cross-talk in such systems. / Graduate