Coherent detection employing multilevel modulation format has become one of the most promising technologies for next generation high speed transmission system due to the high power and spectral efficiencies. With the powerful digital signal processing (DSP), coherent optical receivers allow the significant equalization of chromatic dispersion (CD), polarization mode dispersion (PMD), phase noise (PN) and nonlinear effects in the electrical domain. Recently, the realizations of these DSP algorithms for mitigating the channel distortions in the transmission system are the most attractive investigations.  The CD equalization can be performed by the digital filters developed in the time and the frequency domain, which can suppress the fiber dispersion effectively. The PMD compensation is usually performed in the time domain with the adaptive least mean square (LMS) and constant modulus algorithms (CMA) equalization. Feed-forward and feed-back carrier phase estimation algorithms are employed to mitigate the phase noise from the transmitter and local oscillator lasers. The fiber nonlinearities are compensated by using the digital backward propagation methods based on solving the nolinear Schrodinger (NLS) equation and the Manakov equation.  In this dissertation, we present a comparative analysis of three digital filters for chromatic dispersion compensation, an analytical evaluation of carrier phase estimation with digital equalization enhanced phase noise and a brief discussion for PMD adaptive equalization. To implement these investigations, a 112-Gbit/s non-return-to-zero polarization division multiplexed quadrature phase shift keying (NRZ-PDM-QPSK) coherent transmission system is realized in the VPI simulation platform. With the coherent transmission system, these CD equalizers have been compared by evaluating their applicability for different fiber lengths, their usability for dispersion perturbations and their computational complexity. Meanwhile, the bit-error-rate (BER) floor in carrier phase estimation using a one-tap normalized LMS filter is evaluated analytically, and the numerical results are compared to a differential QPSK detection system. / QC 20110629

[pt] A comunicação por fibras ópticas utiliza diversos modelos herdados dos sistemas de telecomunicações tradicionais. Recentemente, a necessidade de maior controle sobre o fluxo de dados tem atraído muita atenção para as vantagens da comunicação óptica adaptativa. Num sistema de comunicação adaptativo, o fluxo de dados pode ser alterado em função de variações na qualidade do canal ou simplesmente pela necessidade de racionalizar a utilização dos recursos disponíveis. A interoperação entre redes pressiona pela necessidade de uma rede elástica e a expectativa é que este tipo de rede permita o controle sobre vários níveis da estrutura de comunicação. Nesta tese, a análise deste tema se concentra na camada física da rede óptica, em que a elasticidade pode ser obtida através de diferentes técnicas de modulação e multiplexação. A camada física de uma rede óptica adaptativa deve responder a variações e restrições do meio de transmissão. O consumo de energia, por exemplo, é um requisito cada vez mais presente nos projetos das redes de comunicação e a relevância deste requisito tende a aumentar na medida em que as redes ópticas aumentam sua capilaridade. O principal objetivo desta tese é analisar uma solução de comunicação óptica adaptativa que atenda aos requisitos básicos de uma rede elástica. O sistema de comunicação em análise é baseado em modulações realizadas no espaço de sinais de quatro dimensões, também conhecidas como modulações 4D. A perspectiva adotada privilegia a polarização da portadora óptica. A vantagem em adotar esta perspectiva, reside no fato de que ela permite a construção de modulações multidimensionais utilizando os fibrados de Hopf. Conforme será observado, o uso dos fibrados de Hopf em conjunto com o conceito matemático denominado vértice embutido de politopos, potencializa as soluções de engenharia para o problema da comunicação óptica adaptativa. / [en] Fiber-optic communications use several models inherited from traditional telecommunications systems. Recently, the need to improve the control over the data flow has attracted attention to the advantages of adaptive optical communication. In adaptive systems, the data flow can be altered due to changes in the channel quality or simply to rationalize the use of available resources. Interoperation between networks further presses on the need for an elastic network and the expectation is that this type of network will allow control over various levels of the communication structure. In this thesis, the analysis of this theme focuses on the physical layer of the optical network, where elasticity can be obtained through different modulation and multiplexing techniques. The physical layer of an adaptive optical network must respond to variations and restrictions of the transmission medium. Energy consumption, for example, is a requirement that is increasingly present in communication network projects and the relevance of this requirement tends to increase as optical networks expands in capillarity. The main objective of this thesis is to analyze an adaptive optical communication solution that meets the basic requirements of an elastic network. The communication system under analysis is based on the four-dimensional signal space modulations, also known as 4D modulations. The perspective adopted favors the polarization of the optical carrier. The advantage in adopting this perspective resides in the fact that it allows the construction of multidimensional modulations using Hopf bundles. As will be observed, the use of Hopf bundles in conjunction with the mathematical concept called embedded vertex polytopes, improves the engineering solutions to the problem of adaptive optical communication.