A necessidade de transmissão de canais modulados a taxas a partir de 400 Gb/s tem motivado a pesquisa e os esforços relativos às tecnologias de camada física habilitadores desta alta capacidade. A atenção se volta, principalmente, aos frontends (transmissores e receptores), aliados aos processadores digitais de sinal (Digital Signal Processors, DSPs), às técnicas de amplificação óptica e a novos tipos de fibra óptica. Em particular a técnica baseada no emprego de filtros de Nyquist combinados à multiplexação de comprimentos de onda (Wavelength Division Multiplexing, DWM), conhecida como Nyquist-WDM, ou N-WDM, tem atraído grande interesse para geração de supercanais ópticos, hoje um dos elementos chave nos sistemas de redes ópticas. O estudo dos fundamentos e casos particulares dos filtros de Nyquist são aprofundados nesta dissertação para o controle de seus parâmetros, em especial o parâmetro conhecido como fator de roll-off, em aplicações que requerem flexibilidade na ocupação espectral e até o reaproveitamento das limitações do filtro para atenuar alguns efeitos lineares e não lineares na fibra. A técnica utiliza um tipo de formatação geométrica de pulso e é limitada pelo ajuste grosso do fator de roll-off, mas como abordagem inicial, permite estabelecer uma série de compromissos na concepção do circuito eletrônico de um transponder sintonizável. Uma investigação teórica foi feita em um sistema PM-16QAM de 21x256 Gb/s, a partir de dados experimentais obtidos com roll-off igual 0,1, para análise do efeito no desempenho sistêmico do ajuste do excesso de largura de banda (em relação à banda de Nyquist) de um filtro formatador de pulso. O fator de roll-off foi ajustado e seu impacto no desempenho do sistema, em termos de alcance, foi verificado. A partir dos resultados, foi observado que, desde que a taxa de erro de bit, BER, esteja dentro do limite do código corretor de erro (forward error corrector, FEC), o valor de roll-off pode ser ajustado para um valor ótimo de acordo com a configuração do sistema e as metas requeridas. Uma vez encontrada a relação entre a BER e o fator de roll-off, foi possível determinar um fator de mérito que relaciona a resolução do filtro de Nyquist, em função do número de taps que ele emprega, o consumo de energia da DSP e, consequentemente, a BER. O compromisso assim estabelecido entre o desempenho sistêmico, o consumo de energia e o fator de roll-off representa a principal contribuição desta dissertação. / The need for transmission of channels modulated at rates greater than 400 Gb/s has motivated the research and efforts related to the physical layer technologies that will enable this high capacity. The attention turns mainly to the frontends (transmitters and receivers), allied to digital signal processors (DSPs), optical amplification techniques and new types of optical fiber. The technique based on the use of Nyquist filters combined withWavelength Division Multiplexing (WDM), known as Nyquist-WDM, or N-WDM, has attracted great interest for the generation of optical super-channels, today one of the key elements in optical network systems. The study of the fundamentals and particular cases of the Nyquist filters are detailed in this dissertation for mastering the control of the parameters, especially the parameter known as roll-off factor, for applications that require flexibility in the spectral occupation and even the reutilization of the limitations of the filter to attenuate some linear and non-linear effects on the fiber. The technique uses a geometric type of pulse-shaping, and is limited by the roll-off factor tunning, but as an initial approach, it allows to establish a series of compensations in the design of the electronic circuit of a tunable transponder. A theoretical investigation was made on a 21x256 Gb/s PM-16QAM system, taken as reference the experimental data obtained with roll-off equal to 0.1, to analyze the effects of adjusting the excess bandwidth (relative to the Nyquist band) of a pulse-shaping filter. The roll-off factor was tunned and its impact on the system performance in terms of range effects was verified. From the results, it was observed that, as long as the bit error ratio, BER, is within the FEC limit, the roll-off parameter can be set to an optimum value according to the system configuration and required targets. Once the relationship between the BER and the roll-off factor was found, it was possible to determine a merit factor that relates the resolution of the Nyquist filter, as a function of the number of taps it uses, the energy consumption of the DSP and, consequently, the BER. The compromise thus established between system performance, energy consumption and roll off represents the main contribution of this work.

A necessidade de transmissão de canais modulados a taxas a partir de 400 Gb/s tem motivado a pesquisa e os esforços relativos às tecnologias de camada física habilitadores desta alta capacidade. A atenção se volta, principalmente, aos frontends (transmissores e receptores), aliados aos processadores digitais de sinal (Digital Signal Processors, DSPs), às técnicas de amplificação óptica e a novos tipos de fibra óptica. Em particular a técnica baseada no emprego de filtros de Nyquist combinados à multiplexação de comprimentos de onda (Wavelength Division Multiplexing, DWM), conhecida como Nyquist-WDM, ou N-WDM, tem atraído grande interesse para geração de supercanais ópticos, hoje um dos elementos chave nos sistemas de redes ópticas. O estudo dos fundamentos e casos particulares dos filtros de Nyquist são aprofundados nesta dissertação para o controle de seus parâmetros, em especial o parâmetro conhecido como fator de roll-off, em aplicações que requerem flexibilidade na ocupação espectral e até o reaproveitamento das limitações do filtro para atenuar alguns efeitos lineares e não lineares na fibra. A técnica utiliza um tipo de formatação geométrica de pulso e é limitada pelo ajuste grosso do fator de roll-off, mas como abordagem inicial, permite estabelecer uma série de compromissos na concepção do circuito eletrônico de um transponder sintonizável. Uma investigação teórica foi feita em um sistema PM-16QAM de 21x256 Gb/s, a partir de dados experimentais obtidos com roll-off igual 0,1, para análise do efeito no desempenho sistêmico do ajuste do excesso de largura de banda (em relação à banda de Nyquist) de um filtro formatador de pulso. O fator de roll-off foi ajustado e seu impacto no desempenho do sistema, em termos de alcance, foi verificado. A partir dos resultados, foi observado que, desde que a taxa de erro de bit, BER, esteja dentro do limite do código corretor de erro (forward error corrector, FEC), o valor de roll-off pode ser ajustado para um valor ótimo de acordo com a configuração do sistema e as metas requeridas. Uma vez encontrada a relação entre a BER e o fator de roll-off, foi possível determinar um fator de mérito que relaciona a resolução do filtro de Nyquist, em função do número de taps que ele emprega, o consumo de energia da DSP e, consequentemente, a BER. O compromisso assim estabelecido entre o desempenho sistêmico, o consumo de energia e o fator de roll-off representa a principal contribuição desta dissertação. / The need for transmission of channels modulated at rates greater than 400 Gb/s has motivated the research and efforts related to the physical layer technologies that will enable this high capacity. The attention turns mainly to the frontends (transmitters and receivers), allied to digital signal processors (DSPs), optical amplification techniques and new types of optical fiber. The technique based on the use of Nyquist filters combined withWavelength Division Multiplexing (WDM), known as Nyquist-WDM, or N-WDM, has attracted great interest for the generation of optical super-channels, today one of the key elements in optical network systems. The study of the fundamentals and particular cases of the Nyquist filters are detailed in this dissertation for mastering the control of the parameters, especially the parameter known as roll-off factor, for applications that require flexibility in the spectral occupation and even the reutilization of the limitations of the filter to attenuate some linear and non-linear effects on the fiber. The technique uses a geometric type of pulse-shaping, and is limited by the roll-off factor tunning, but as an initial approach, it allows to establish a series of compensations in the design of the electronic circuit of a tunable transponder. A theoretical investigation was made on a 21x256 Gb/s PM-16QAM system, taken as reference the experimental data obtained with roll-off equal to 0.1, to analyze the effects of adjusting the excess bandwidth (relative to the Nyquist band) of a pulse-shaping filter. The roll-off factor was tunned and its impact on the system performance in terms of range effects was verified. From the results, it was observed that, as long as the bit error ratio, BER, is within the FEC limit, the roll-off parameter can be set to an optimum value according to the system configuration and required targets. Once the relationship between the BER and the roll-off factor was found, it was possible to determine a merit factor that relates the resolution of the Nyquist filter, as a function of the number of taps it uses, the energy consumption of the DSP and, consequently, the BER. The compromise thus established between system performance, energy consumption and roll off represents the main contribution of this work.

Les systèmes de communications optiques jouent un role important pour satisfaire la demande incessante de trafics de données. Cette demande, induite par des applications gourmandes en termes de bande passante et débit, necéssite une augmentation de la capacité des réseaux optiques d’accès et par conséquent une augmentation des capacités de réseaux de transports métropolitains et longues distances. La prochaine génération de systèmes WDM longue distance devrait opérée à des débits de 400Gbps ou 1Tbps. Cette montée en débit s’appuiera sur des nouvelles formes d’ondes avancées de type mono-porteuse (Nyquist-WDM) ou multi-porteuse (OFDM multi-bande). Ces approches sont basées sur le multipléxage de plusieurs porteuses espacées par des intervalles de garde réduits. D’autre part, pour générer ces très haut débits, des modulations multi-états sont utilisées pour chaque porteuse grâce à leur efficacité spectrale élevée. Ces types de systèmes, qui combinent à la fois les approches multi-bande et les modulations multi-états, sont extrêmement vulnérables aux effets nonlinéaires de la fibre optique. En fait, les effets nonlinéaires sont dépendants de la puissance de transmission et inversement proportionels à l’intervalle de garde. Cela rend leur compensation indispensable pour maintenir des bonnes performances des systèmes en terme de distance de transmission. Grâce à l’emploi de récepteurs à détection cohérente, des techniques de traitement du signal numérique sont utlisées pour combattre les effets nonlinéaires. Dans cette thèse, nous avons proposé des nouvelles techniques basées sur les séries de Volterra et les égaliseurs à retour de decision pour compenser respectivement les effets nonlinéaires intrabande et les interférences nonlinéaires inter-bande. / Optical communication systems have evolved since their deployment to meet the growing demand for high-speed communications. Over the past decades, the global demand for communication capacity has increased exponentially and the most of the growth has occurred in the last few years when data started dominating network traffic. In order to meet the increase of traffic demands fueled by the growth of internet services, an increase of access network capacity and consequently metro and long-haul network capacities is required. Next generation of long-haul WDM transmission systems is expected to operate at 400Gbps or 1Tbps bit rate. Superchannel approaches, such as Nyquist WDM and multi-band OFDM, allow both high spectral efficiency and small guardband which makes them promising candidates to generate these high bit rates in combination with multi-level modulations formats. Such transmission systems are strongly disturbed by fiber nonlinear effects which increase with the data rate and the small guard band. Therefore, fiber nonlinearities compensation is required to get the desired performance in terms of transmission reach. DSP based approaches such as digital back propagation and third-order Volterra based nonlinear equalizer have been already proposed to deal with intra-channel or intra-band nonlinear effects. In the context of superchannel systems, we have proposed two new compensation techniques to deal with fiber nonlinear effects. The first one, called fifth-order inverse Volterra based nonlinear equalizer, compensate for intra-band nonlinear effects. The second approach, which is the interband/ subcarrier nonlinear interference canceler, is proposed to combat the nonlinear interference insuperchannel systems.