The Nyquist WDM system realizes a terabit high spectral efficiency transmission system by allocating several subcarriers close to or equal to the baud rate. This system achieves optimal performance by maintaining both temporal and spectral orthogonality. However, ISI and ICI effects are inevitable in practical Nyquist WDM implementations due to the imperfect channel response and tight channel spacing and may cause significant performance degradations. Our primary research goals are to combat the ISI effects via the transmitter digital pre-shaping and to remove the ICI impairments at the receiver using MIMO signal processing. First we propose two novel blind channel estimation techniques that enable the transmitter pre-shaping design for the ISI effects mitigation. Both numerical and experimental results demonstrate that the two methods are very effective in compensating the narrow band filtering and are very robust to channel estimation noise. Besides pre-shaping, the DAC-enabled transmitter chromatic dispersion compensation is also demonstrated in a system with high LO laser linewidth. Next a novel “super-receiver” structure is proposed, where different subchannels are synchronously sampled, and the baseband signals from three adjacent subchannels are processed jointly to remove ICI penalty. Three different ICI compensation methods are introduced and their performances compared. The important pre-processes that enable a successful ICI compensation are also elaborated. Despite ICI compensation, the joint carrier phase recovery based on the Viterbi-Viterbi algorithm is also studied in the carrier phase locked systems. In-band crosstalk arises from the imperfect switch elements in the add-drop process of ROADM-enabled DWDM systems and may cause significant performance degradation. Our third research topic is to demonstrate a systematic way to analyze and predict the in-band crosstalk-induced penalty. In this work, we propose a novel crosstalk-to-ASE noise weighting factor that can be combined with the weighted crosstalk weighting metric to incorporate the in-band crosstalk noise into the Gaussian noise model for performance prediction and analysis. With the aid of the Gaussian noise model, the in-band crosstalk-induced nonlinear noise is also studied. Both simulations and experiments are used to validate the proposed methods.

Les systèmes de communications optiques jouent un role important pour satisfaire la demande incessante de trafics de données. Cette demande, induite par des applications gourmandes en termes de bande passante et débit, necéssite une augmentation de la capacité des réseaux optiques d’accès et par conséquent une augmentation des capacités de réseaux de transports métropolitains et longues distances. La prochaine génération de systèmes WDM longue distance devrait opérée à des débits de 400Gbps ou 1Tbps. Cette montée en débit s’appuiera sur des nouvelles formes d’ondes avancées de type mono-porteuse (Nyquist-WDM) ou multi-porteuse (OFDM multi-bande). Ces approches sont basées sur le multipléxage de plusieurs porteuses espacées par des intervalles de garde réduits. D’autre part, pour générer ces très haut débits, des modulations multi-états sont utilisées pour chaque porteuse grâce à leur efficacité spectrale élevée. Ces types de systèmes, qui combinent à la fois les approches multi-bande et les modulations multi-états, sont extrêmement vulnérables aux effets nonlinéaires de la fibre optique. En fait, les effets nonlinéaires sont dépendants de la puissance de transmission et inversement proportionels à l’intervalle de garde. Cela rend leur compensation indispensable pour maintenir des bonnes performances des systèmes en terme de distance de transmission. Grâce à l’emploi de récepteurs à détection cohérente, des techniques de traitement du signal numérique sont utlisées pour combattre les effets nonlinéaires. Dans cette thèse, nous avons proposé des nouvelles techniques basées sur les séries de Volterra et les égaliseurs à retour de decision pour compenser respectivement les effets nonlinéaires intrabande et les interférences nonlinéaires inter-bande. / Optical communication systems have evolved since their deployment to meet the growing demand for high-speed communications. Over the past decades, the global demand for communication capacity has increased exponentially and the most of the growth has occurred in the last few years when data started dominating network traffic. In order to meet the increase of traffic demands fueled by the growth of internet services, an increase of access network capacity and consequently metro and long-haul network capacities is required. Next generation of long-haul WDM transmission systems is expected to operate at 400Gbps or 1Tbps bit rate. Superchannel approaches, such as Nyquist WDM and multi-band OFDM, allow both high spectral efficiency and small guardband which makes them promising candidates to generate these high bit rates in combination with multi-level modulations formats. Such transmission systems are strongly disturbed by fiber nonlinear effects which increase with the data rate and the small guard band. Therefore, fiber nonlinearities compensation is required to get the desired performance in terms of transmission reach. DSP based approaches such as digital back propagation and third-order Volterra based nonlinear equalizer have been already proposed to deal with intra-channel or intra-band nonlinear effects. In the context of superchannel systems, we have proposed two new compensation techniques to deal with fiber nonlinear effects. The first one, called fifth-order inverse Volterra based nonlinear equalizer, compensate for intra-band nonlinear effects. The second approach, which is the interband/ subcarrier nonlinear interference canceler, is proposed to combat the nonlinear interference insuperchannel systems.