Optical fibers play a vital role in modern telecommunication systems and networks. An optical fiber link imposes some linear and nonlinear distortions on the propagating light-wave signal due to the inherent dispersive nature and nonlinear behavior of the fiber. These distortions impede the increasing demand for higher data rate transmission over longer distances. Developing efficient and computationally non-expensive digital signal processing (DSP) techniques to effectively compensate for the fiber impairments is therefore essential and of preeminent importance. This thesis proposes two DSP-based approaches for mitigating the induced distortions in short-reach and long-haul fiber-optic communication systems. The first approach introduces a powerful digital nonlinear feed-forward equalizer (NFFE), exploiting multilayer artificial neural network (ANN). The proposed ANN-NFFE mitigates nonlinear impairments of short-haul optical fiber communication systems, arising due to the nonlinearity introduced by direct photo-detection. In a direct detection system, the detection process is nonlinear due to the fact that the photo-current is proportional to the absolute square of the electric field intensity. The proposed equalizer provides the most efficient computational cost with high equalization performance. Its performance is comparable to the benchmark compensation performance achieved by maximum-likelihood sequence estimator. The equalizer trains an ANN to act as a nonlinear filter whose impulse response removes the intersymbol interference (ISI) distortions of the optical channel. Owing to the proposed extensive training of the equalizer, it achieves the ultimate performance limit of any feed-forward equalizer. The performance and efficiency of the equalizer are investigated by applying it to various practical short-reach fiber-optic transmission system scenarios. These scenarios are extracted from practical metro/media access networks and data center applications. The obtained results show that the ANN-NFFE compensates for the received BER degradation and significantly increases the tolerance to the chromatic dispersion distortion. The second approach is devoted for blindly combating impairments of long-haul fiber-optic systems and networks. A novel adjoint sensitivity analysis (ASA) approach for the nonlinear Schrödinger equation (NLSE) is proposed. The NLSE describes the light-wave propagation in optical fiber communication systems. The proposed ASA approach significantly accelerates the sensitivity calculations in any fiber-optic design problem. Using only one extra adjoint system simulation, all the sensitivities of a general objective function with respect to all fiber design parameters are estimated. We provide a full description of the solution to the derived adjoint problem. The accuracy and efficiency of our proposed algorithm are investigated through a comparison with the accurate but computationally expensive central finite-differences (CFD) approach. Numerical simulation results show that the proposed ASA algorithm has the same accuracy as the CFD approach but with a much lower computational cost. Moreover, we propose an efficient, robust, and accelerated adaptive digital back propagation (A-DBP) method based on adjoint optimization technique. Provided that the total transmission distance is known, the proposed A-DBP algorithm blindly compensates for the linear and nonlinear distortions of point-to-point long-reach optical fiber transmission systems or multi-point optical fiber transmission networks, without knowing the launch power and channel parameters. The NLSE-based ASA approach is extended for the sensitivity analysis of general multi-span DBP model. A modified split-step Fourier scheme method is introduced to solve the adjoint problem, and a complete analysis of its computational complexity is studied. An adjoint-based optimization (ABO) technique is introduced to significantly accelerate the parameters extraction of the A-DBP. The ABO algorithm utilizes a sequential quadratic programming (SQP) technique coupled with the extended ASA algorithm to rapidly solve the A-DBP training problem and optimize the design parameters using minimum overhead of extra system simulations. Regardless of the number of A-DBP design parameters, the derivatives of the training objective function with respect to all parameters are estimated using only one extra adjoint system simulation per optimization iterate. This is contrasted with the traditional finite-difference (FD)-based optimization methods whose sensitivity analysis calculations cost per iterate scales linearly with the number of parameters. The robustness, performance, and efficiency of the proposed A-DBP algorithm are demonstrated through applying it to mitigate the distortions of a 4-span optical fiber communication system scenario. Our results show that the proposed A-DBP achieves the optimal compensation performance obtained using an ideal fine-mesh DBP scheme utilizing the correct channel parameters. Compared to A-DBPs trained using SQP algorithms based on forward, backward, and central FD approaches, the proposed ABO algorithm trains the A-DBP with 2.02 times faster than the backward/forward FD-based optimizers, and with 3.63 times faster than the more accurate CFD-based optimizer. The achieved gain further increases as the number of design parameters increases. A coarse-mesh A-DBP with less number of spans is also adopted to significantly reduce the computational complexity, achieving compensation performance higher than that obtained using the coarse-mesh DBP with full number of spans. / Thesis / Doctor of Philosophy (PhD) / This thesis proposes two powerful and computationally efficient digital signal processing (DSP)-based techniques, namely, artificial neural network nonlinear feed forward equalizer (ANN-NFFE) and adaptive digital back propagation (A-DBP) equalizer, for mitigating the induced distortions in short-reach and long-haul fiber-optic communication systems, respectively. The ANN-NFFE combats nonlinear impairments of direct-detected short-haul optical fiber communication systems, achieving compensation performance comparable to the benchmark performance obtained using maximum-likelihood sequence estimator with much lower computational cost. A novel adjoint sensitivity analysis (ASA) approach is proposed to significantly accelerate sensitivity analyses of fiber-optic design problems. The A-DBP exploits a gradient-based optimization method coupled with the ASA algorithm to blindly compensate for the distortions of coherent-detected fiber-optic communication systems and networks, utilizing the minimum possible overhead of performed system simulations. The robustness and efficiency of the proposed equalizers are demonstrated using numerical simulations of varied examples extracted from practical optical fiber communication systems scenarios.

La majeure partie des communications mondiales est transportée par des systèmes transocéaniques à fibre optique. Il est estimé que d'ici 2020 le trafic de données atteindra 4.3 ZB par an. Afin de faire face à cette demande, différentes technologies sont actuellement étudiées pour augmenter la capacité des systèmes de transmission très longue distance. Avec l'avènement des circuits intégrés à haute vitesse, des formats de modulation avancés et des techniques de traitement de signal numérique (DSP) peuvent être utilisés pour maximiser l'efficacité spectrale de transmission. Par ailleurs, la capacité des systèmes modernes est fortement limitée par les effets non-linéaires de type Kerr dans la fibre. Ainsi, la première partie de ce travail est axée sur l’étude de la performance et des gains réalisables par des techniques DSP à faible complexité pour mitiger les effets non-linéaires monocanal. En outre, l’utilisation des formats de modulation multiniveaux à haute efficacité spectrale au-delà de 16QAM a pris de l'ampleur pour augmenter le débit de transmission des systèmes, notamment avec l’introduction des formats QAM avec mise en forme probabiliste (PCS-QAM), plus performants que les formats QAM classiques. La deuxième partie de ce travail présente donc une comparaison théorique ainsi qu’expérimentale du format PCS-64QAM avec d’autres formats à haute efficacité spectrale pour les distances transatlantiques. La mise en œuvre d’un format PCS-64QAM conçu pour les distances transpacifiques est également abordée. Enfin, la dernière partie de ce travail concrétise les résultats des travaux menés dans les deux sections précédentes en présentant plusieurs records de transmission / Global data traffic is expected to reach up to 4.3 ZB per year by 2020. With the majority of the global communications being transported on submarine point-to-point fiber-optic systems, different cutting-edge technologies have been under research to cope with this unprecedented traffic growth. Continuous advances in high-speed integrated circuits have allowed the use of advanced modulation formats and digital signal processing (DSP) techniques to maximize the transmission spectral efficiency. With mitigation of fiber linear effects efficiently carried out by DSP with relative low-complexity, the capacity of modern fiber optic systems rests limited by fiber nonlinearities. To this extent, in the first part of this work, the performance and achievable benefits of low-complexity DSP techniques aiming to mitigate fiber Kerr nonlinear effects are investigated. Besides nonlinear compensation techniques, the use of multi-level modulation formats beyond 16QAM and high symbol rate channels have gained momentum to increase the system spectral efficiency. One of the major breakthroughs in the recent years, has been the introduction of QAM-based probabilistic constellation shaping (PCS-QAM), which has proven to outperform regular QAM formats. In this sense, in the second part of this work, the practical achievable rate increase brought by PCS-QAM for transoceanic distances is investigated. A theoretical and experimental comparison with other high-capacity formats is performed, and the design of a PCS-QAM for trans-Pacific distances is addressed. Finally, in the last section, several transmission records using the two above techniques are reported