Optical fibers play a vital role in modern telecommunication systems and networks. An optical fiber link imposes some linear and nonlinear distortions on the propagating light-wave signal due to the inherent dispersive nature and nonlinear behavior of the fiber. These distortions impede the increasing demand for higher data rate transmission over longer distances. Developing efficient and computationally non-expensive digital signal processing (DSP) techniques to effectively compensate for the fiber impairments is therefore essential and of preeminent importance. This thesis proposes two DSP-based approaches for mitigating the induced distortions in short-reach and long-haul fiber-optic communication systems. The first approach introduces a powerful digital nonlinear feed-forward equalizer (NFFE), exploiting multilayer artificial neural network (ANN). The proposed ANN-NFFE mitigates nonlinear impairments of short-haul optical fiber communication systems, arising due to the nonlinearity introduced by direct photo-detection. In a direct detection system, the detection process is nonlinear due to the fact that the photo-current is proportional to the absolute square of the electric field intensity. The proposed equalizer provides the most efficient computational cost with high equalization performance. Its performance is comparable to the benchmark compensation performance achieved by maximum-likelihood sequence estimator. The equalizer trains an ANN to act as a nonlinear filter whose impulse response removes the intersymbol interference (ISI) distortions of the optical channel. Owing to the proposed extensive training of the equalizer, it achieves the ultimate performance limit of any feed-forward equalizer. The performance and efficiency of the equalizer are investigated by applying it to various practical short-reach fiber-optic transmission system scenarios. These scenarios are extracted from practical metro/media access networks and data center applications. The obtained results show that the ANN-NFFE compensates for the received BER degradation and significantly increases the tolerance to the chromatic dispersion distortion. The second approach is devoted for blindly combating impairments of long-haul fiber-optic systems and networks. A novel adjoint sensitivity analysis (ASA) approach for the nonlinear Schrödinger equation (NLSE) is proposed. The NLSE describes the light-wave propagation in optical fiber communication systems. The proposed ASA approach significantly accelerates the sensitivity calculations in any fiber-optic design problem. Using only one extra adjoint system simulation, all the sensitivities of a general objective function with respect to all fiber design parameters are estimated. We provide a full description of the solution to the derived adjoint problem. The accuracy and efficiency of our proposed algorithm are investigated through a comparison with the accurate but computationally expensive central finite-differences (CFD) approach. Numerical simulation results show that the proposed ASA algorithm has the same accuracy as the CFD approach but with a much lower computational cost. Moreover, we propose an efficient, robust, and accelerated adaptive digital back propagation (A-DBP) method based on adjoint optimization technique. Provided that the total transmission distance is known, the proposed A-DBP algorithm blindly compensates for the linear and nonlinear distortions of point-to-point long-reach optical fiber transmission systems or multi-point optical fiber transmission networks, without knowing the launch power and channel parameters. The NLSE-based ASA approach is extended for the sensitivity analysis of general multi-span DBP model. A modified split-step Fourier scheme method is introduced to solve the adjoint problem, and a complete analysis of its computational complexity is studied. An adjoint-based optimization (ABO) technique is introduced to significantly accelerate the parameters extraction of the A-DBP. The ABO algorithm utilizes a sequential quadratic programming (SQP) technique coupled with the extended ASA algorithm to rapidly solve the A-DBP training problem and optimize the design parameters using minimum overhead of extra system simulations. Regardless of the number of A-DBP design parameters, the derivatives of the training objective function with respect to all parameters are estimated using only one extra adjoint system simulation per optimization iterate. This is contrasted with the traditional finite-difference (FD)-based optimization methods whose sensitivity analysis calculations cost per iterate scales linearly with the number of parameters. The robustness, performance, and efficiency of the proposed A-DBP algorithm are demonstrated through applying it to mitigate the distortions of a 4-span optical fiber communication system scenario. Our results show that the proposed A-DBP achieves the optimal compensation performance obtained using an ideal fine-mesh DBP scheme utilizing the correct channel parameters. Compared to A-DBPs trained using SQP algorithms based on forward, backward, and central FD approaches, the proposed ABO algorithm trains the A-DBP with 2.02 times faster than the backward/forward FD-based optimizers, and with 3.63 times faster than the more accurate CFD-based optimizer. The achieved gain further increases as the number of design parameters increases. A coarse-mesh A-DBP with less number of spans is also adopted to significantly reduce the computational complexity, achieving compensation performance higher than that obtained using the coarse-mesh DBP with full number of spans. / Thesis / Doctor of Philosophy (PhD) / This thesis proposes two powerful and computationally efficient digital signal processing (DSP)-based techniques, namely, artificial neural network nonlinear feed forward equalizer (ANN-NFFE) and adaptive digital back propagation (A-DBP) equalizer, for mitigating the induced distortions in short-reach and long-haul fiber-optic communication systems, respectively. The ANN-NFFE combats nonlinear impairments of direct-detected short-haul optical fiber communication systems, achieving compensation performance comparable to the benchmark performance obtained using maximum-likelihood sequence estimator with much lower computational cost. A novel adjoint sensitivity analysis (ASA) approach is proposed to significantly accelerate sensitivity analyses of fiber-optic design problems. The A-DBP exploits a gradient-based optimization method coupled with the ASA algorithm to blindly compensate for the distortions of coherent-detected fiber-optic communication systems and networks, utilizing the minimum possible overhead of performed system simulations. The robustness and efficiency of the proposed equalizers are demonstrated using numerical simulations of varied examples extracted from practical optical fiber communication systems scenarios.