Fiber-optic systems represent the backbone of the communication networks, carrying most of the world’s data traffic. The main bottleneck in today’s fiber-optic communication systems has roots in the inherent nonlinearity of the fiber. Hence, developing new transmission schemes that are compatible with the nonlinear behavior of the optical fiber has become necessary. To utilize the full transmission capacity of an optical fiber, this thesis investigates two different techniques---compensation-based method and nonlinear Fourier transform (NFT). For the purpose of suppressing the nonlinear distortion in real time, an optical back propagation (OBP) technique using Raman pumped dispersion compensating fibers (DCF). OBP, as an all-optical signal processing technique, can compensate for both intra- and inter-channel nonlinear impairments in real time in point-to-point systems as well as in optical networks. The proposed inline OBP module consists of an optical phase conjugator (OPC), amplifiers and a Raman pumped DCF. In order to suppress the nonlinear effects of the transmission fiber, the power in the OBP fiber should increase exponentially with distance. This can be approximately achieved by using Raman pumping of the backpropagation fiber. Simulation results show that this technique provides 7.7 dB performance improvement in Q-factor over conventional systems. The second part of this thesis is dedicated to the NFT as a promising framework to exploit the inherent nonlinearity of optical fiber rather than treating it as an undesirable effect and using perturbation and approximation-based methods to mitigate it. A novel multistage perturbation technique to realize the NFT as a cascade of linear discrete Fourier transforms is developed. The linear Fourier transform can be easily implemented in the optical domain using a time lens or discrete photonic components, which can be implemented in silicon photonics. The proposed technique provides a promising way to implement NFT in the optical domain, which will fully utilize the potential of NFT for wavelength-division multiplexed fiber-optic systems in the optical domain. Moreover, a nonlinear frequency-division multiplexed (NFDM) transmission scheme with midpoint OPC is investigated. The proposed mid-OPC NFDM system offers a degree of freedom to have a flexible power normalization factor, P_n to minimize the signal-noise mixing in NFT processing for a specific launch power, resulting in significant system performance improvement up to 5.6 dB in Q-factor over conventional NFDM systems. Another advantage of the proposed scheme is that the mid-OPC NFDM system extends the transmission reach without having to increase the guard interval, which leads to higher spectral efficiency. / Thesis / Doctor of Philosophy (PhD)