Optical fibre communication using the Nonlinear Fourier transform (NFT) is one of the potential solutions to tackle the so-called capacity crunch problem in long-haul optical fibre networks. The NFT transforms the nonlinear propagation of temporal signal, governed by the nonlinear Schr ̈odinger equation (NLSE), into simple linear evolutions of continuous and discrete spectra in the so-called nonlinear spectral domain. These spectra and the corresponding nonlinear spectral domain, defined by the NFT, are the generalized counterparts of the linear spectrum and frequency domain defined by the ordinary Fourier transform. Using the NFT, the optical fibre channel is effectively linearised, and the basic idea is to utilize degrees of freedom in the nonlinear spectral domain for data transmission. However, many aspects of this concept require rigorous investigation due to complexity and infancy of the approach. In this thesis, the aim is to provide a comprehensive investigation of data transmission over mainly the continues spectrum (CS) and partly over of the discrete spectrum (DS) of nonlinear optical fibres. First, an optical fibre communication system is defined, in which solely the CS carries the information. A noise model in the nonlinear spectral domain is derived for such a system by asymptotic analysis as well as extensive simulations for different scenarios of practical interest. It is demonstrated that the noise added to the signal in CS is severely signal-dependent such that the effective signalling space is limited. The variance normalizing transform (VNT) is used to mathematically verify the limits of signalling spaces and also estimate the channel capacity. The numerical results predict a remarkable capacity for signalling only on the CS (e.g., 6 bits/symbol for a 2000-km link), yet it is demonstrated that the capacity saturates at high power. Next, the broadening effect of chromatic dispersion is analysed, and it is confirmed that some system parameters, such as symbol rate in the nonlinear spectral domain, can be optimized so that the required temporal guard interval between the subsequently transmitted data packets is minimized, and thus the effective data rate is significantly enhanced. Furthermore, three modified signalling techniques are proposed and analysed based on the particular statistics of the noise added to the CS. All proposed methods display improved performance in terms of error rate and reach distance. For instance, using one of the proposed techniques and optimized parameters, a 7100-km distance can be reached by signalling on the CS at a rate of 9.6 Gbps. Furthermore, the impact of polarization mode dispersion (PMD) is examined for the first time, as an inevitable impairment in long-haul optical fibre links. By semi-analytical and numerical investigation, it is demonstrated that the PMD affects the CS by causing signal-dependent phase shift and noise-like errors. It is also verified that the noise is still the dominant cause of performance degradation, yet the effect of PMD should not be neglected in the analysis of NFT-based systems. Finally, the capacity of soliton communication with amplitude modulation (part of the degrees of freedom of DS) is also estimated using VNT. For the first time, the practical constraints, such as the restricted signalling space due to limited bandwidth, are included in this capacity analysis. Furthermore, the achievable data rates are estimated by considering an appropriately defined guard time between soliton pulses. Moreover, the possibility of transmitting data on DS accompanied by an independent CS signalling is also validated, which confirms the potentials of the NFT approach for combating the capacity crunch.

La capacité ultime du canal dans les systèmes de transmission optique à longue distance est limitée par les effets non linéaires liés à la propagation dans les fibres optiques. Des techniques de compensation des effets non-linéaires, tel que la DBP (Digital Back Propagation), ont été proposées pour surmonter ces limitations et accroître la capacité. Compte tenu de leur complexité d’implémentation, leur gain en performance reste très limité. Cela a déclenché très récemment la recherche de nouvelles techniques de communication prenant en compte la non-linéarité de la fibre. Une nouvelle méthode de communication en régime non-linéaire, basée sur la théorie de la transformation spectrale inverse (IST pour Inverse Spectral Transform), a été proposée pour surmonter la limitation induite par ces effets. Cette méthode, proposée à l'origine par Hasegawa en 1993, encore appelée communication aux valeurs propres (ou multi-solitons), est basée sur l'observation fondamentale selon laquelle le spectre non linéaire d'un signal optique est invariant (à l'exception d'un déphasage linéaire trivial) lors de la propagation dans la fibre optique, comme décrit par l’équation non linéaire de Schrödinger (NLSE pour Non-Linear Schrödinger Equation). Cela signifie que si la transformée spectrale directe (DST) (également appelée NFT pour Nonlinear Fourier Transform) du signal reçu peut être calculée, le spectre de valeurs propres peut être entièrement récupéré.Cette thèse porte sur une technique de communication de type NFT connue sous le nom de multiplexage non linéaire en fréquence (NFDM pour Non-Linear Fourier Transform). Différentes configurations de systèmes optiques NFDM ont été évalués numériquement et validés expérimentalement. Dans un premier temps, la structure d’un système NFDM en mono-polarisation utilisant le spectre continu des fréquences non-linéaires dans une fibre en régime de dispersion normale est décrite. Pour ce faire, une forme NFT du vecteur NLSE, encore appelé système de Manakov, a été développé numériquement. Sur la base de ces algorithmes, la méthode NFDM a été étendue aux systèmes multiplexés par division de polarisation (PMD) et validée expérimentalement pour la première fois en utilisant le spectre continu. Finalement, l’expérience a été répliquée en régime de dispersion anormale. Afin d'étudier les contraintes de mise en œuvre, des études numériques supplémentaires ont été effectués pour la transmission de signaux NFDM utilisant la modulation du spectre continu. / Nonlinear effects in optical fiber set the ultimate limit to the channel capacity in long-haul optical transmission systems. Advanced nonlinear compensation techniques such as digital backpropagation (DBP) have been proposed as a solution to overcome the channel capacity crunch. However, given theircomputational complexity, in a practical environment their performance gainremains very limited. This triggered a search for a novel communication system design that takes fiber nonlinearity into consideration. A new nonlinearcommunication method, based on the theory of the inverse spectral transform, has been proposed to overcome the nonlinear capacity crunch. Thismethod, originally proposed by Hasegawa in 1993 and called eigenvalue (ormulti-soliton) communication, is based on the fundamental observation thatthe nonlinear spectrum of an optical signal is invariant (except for a triviallinear phase shift) upon propagation in the fiber channel, as described bythe nonlinear Schrödinger equation (NLSE). This means that if the directspectral transform (also known as nonlinear Fourier transform (NFT)) ofthe received signal can be computed, the eigenvalue spectrum can be fullyrecovered.This thesis focuses on a NFT-based communication technique known as nonlinear frequency division multiplexing (NFDM). The NFDM optical systemis numerically assessed and experimentally demonstrated. First, the structure of the proposed single-polarization NFDM system using the continuousspectrum in the normal dispersion regime is presented. To that end, theNFT of the vector NLSE, or Manakov system, was numerically developed.Based on these algorithms the NFDM method was extended to polarizationdivision multiplexed (PMD) systems, and experimentally validated for thefirst time using the continuous spectrum. Finally, the experiment will bereplicated in the anomalous dispersion regime.Additional numerical studies are presented, in order to investigate the implementation challenges of the proposed NFDM techniques for the continuousspectrum modulation.