Fiber-optic networks are continually evolving to accommodate the ever increasing data rates demanded by modern applications and devices. The current state-of-the art systems are being deployed with 100 Gb/s rates per wavelength while maintaining the 50 GHz channel spacing established for 10 Gb/s dense wavelength division multiplexed (DWDM) systems. Phase modulation formats (in particular quadrature phase shift keying - QPSK) are necessary to meet the spectral efficiency (SE) requirements of the application. The main challenge for phase modulated optical systems is fiber nonlinearities, where changes in intensity of the combined optical signal result in changes to the fiber's refractive index. Limiting launch power is the primary means to avoid dramatic intensity fluctuations, a strategy which in turn limits the available signal-to-noise ratio (SNR) within the channel. Continuous phase modulation (CPM) is a format in which data is encoded in the phase, while the amplitude is constant throughout all transmission (even during transitions between symbols). With the goal of reducing the impact of nonlinearities, the purpose of this research was to identify a set of CPM signals best suited for high speed fiber-optic transmission, and quantify their performance against other formats. The secondary goal was to identify techniques appropriate for demodulation of high speed fiber-optic systems and implement them for simulation and experimental research. CPM encompasses a number of variable parameters that combine to form an infinite number of unique schemes, each of which is characterized by its own SE, minimum distance, and implementation complexity. A method for computing minimum distance of DWDM-filtered CPM formats is presented and utilized to narrow down to a range of candidate schemes. A novel transmitter design is presented for CPM signal generation, as well as a number of novel reception techniques to achieve proper demodulation of the CPM signal from the coherent optical receiver. Using these methods, the identified range of candidate schemes was compared in simulation to the conventional QPSK format, showing that some modest gain can be expected from CPM. Through these and other simulations, it is revealed that fiber nonlinearities depend on the aggregate sum of all wavelengths rather than the imposition of each separate carrier on its neighbors. Therefore the constant envelope of CPM does not directly impact the nonlinearities since multiple carriers will photonically interfere and result in intensity fluctuations regardless of modulation format. Additionally, dispersive effects in fiber decompose the underlying channels so that the intensity throughout propagation is nearly Gaussian distributed, regardless of format. The benefits gained from CPM are thus limited to schemes that attain a higher minimum distance than alternative formats (in the given channel passband), and for optically compensated links in which low dispersion is maintained throughout the fiber link.

Les systèmes de communications optiques jouent un role important pour satisfaire la demande incessante de trafics de données. Cette demande, induite par des applications gourmandes en termes de bande passante et débit, necéssite une augmentation de la capacité des réseaux optiques d’accès et par conséquent une augmentation des capacités de réseaux de transports métropolitains et longues distances. La prochaine génération de systèmes WDM longue distance devrait opérée à des débits de 400Gbps ou 1Tbps. Cette montée en débit s’appuiera sur des nouvelles formes d’ondes avancées de type mono-porteuse (Nyquist-WDM) ou multi-porteuse (OFDM multi-bande). Ces approches sont basées sur le multipléxage de plusieurs porteuses espacées par des intervalles de garde réduits. D’autre part, pour générer ces très haut débits, des modulations multi-états sont utilisées pour chaque porteuse grâce à leur efficacité spectrale élevée. Ces types de systèmes, qui combinent à la fois les approches multi-bande et les modulations multi-états, sont extrêmement vulnérables aux effets nonlinéaires de la fibre optique. En fait, les effets nonlinéaires sont dépendants de la puissance de transmission et inversement proportionels à l’intervalle de garde. Cela rend leur compensation indispensable pour maintenir des bonnes performances des systèmes en terme de distance de transmission. Grâce à l’emploi de récepteurs à détection cohérente, des techniques de traitement du signal numérique sont utlisées pour combattre les effets nonlinéaires. Dans cette thèse, nous avons proposé des nouvelles techniques basées sur les séries de Volterra et les égaliseurs à retour de decision pour compenser respectivement les effets nonlinéaires intrabande et les interférences nonlinéaires inter-bande. / Optical communication systems have evolved since their deployment to meet the growing demand for high-speed communications. Over the past decades, the global demand for communication capacity has increased exponentially and the most of the growth has occurred in the last few years when data started dominating network traffic. In order to meet the increase of traffic demands fueled by the growth of internet services, an increase of access network capacity and consequently metro and long-haul network capacities is required. Next generation of long-haul WDM transmission systems is expected to operate at 400Gbps or 1Tbps bit rate. Superchannel approaches, such as Nyquist WDM and multi-band OFDM, allow both high spectral efficiency and small guardband which makes them promising candidates to generate these high bit rates in combination with multi-level modulations formats. Such transmission systems are strongly disturbed by fiber nonlinear effects which increase with the data rate and the small guard band. Therefore, fiber nonlinearities compensation is required to get the desired performance in terms of transmission reach. DSP based approaches such as digital back propagation and third-order Volterra based nonlinear equalizer have been already proposed to deal with intra-channel or intra-band nonlinear effects. In the context of superchannel systems, we have proposed two new compensation techniques to deal with fiber nonlinear effects. The first one, called fifth-order inverse Volterra based nonlinear equalizer, compensate for intra-band nonlinear effects. The second approach, which is the interband/ subcarrier nonlinear interference canceler, is proposed to combat the nonlinear interference insuperchannel systems.