The Nyquist WDM system realizes a terabit high spectral efficiency transmission system by allocating several subcarriers close to or equal to the baud rate. This system achieves optimal performance by maintaining both temporal and spectral orthogonality. However, ISI and ICI effects are inevitable in practical Nyquist WDM implementations due to the imperfect channel response and tight channel spacing and may cause significant performance degradations. Our primary research goals are to combat the ISI effects via the transmitter digital pre-shaping and to remove the ICI impairments at the receiver using MIMO signal processing. First we propose two novel blind channel estimation techniques that enable the transmitter pre-shaping design for the ISI effects mitigation. Both numerical and experimental results demonstrate that the two methods are very effective in compensating the narrow band filtering and are very robust to channel estimation noise. Besides pre-shaping, the DAC-enabled transmitter chromatic dispersion compensation is also demonstrated in a system with high LO laser linewidth. Next a novel “super-receiver” structure is proposed, where different subchannels are synchronously sampled, and the baseband signals from three adjacent subchannels are processed jointly to remove ICI penalty. Three different ICI compensation methods are introduced and their performances compared. The important pre-processes that enable a successful ICI compensation are also elaborated. Despite ICI compensation, the joint carrier phase recovery based on the Viterbi-Viterbi algorithm is also studied in the carrier phase locked systems. In-band crosstalk arises from the imperfect switch elements in the add-drop process of ROADM-enabled DWDM systems and may cause significant performance degradation. Our third research topic is to demonstrate a systematic way to analyze and predict the in-band crosstalk-induced penalty. In this work, we propose a novel crosstalk-to-ASE noise weighting factor that can be combined with the weighted crosstalk weighting metric to incorporate the in-band crosstalk noise into the Gaussian noise model for performance prediction and analysis. With the aid of the Gaussian noise model, the in-band crosstalk-induced nonlinear noise is also studied. Both simulations and experiments are used to validate the proposed methods.