With the technological shift towards big data, internet of things (IoT), 5G applications and cloud computing, the demand for high capacity networks is dramatically increasing. To avoid congestion and saturation, content and service providers are re-designing their network (backbone, metro and data-centers interconnects) connectivity using gridless optical line systems along with programmable coherent transponders. The latter are expected to transmit data at different data rates up to 400 Gb/s. In 2008, the first coherent receiver was commercially available [1]. By means of high-speed analog to digital converters and adaptive digital signal processing (DSP) algorithms, such revolution in modern optical communication was possible. That allowed a better spectral efficiency using higher order modulation formats and further signal reach by means of compensating both linear and nonlinear impairments. Another key development was leveraging light polarization-diversity, that permits to double the data rate at the expense of receiver complexity. To further increase the capacity of fiber links, gridless DWDM networks are being developed for deployment in the next few years. The key idea is to allow variable bandwidth signals to be allocated on optical links and by performing the appropriate network layer optimization improved throughput can be achieved. These innovations are driving new types of challenges for routing and assignment methods, as well, DSP algorithms such as clock recovery and compensation of fiber non-linearity. This thesis is organized as a collection of contributions and composed of five major parts. The first part, consisting of chapters 2 and 3. Chapter 4 deals with tracking of fast state of polarization transient, i.e. dynamic aspect of optical channels, in presence of polarization dependent loss (PDL) and filtering effects due to reconfigurable optical add-drop multiplexers (ROADMs). Chapters 5 and 6 study the impact of filtering effects, quasi-static effects in optical links and transponders, represented by ROADMs in fixed-grid and Silicon Photonics (SiPh) modulators in flexible-grid networks, respectively. Chapters 7, 8 and 9, are related to clock recovery in digital coherent receivers. They cover mitigation of jitter in gridless applications, improving jitter when deploying phase interpolators (PI) and jitter injection as a test-mean to evaluate performance.