The ever-increasing data traffic demand drives the evolution of telecommunication networks, including the last-mile access networks as well as the long-haul backbone networks. This Ph.D. dissertation focuses on system design and signal processing techniques for next-generation converged optical-wireless access systems and the high-speed long-haul coherent optical communication systems. The convergence of high-speed millimeter-wave wireless communications and high-capacity fiber-optic backhaul networks provides tremendous potential to meet the capacity requirements of future access networks. In this work, a cloud-radio-over-fiber access architecture is proposed. The proposed architecture enables a large-scale small-cell system to be deployed in a cost-effective, power-efficient, and flexible way. Based on the proposed architecture, a multi-service reconfigurable small-cell backhaul network is developed and demonstrated experimentally. Additionally, the combination of high-speed millimeter-wave radio and fiber-optic backhaul is investigated. Several novel methods that enable high-spectral-efficient vector signal transmission in millimeter-wave radio-over-fiber systems are proposed and demonstrated through both theoretical analysis and experimental verification. For long-haul core networks, ultra-high-speed optical communication systems which can support 1Terabit/s per channel transmission will soon be required to meet the increasing capacity demand in the core networks. Grouping a number of tightly spaced optical subcarriers to form a terabit superchannel has been considered as a promising solution to increases channel capacity while minimizing the need for high-level modulation formats and high baud rate. Conventionally, precise spectral control at transmitter side is required to avoid strong inter-channel interference (ICI) at tight channel spacing. In this work, a novel receiver-side approach based on “super receiver” architecture is proposed and demonstrated. By jointly detecting and demodulating multiple channels simultaneously, the penalties associated with the limitations of generating ideal spectra can be mitigated. Several joint DSP algorithms are developed for linear ICI cancellation and joint carrier-phase recovery. Performance analysis under different system configurations is conducted to demonstrate the feasibility and robustness of the proposed joint DSP algorithms, and improved system performance is observed with both experimental and simulation data.