The ever-increasing demand for high date rate is beyond what is provided by the present wireless and wired access networks. Radio-over-fiber (RoF) technology which can provide broadband wireless access has been considered the most practical and efficient solution. In recent years, RoF with coherent detection has been shown to have better performance than that with direct detection in terms of receiver sensitivity and spectral efficiency. However, RoF with coherent detection suffers from phase noise introduced from both the transmitter and local oscillator (LO) laser sources, which degrades the performance significantly. This study is focused on coherent RoF links for broadband wireless access networks. The thesis consists of four parts. In the first part, a new approach to cancel the phase noise and the unstable frequency difference introduced from the transmitter and LO laser sources based on digital signal processing (DSP) in an RoF link with coherent detection is presented. The proposed schemes rival the RoF link with direct detection in complexity while maintaining a high receiver sensitivity. In addition, a high spectral efficiency coherent RoF link with phase noise cancellation, which can detect both intensity- and phase- modulated signals carried by the same optical carrier, is studied and demonstrated. In the second part, to achieve full-duplex transmission and increase the capacity of an RoF link, radio over wavelength division multiplexing passive optical network (WDM-PON) is studied. To eliminate the requirements of light sources and wavelength management at the optical network units (ONUs), which reduces the cost and eases the installation for a radio over WDM-PON system, a new approach to reuse the downstream wavelength at the ONU with coherent detection and DSP at the optical line terminal (OLT) is investigated. The performance in terms of error vector magnitude (EVM) and bit rate error (BER) is evaluated for both downlink and uplink. In the scheme, the coherent detection improves the receiver sensitivity for the uplink and compensate for the degraded data transmission performance due to the utilization of a wavelength-reused downstream optical signal. Furthermore, since the future internet traffic will become highly symmetric, a symmetrical radio over a colorless WDM passive optical network (PON) with wavelength reuse based on polarization multiplexing and coherent detection is proposed and studied. In the third part, a coherent RoF link based on optical single sideband with no optical carrier (OSSB) modulation with low-cost free-running laser sources for ultra-dense wavelength division multiplexing passive optical networks (UDWDM-PONs) is studied. In a UDWDM-RoF-PON, the channel spacing is very small, thus a WDM filter may not be able to de-multiplex the ultra-dense channels. However, through coherent detection, the channel separation can be realized by using electrical filters at the output of the coherent receiver. In addition, to utilize the spectrum in each channel more efficiently, OSSB modulation is employed. In the proposed scheme, an RoF signal based OSSB modulation with coherent detection is experimentally demonstrated. The channel spacing can be as narrow as 3 GHz. Finally, for 5th generation wireless systems (5G), multi-input and multi-output (MIMO) is a key technology which can multiple the capacity. To seamlessly integrate MIMO into RoF links, it is required that an RoF link can transmit multiple wireless signals over a single wavelength. To enable 4 × 4 MIMO, in the fourth part, an RoF link to transmit four wireless signals with an identical microwave center frequency without using frequency-division multiplexing (FDM) over a single optical wavelength based on optical independent sideband (OISB) modulation and optical orthogonal modulation incorporating optical coherent detection and digital signal processing (DSP) is studied. To increase the spectral efficiency further, a novel high spectral efficiency (20.62 bit/s/Hz) RoF link based on coherent detection and DSP with the spectral efficiency improved by employing both intensity and phase modulation and polarization multiplexing to transmit four microwave signals over a single optical carrier is investigated.