abstract: Both two-way relays (TWR) and full-duplex (FD) radios are spectrally efficient, and their integration shows great potential to further improve the spectral efficiency, which offers a solution to the fifth generation wireless systems. High quality channel state information (CSI) are the key components for the implementation and the performance of the FD TWR system, making channel estimation in FD TWRs crucial. The impact of channel estimation on spectral efficiency in half-duplex multiple-input-multiple-output (MIMO) TWR systems is investigated. The trade-off between training and data energy is proposed. In the case that two sources are symmetric in power and number of antennas, a closed-form for the optimal ratio of data energy to total energy is derived. It can be shown that the achievable rate is a monotonically increasing function of the data length. The asymmetric case is discussed as well. Efficient and accurate training schemes for FD TWRs are essential for profiting from the inherent spectrally efficient structures of both FD and TWRs. A novel one-block training scheme with a maximum likelihood (ML) estimator is proposed to estimate the channels between the nodes and the residual self-interference (RSI) channel simultaneously. Baseline training schemes are also considered to compare with the one-block scheme. The Cramer-Rao bounds (CRBs) of the training schemes are derived and analyzed by using the asymptotic properties of Toeplitz matrices. The benefit of estimating the RSI channel is shown analytically in terms of Fisher information. To obtain fundamental and analytic results of how the RSI affects the spectral efficiency, one-way FD relay systems are studied. Optimal training design and ML channel estimation are proposed to estimate the RSI channel. The CRBs are derived and analyzed in closed-form so that the optimal training sequence can be found via minimizing the CRB. Extensions of the training scheme to frequency-selective channels and multiple relays are also presented. Simultaneously sensing and transmission in an FD cognitive radio system with MIMO is considered. The trade-off between the transmission rate and the detection accuracy is characterized by the sum-rate of the primary and the secondary users. Different beamforming and combining schemes are proposed and compared. / Dissertation/Thesis / Doctoral Dissertation Electrical Engineering 2018