Cooperative relaying techniques have recently received significant interests from both academia and industry due to their ability to provide spatial diversity to address the ever increasing demand for extended network coverage, higher data rates without sacrificing extra power resources, greater mobility and enhanced reliability. This thesis mainly considers two themes. Firstly, in the context of self-powered multiple-input-multiple-output (MIMO) full-duplex (FD) relaying, our research focuses on design and performance analysis of MIMO FD relaying systems in the presence of practical transmission impairments. Namely, the impact of spatial fading correlation, imperfect channel state information (CSI), loopback self-interference (LI), and co-channel interference (CCI) on the system performance are investigated. Secondly, in the context of wirelessly-powered MIMO HD relaying, our research focuses on energy beamforming which is used to maximize the overall harvested energy so as to enable longer-distance wireless power transfer when compared to the single antenna nodes. Namely, in the presence of MIMO relaying systems, hop-by-hop information and energy beamforming is proposed where the transmitted signal is steered along the strongest eigenmode of each hop. The wirelessly powered relay scavenge energy from the source information radio-frequency (RF) signal through energy beamforming, where both the time-switching receiver (TSR) and power-splitting receiver (PSR) are considered, then uses the harvested energy to forward the source message to the destination. Our research focuses on developing a comprehensive analytical framework for deriving new closed-form expressions for the outage probability and ergodic capacity for amplify-and-forward (AF) relaying systems, including simpler tight bounds and asymptotic high signal-to-noise (SNR) ratio analysis. First, the optimization problem for the design of source, relay, and destination precoding and/or decoding weight vectors which maximizes the overall signal-to-interference-plus-noise ratio (SINR) is formulated. Then, in order to get closed-form precoding and decoding weight vectors, a sub-optimal solution based on null space projection designed to completely suppress the LI and/or CCI is proposed, through which a closed-form overall SINR is presented. Simulation results show the exactness and tightness of the proposed exact and bound analytical expressions, respectively.

In our thesis work, we analyze the performance analysis of a power domain NonOrthogonal Multiple Access (NOMA) system in which the closer user acts as fullduplex relaying to forward the signal to farther user. Because Nakagami-m distribution is a generalized case including the two common fading distributions as specialcases: Rayleigh distribution (m=1), Rician distribution (m>1). We assume that thesystem experiences Nakagami-m fading. Then, we have to analyze outage probabilityof NOMA system. Numerical results are provided for outage probability to show theeffect of system parameters on the performance of the NOMA system in full duplexrelaying over Nakagami-m fading.