Today’s communication systems are in need of spectrally efficient and high throughput techniques more than ever because of high data rate applications and the scarcity and expense of bandwidth. To cope with increased data rate demands, more base stations are needed which is not cost and energy efficient in cellular networks. It has been shown that wireless relay networks can provide higher network throughput and increase power efficiency with low complexity and cost. Furthermore, network resources can be utilized more efficiently by using network coding in relay networks. A wireless relay network in which multiple nodes exchange information with the help of relay node(s) is called a multi-way relay channel (MWRC). MWRCs are expected to be an integral part of next generation wireless standards. The main focus of this dissertation is the investigation of transmission schemes in an MWRC to improve the throughput and error performance. An MWRC with full data exchange is assumed in which a half-duplex relay station (RS) is the enabler of communication. One of the challenges with signal demodulation in MWRCs is the existence of ambiguous points in the received constellation. The first part of this dissertation investigates a transmission scheme for full data exchange in MWRC that benefits from these points and improves its throughput by 33% compared to traditional relaying. Then an MWRC is considered where a RS assists multiple nodes to exchange messages. A different approach is taken to avoid ambiguous points in the superposition of user symbols at the relay. This can be achieved by employing complex field network coding (CFNC) which results in full data exchange in two communication phases. CFNC may lead to small Euclidean distances between constellation points, resulting in poor error performance. To improve this performance, the optimal user precoding values are derived such that the power efficiency of the relay constellation is highest when channel state information is available at the users. The error performance of each user is then analyzed and compared with other relaying schemes. Finally, focusing on the uplink of multi-way relay systems, the performance of an MWRC is studied in which users can employ arbitrary modulation schemes and the links between the users and the relay have different gains, e.g. Rayleigh fading. Analytical expressions for the exact average pairwise error probability of these MWRCs are derived. The probability density function (PDF) and the mean of the minimum Euclidean distance of the relay constellation are closely approximated, and a tight upper bound on the symbol error probability is developed. / Graduate