This thesis is about the researching for 5th generation (5G) communication system, which focus on the improvement of 3D beamforming technology in the antenna array using in the Full Dimension Multiple-Input Multiple-Output (FD-MIMO) system and Millimeter-wave (mm-wave) system. When the 3D beamforming technology has been used in 5G communication system, the beam needs a weighting matrix to direct the beam to cover the UEs, but some compromises should be considered. If the narrow beams are used to transmit signals, then more energy is focused in the desired direction, but this has a restricted coverage area to a single or few User Equipments (UEs). If the BS covers multiple UEs, then multiple beams need to be steered towards more groups of UEs, but there is more interference between these beams from their side lobes when they are transmitted at same time. These challenges are waiting to be solved, which are about interference between each beam when the 3D beamforming technology is used. Therefore, there needs to be one method to decrease the generated interference between each beam through directing the side lobe beams and nulls to minimize interference in the 3D beamforming system. Simultaneously, energy needs to be directed towards the desired direction. If it has been decided that one beam should covera cluster of UEs, then there will be a range of received Signal to Interference plus Noise Ratio (SINR) depending on the location of the UEs relative to the direction of the main beam. If the beam is directed towards a group of UEs then there needs be a clustering method to cluster the UEs. In order to cover multiple UEs, an improved K-means clustering algorithm is used to cluster the multiple UEs into different groups, which is based on the cosine distance. Itcan decrease the number of beams when multiple UEs need be covered by multiple beams at same time. Moreover, a new method has been developed to calculate the weighting matrix for beamforming. It can adjust the values of weighting matrix according to the UEs' location and direct the main beam in a desired direction whilst minimizing its side lobes in other undesired directions. Then the minimum side lobe beamforming system only needs to know the UEs' location and can be used to estimate the Channel State Information (CSI) of UEs. Therefore, the scheme also shows lower complexity when compared to the beamforming methods with pre-coding. In order to test the improved K-means clustering algorithm and the new weighting method that can enhance the performance for 3D beamforming system, the two simulation systems are simulated to show the results such as 3D beamforming LTE system and mm-wave system.

Multiple-input-multiple-output (MIMO) systems of current LTE releases are capable of adaptation in the azimuth only. More recently, the trend is to enhance the system performance by exploiting the channel's degrees of freedom in the elevation through the dynamic adaptation of the vertical antenna beam pattern. This necessitates the derivation and characterization of three-dimensional (3D) channels. Over the years, channel models have evolved to address the challenges of wireless communication technologies. In parallel to theoretical studies on channel modeling, many standardized channel models like COST-based models, 3GPP SCM, WINNER, ITU have emerged that act as references for industries and telecommunication companies to assess system-level and link-level performances of advanced signal processing techniques over real-like channels. Given the existing channels are only two dimensional (2D) in nature; a large effort in channel modeling is needed to study the impact of the channel component in the elevation direction. The first part of this work sheds light on the current 3GPP activity around 3D channel modeling and beamforming, an aspect that to our knowledge has not been extensively covered by a research publication. The standardized MIMO channel model is presented, that incorporates both the propagation effects of the environment and the radio effects of the antennas. In order to facilitate future studies on the use of 3D beamforming, the main features of the proposed 3D channel model are discussed. A brief overview of the future 3GPP 3D channel model being outlined for the next generation of wireless networks is also provided. In the subsequent part of this work, we present an information-theoretic channel model for MIMO systems that supports the elevation dimension. The model is based on the principle of maximum entropy, which enables us to determine the distribution of the channel matrix consistent with the prior information on the angles of departure and angles of arrival of the propagation paths. Based on this model, an analytical expression for the cumulative density function (CDF) of the mutual information (MI) for systems with a single receive and finite number of transmit antennas in the general signal-to-interference-plus-noise-ratio (SINR) regime is provided. The result is extended to systems with multiple receive antennas in the low SINR regime. A Gaussian approximation to the asymptotic behavior of the MI distribution is derived for the large number of transmit antennas and paths regime. Simulation results study the performance gains realizable through meticulous selection of the transmit antenna down tilt angles, confirming the potential of elevation beamforming to enhance system performance. The results validate the proposed analytical expressions and elucidate the dependence of system performance on azimuth and elevation angular spreads and antenna patterns. We believe that the derived expressions will help evaluate the performance of 3D 5G massive MIMO systems in the future.