To meet the exponentially increasing traffic data driven by the rapidly growing mobile subscriptions, both industry and academia are exploring the potential of a new genera- tion (5G) of wireless technologies. An important 5G goal is to achieve high data rate. Small cells with spectrum sharing and multiple-input multiple-output (MIMO) techniques are one of the most promising 5G technologies, since it enables to increase the aggregate data rate by improving the spectral efficiency, nodes density and transmission bandwidth, respectively. However, the increased interference in the densified networks will in return limit the achievable rate performance if not properly managed. The considered setup can be modeled as MIMO interference networks, which can be classified into the K-user MIMO interference channel (IC) and the K-cell MIMO interfering broadcast channel/multiple access channel (MIMO-IBC/IMAC) according to the number of mobile stations (MSs) simultaneously served by each base station (BS). The thesis considers two physical layer (PHY) resource allocation problems that deal with the interference for both models: 1) Pareto boundary computation for the achiev- able rate region in a K-user single-stream MIMO IC and 2) grouping-based interference alignment (GIA) with optimized IA-Cell assignment in a MIMO-IMAC under limited feedback. In each problem, the thesis seeks to provide a deeper understanding of the system and novel mathematical results, along with supporting numerical examples. Some of the main contributions can be summarized as follows. It is an open problem to compute the Pareto boundary of the achievable rate region for a K-user single-stream MIMO IC. The K-user single-stream MIMO IC models multiple transmitter-receiver pairs which operate over the same spectrum simultaneously. Each transmitter and each receiver is equipped with multiple antennas, and a single desired data stream is communicated in each transmitter-receiver link. The individual achievable rates of the K users form a K-dimensional achievable rate region. To find efficient operating points in the achievable rate region, the Pareto boundary computation problem, which can be formulated as a multi-objective optimization problem, needs to be solved. The thesis transforms the multi-objective optimization problem to two single-objective optimization problems–single constraint rate maximization problem and alternating rate profile optimization problem, based on the formulations of the ε-constraint optimization and the weighted Chebyshev optimization, respectively. The thesis proposes two alternating optimization algorithms to solve both single-objective optimization problems. The convergence of both algorithms is guaranteed. Also, a heuristic initialization scheme is provided for each algorithm to achieve a high-quality solution. By varying the weights in each single-objective optimization problem, numerical results show that both algorithms provide an inner bound very close to the Pareto boundary. Furthermore, the thesis also computes some key points exactly on the Pareto boundary in closed-form. A framework for interference alignment (IA) under limited feedback is proposed for a MIMO-IMAC. The MIMO-IMAC well matches the uplink scenario in cellular system, where multiple cells share their spectrum and operate simultaneously. In each cell, a BS receives the desired signals from multiple MSs within its own cell and each BS and each MS is equipped with multi-antenna. By allowing the inter-cell coordination, the thesis develops a distributed IA framework under limited feedback from three aspects: the GIA, the IA-Cell assignment and dynamic feedback bit allocation (DBA), respec- tively. Firstly, the thesis provides a complete study along with some new improvements of the GIA, which enables to compute the exact IA precoders in closed-form, based on local channel state information at the receiver (CSIR). Secondly, the concept of IA-Cell assignment is introduced and its effect on the achievable rate and degrees of freedom (DoF) performance is analyzed. Two distributed matching approaches and one centralized assignment approach are proposed to find a good IA-Cell assignment in three scenrios with different backhaul overhead. Thirdly, under limited feedback, the thesis derives an upper bound of the residual interference to noise ratio (RINR), formulates and solves a corresponding DBA problem. Finally, numerical results show that the proposed GIA with optimized IA-Cell assignment and the DBA greatly outperforms the traditional GIA algorithm.

To meet the exponentially increasing traffic data driven by the rapidly growing mobile subscriptions, both industry and academia are exploring the potential of a new genera- tion (5G) of wireless technologies. An important 5G goal is to achieve high data rate. Small cells with spectrum sharing and multiple-input multiple-output (MIMO) techniques are one of the most promising 5G technologies, since it enables to increase the aggregate data rate by improving the spectral efficiency, nodes density and transmission bandwidth, respectively. However, the increased interference in the densified networks will in return limit the achievable rate performance if not properly managed. The considered setup can be modeled as MIMO interference networks, which can be classified into the K-user MIMO interference channel (IC) and the K-cell MIMO interfering broadcast channel/multiple access channel (MIMO-IBC/IMAC) according to the number of mobile stations (MSs) simultaneously served by each base station (BS). The thesis considers two physical layer (PHY) resource allocation problems that deal with the interference for both models: 1) Pareto boundary computation for the achiev- able rate region in a K-user single-stream MIMO IC and 2) grouping-based interference alignment (GIA) with optimized IA-Cell assignment in a MIMO-IMAC under limited feedback. In each problem, the thesis seeks to provide a deeper understanding of the system and novel mathematical results, along with supporting numerical examples. Some of the main contributions can be summarized as follows. It is an open problem to compute the Pareto boundary of the achievable rate region for a K-user single-stream MIMO IC. The K-user single-stream MIMO IC models multiple transmitter-receiver pairs which operate over the same spectrum simultaneously. Each transmitter and each receiver is equipped with multiple antennas, and a single desired data stream is communicated in each transmitter-receiver link. The individual achievable rates of the K users form a K-dimensional achievable rate region. To find efficient operating points in the achievable rate region, the Pareto boundary computation problem, which can be formulated as a multi-objective optimization problem, needs to be solved. The thesis transforms the multi-objective optimization problem to two single-objective optimization problems–single constraint rate maximization problem and alternating rate profile optimization problem, based on the formulations of the ε-constraint optimization and the weighted Chebyshev optimization, respectively. The thesis proposes two alternating optimization algorithms to solve both single-objective optimization problems. The convergence of both algorithms is guaranteed. Also, a heuristic initialization scheme is provided for each algorithm to achieve a high-quality solution. By varying the weights in each single-objective optimization problem, numerical results show that both algorithms provide an inner bound very close to the Pareto boundary. Furthermore, the thesis also computes some key points exactly on the Pareto boundary in closed-form. A framework for interference alignment (IA) under limited feedback is proposed for a MIMO-IMAC. The MIMO-IMAC well matches the uplink scenario in cellular system, where multiple cells share their spectrum and operate simultaneously. In each cell, a BS receives the desired signals from multiple MSs within its own cell and each BS and each MS is equipped with multi-antenna. By allowing the inter-cell coordination, the thesis develops a distributed IA framework under limited feedback from three aspects: the GIA, the IA-Cell assignment and dynamic feedback bit allocation (DBA), respec- tively. Firstly, the thesis provides a complete study along with some new improvements of the GIA, which enables to compute the exact IA precoders in closed-form, based on local channel state information at the receiver (CSIR). Secondly, the concept of IA-Cell assignment is introduced and its effect on the achievable rate and degrees of freedom (DoF) performance is analyzed. Two distributed matching approaches and one centralized assignment approach are proposed to find a good IA-Cell assignment in three scenrios with different backhaul overhead. Thirdly, under limited feedback, the thesis derives an upper bound of the residual interference to noise ratio (RINR), formulates and solves a corresponding DBA problem. Finally, numerical results show that the proposed GIA with optimized IA-Cell assignment and the DBA greatly outperforms the traditional GIA algorithm.