Conventional multicell wireless systems operate with out-of-cell interference treated as background noise; consequently, their performance faces two major limitations: 1)Signal processing is performed on a per-cell basis; and 2)Intercell interference detection is infeasible as intercell interference, although significantly above the noise level, is typically quite weak. In this thesis, we consider a multicell downlink scenario, where base-stations are equipped with multiple transmit antennas, the remote users are equipped with a single antenna, and multiple remote users are active simultaneously via spatial division multiplexing. We propose solutions for the above limitations by considering techniques for mitigating interference. The first part of the thesis proposes solutions for the first limitation. It considers the benefit of coordinating base-stations across multiple cells, where multiple base-stations may jointly optimize their respective beamformers to improve the overall system performance. It focuses on the design criteria of minimizing either the total weighted transmitted power or the maximum per-antenna power across the base-stations subject to signal-to-interference-and-noise-ratio (SINR) constraints at the remote users. The main contribution of this part is an efficient algorithm for finding the joint globally optimal beamformers across all base-stations. The proposed algorithm is based on a generalization of uplink-downlink duality to the multicell setting using the Lagrangian duality theory. An important feature is that it naturally leads to a distributed implementation in time-division duplex (TDD) systems. Simulation results suggest that coordinating the beamforming vectors alone already provides appreciable performance improvements as compared to the conventional per-cell optimized network. The second part of the thesis considers the transmission of both private and common messages for the sole purpose of intercell interference mitigation. It solves the issues of the second limitation mentioned above. It considers the benefit of designing decodable interference signals by allowing common-private message splitting at the transmitter and common message decoding by users in adjacent cells. It solves a network optimization problem of jointly determining the appropriate users in adjacent cells for rate splitting, the optimal beamforming vectors for both common and private messages, and the optimal common-private rates to minimize the total transmit power across the base-stations subject to service rate requirements for remote users. Observe that for fixed user selection and fixed common-private rate splitting, the optimization of beamforming vectors can be performed using a semidefinite programming approach. Further, this part of the thesis proposes a heuristic user-selection and rate splitting strategy to maximize the benefit of common message decoding. This part proposes a heuristic algorithm to characterize the improvement in the feasible rates with common-message decoding. Simulation results show that common message decoding can significantly improve both the total transmit power and the feasibility region for cell-edge users when base-stations are closely spaced from each other.

Conventional multicell wireless systems operate with out-of-cell interference treated as background noise; consequently, their performance faces two major limitations: 1)Signal processing is performed on a per-cell basis; and 2)Intercell interference detection is infeasible as intercell interference, although significantly above the noise level, is typically quite weak. In this thesis, we consider a multicell downlink scenario, where base-stations are equipped with multiple transmit antennas, the remote users are equipped with a single antenna, and multiple remote users are active simultaneously via spatial division multiplexing. We propose solutions for the above limitations by considering techniques for mitigating interference. The first part of the thesis proposes solutions for the first limitation. It considers the benefit of coordinating base-stations across multiple cells, where multiple base-stations may jointly optimize their respective beamformers to improve the overall system performance. It focuses on the design criteria of minimizing either the total weighted transmitted power or the maximum per-antenna power across the base-stations subject to signal-to-interference-and-noise-ratio (SINR) constraints at the remote users. The main contribution of this part is an efficient algorithm for finding the joint globally optimal beamformers across all base-stations. The proposed algorithm is based on a generalization of uplink-downlink duality to the multicell setting using the Lagrangian duality theory. An important feature is that it naturally leads to a distributed implementation in time-division duplex (TDD) systems. Simulation results suggest that coordinating the beamforming vectors alone already provides appreciable performance improvements as compared to the conventional per-cell optimized network. The second part of the thesis considers the transmission of both private and common messages for the sole purpose of intercell interference mitigation. It solves the issues of the second limitation mentioned above. It considers the benefit of designing decodable interference signals by allowing common-private message splitting at the transmitter and common message decoding by users in adjacent cells. It solves a network optimization problem of jointly determining the appropriate users in adjacent cells for rate splitting, the optimal beamforming vectors for both common and private messages, and the optimal common-private rates to minimize the total transmit power across the base-stations subject to service rate requirements for remote users. Observe that for fixed user selection and fixed common-private rate splitting, the optimization of beamforming vectors can be performed using a semidefinite programming approach. Further, this part of the thesis proposes a heuristic user-selection and rate splitting strategy to maximize the benefit of common message decoding. This part proposes a heuristic algorithm to characterize the improvement in the feasible rates with common-message decoding. Simulation results show that common message decoding can significantly improve both the total transmit power and the feasibility region for cell-edge users when base-stations are closely spaced from each other.