Recently, massive multiple-input multiple-output (MIMO) systems have gained significant attention as a new network architecture to not only achieving unprecedented spectral and energy efficiencies, but also to alleviating propagation losses and inter-user/inter-cell interference. Therefore, massive MIMO has been identified as one of the key candidate technologies for the 5th generation wireless standard. This dissertation thus focuses on (1) developing a performance analysis framework for cognitive massive MIMO systems by investigating the uplink transmissions of multi-cell multi-user massive MIMO secondary systems, which are underlaid in multi-cell multi-user primary massive MIMO systems, with taking into consideration the detrimental effects of practical transmission impairments, (2) proposing a new wireless-powered underlay cognitive massive MIMO system model, as the secondary user nodes is empowered by the ability to efficiently harvest energy from the primary user transmissions, and then access and utilize the primary network spectrum for information transmission, and (3) developing a secure communication strategy for cognitive multi-user massive MIMO systems, where physical layer secure transmissions are provisioned for both primary and secondary systems by exploiting linear precoders and artificial noise (AN) generation in order to degrade the signal decodability at eavesdropper. The key design feature of the proposed cognitive systems is to leverage the spatial multiplexing strategies to serve a large number of spatially distributed user nodes by using very large numbers of antennas at the base-stations. Moreover, the fundamental performance metrics, the secondary transmit power constraints, which constitute the underlay secondary transmissions subject to a predefined primary interference temperature, and the achievable sum rates of the primary and secondary systems, are characterized under different antenna array configurations. Additionally, the detrimental impact of practical wireless transmission impairments on the performance of the aforementioned systems are quantified. The important insights obtained throughout these analyses can be used as benchmarks for designing practical cognitive spectrum sharing networks.

The next-generation wireless technologies are currently being researched to address the ever-increasing demands for higher data rates, massive connectivity, improved reliability, and extended coverage. Recently, massive multiple-input multiple-output (MIMO) has gained significant attention as a new physical-layer transmission technology that can achieve unprecedented spectral and energy efficiency gains via aggressive spatial multiplexing. Thus, massive MIMO has been one of the key enabling technologies for the fifth-generation and subsequent wireless standards. This dissertation thus focuses on developing a system, channel, and signal models by considering the practical wireless transmission impairments for massive MIMO systems, and ascertaining the viability of massive MIMO in fulfilling massive access, improved spectrum, enhanced security, and energy efficiency requirements. Specifically, new system and channel models, pilot sequence designs and channel estimation techniques, secure transmit/receive beamforming techniques, transmit power allocation schemes with enhanced security provisions, energy efficiency, and user fairness, and comprehensive performance analysis frameworks are developed for massive MIMO-aided non-orthogonal multiple access (NOMA), cognitive spectrum-sharing, and wireless relaying architectures.Our first work focuses on developing physical-layer transmission schemes for NOMA-aided massive MIMO systems. A spatial signature-based user-clustering and pilot allocation scheme is first formulated, and thereby, a hybrid orthogonal multiple access (OMA)/NOMA transmission scheme is proposed to boost the number of simultaneous connections. In our second work, the viability of invoking downlink pilots to boost the achievable rate of NOMA-aided massive MIMO is investigated. The third research contribution investigates the performance of underlay spectrum-sharing massive MIMO systems for reverse time division duplexing based transmission strategies, in which primary and secondary systems concurrently operate in opposite directions. Thereby, we show that the secondary system can be operated with its maximum average transmit power independent of the primary system in the limit of infinity many primary/secondary base-station antennas. In our fourth work, signal processing techniques, power allocation, and relay selection schemes are designed and analyzed for massive MIMO relay networks to optimize the trade-off among the achievable user rates, coverage, and wireless resource usage. Finally, the cooperative jamming and artificial noise-based secure transmission strategies are developed for massive MIMO relay networks with imperfect legitimate user channel information and with no channel knowledge of the eavesdropper. The key design criterion of the aforementioned transmission strategies is to efficiently combine the spatial multiplexing gains and favorable propagation conditions of massive MIMO with properties of NOMA, underlay spectrum-sharing, and wireless relay networks via efficient signal processing.