The next-generation wireless technologies are currently being researched to address the ever-increasing demands for higher data rates, massive connectivity, improved reliability, and extended coverage. Recently, massive multiple-input multiple-output (MIMO) has gained significant attention as a new physical-layer transmission technology that can achieve unprecedented spectral and energy efficiency gains via aggressive spatial multiplexing. Thus, massive MIMO has been one of the key enabling technologies for the fifth-generation and subsequent wireless standards. This dissertation thus focuses on developing a system, channel, and signal models by considering the practical wireless transmission impairments for massive MIMO systems, and ascertaining the viability of massive MIMO in fulfilling massive access, improved spectrum, enhanced security, and energy efficiency requirements. Specifically, new system and channel models, pilot sequence designs and channel estimation techniques, secure transmit/receive beamforming techniques, transmit power allocation schemes with enhanced security provisions, energy efficiency, and user fairness, and comprehensive performance analysis frameworks are developed for massive MIMO-aided non-orthogonal multiple access (NOMA), cognitive spectrum-sharing, and wireless relaying architectures.Our first work focuses on developing physical-layer transmission schemes for NOMA-aided massive MIMO systems. A spatial signature-based user-clustering and pilot allocation scheme is first formulated, and thereby, a hybrid orthogonal multiple access (OMA)/NOMA transmission scheme is proposed to boost the number of simultaneous connections. In our second work, the viability of invoking downlink pilots to boost the achievable rate of NOMA-aided massive MIMO is investigated. The third research contribution investigates the performance of underlay spectrum-sharing massive MIMO systems for reverse time division duplexing based transmission strategies, in which primary and secondary systems concurrently operate in opposite directions. Thereby, we show that the secondary system can be operated with its maximum average transmit power independent of the primary system in the limit of infinity many primary/secondary base-station antennas. In our fourth work, signal processing techniques, power allocation, and relay selection schemes are designed and analyzed for massive MIMO relay networks to optimize the trade-off among the achievable user rates, coverage, and wireless resource usage. Finally, the cooperative jamming and artificial noise-based secure transmission strategies are developed for massive MIMO relay networks with imperfect legitimate user channel information and with no channel knowledge of the eavesdropper. The key design criterion of the aforementioned transmission strategies is to efficiently combine the spatial multiplexing gains and favorable propagation conditions of massive MIMO with properties of NOMA, underlay spectrum-sharing, and wireless relay networks via efficient signal processing.