Multi-antenna wireless communication systems that employ a large number of antennas have recently stirred a lot of research interest. This is mainly due to the possibility of achieving very high spectral efficiency, power efficiency, and link reliability in such large-scale multiple-input multiple-output (MIMO) systems. An emerging architecture for large-scale multiuser MIMO communications is one where each base station (BS) is equipped with a large number of antennas (tens to hundreds of antennas) and the user terminals are equipped with fewer antennas (one to four antennas) each. The backhaul communication between base stations is also carried out using large number of antennas. Because of the high dimensionality of large-scale MIMO signals, the computational complexity of various transceiver operations can be prohibitively large. Therefore, low complexity techniques that scale well for transceiver signal processing in such large-scale MIMO systems are crucial. The transceiver operations of interest include signal encoding at the transmitter, and channel estimation, detection and decoding at the receiver. This thesis focuses on the design and analysis of novel low-complexity transceiver signal processing schemes for large-scale MIMO systems. In this thesis, we consider two types of large-scale MIMO systems, namely, massive MIMO systems and generalized spatial modulation MIMO (GSM-MIMO) systems. In massive MIMO, the mapping of information bits to modulation symbols is done using conventional modulation alphabets (e.g., QAM, PSK). In GSM-MIMO, few of the avail-able transmit antennas are activated in a given channel use, and information bits are conveyed through the indices of these active antennas, in addition to the bits conveyed through conventional modulation symbols. We also propose a novel modulation scheme named as precoder index modulation, where information bits are conveyed through the index of the chosen precoder matrix as well as the modulation symbols transmitted. Massive MIMO: In this part of the thesis, we propose a novel MIMO receiver which exploits channel hardening that occurs in large-scale MIMO channels. Channel hardening refers to the phenomenon where the off-diagonal terms of HH H become much weaker compared to the diagonal terms as the size of the channel gain matrix H increases. We exploit this phenomenon to devise a low-complexity channel estimation scheme and a message passing algorithm for signal detection at the BS receiver in massive MIMO systems. We refer to the proposed receiver as the channel hardening-exploiting message passing (CHEMP) receiver. The key novelties in the proposed CHEMP receiver are: (i) operation on the matched filtered system model, (ii) Gaussian approximation on the off-diagonal terms of the HH H matrix, and (iii) direct estimation of HH H instead of H and use of this estimate of HH H for detection The performance and complexity results show that the proposed CHEMP receiver achieves near-optimal performance in large-scale MIMO systems at complexities less than those of linear receivers like minimum mean squared error (MMSE) receiver. We also present a log-likelihood ratio (LLR) analysis that provides an analytical reasoning for this better performance of the CHEMP receiver. Further, the proposed message passing based detection algorithm enables us to combine it with low density parity check (LDPC) decoder to formulate a joint message passing based detector-decoder. For this joint detector-decoder, we design optimized irregular binary LDPC codes specific to the massive MIMO channel and the proposed receiver through EXIT chart matching. The LDPC codes thus obtained are shown to achieve improved coded bit error rate (BER) performance compared to off-the-shelf irregular LDPC codes. The performance of the CHEMP receiver degrades when the system loading factor (ratio of the number of users to the number of BS antennas) and the modulation alpha-bet size are large. It is of interest to devise receiver algorithms that work well for high system loading factors and modulation alphabet sizes. For this purpose, we propose a low-complexity factor-graph based vector message passing algorithm for signal detection. This algorithm uses a scalar Gaussian approximation of interference on the basic sys-tem model. The performance results show that this algorithm performs well for large modulation alphabets and high loading factors. We combine this detection algorithm with a non-binary LDPC decoder to obtain a joint detector-decoder, where the field size of the non-binary LDPC code is same as the size of the modulation alphabet. For this joint message passing based detector-decoder, we design optimized non-binary irregular LDPC codes tailored to the massive MIMO channel and the proposed detector. GSM-MIMO: In this part of the thesis, we consider GSM-MIMO systems in point-to-point as well as multiuser communication settings. GSM-MIMO has the advantage of requiring only fewer transmit radio frequency (RF) chains than the number of transmit antennas. We analyze the capacity of point-to-point GSM-MIMO, and obtain lower and upper bounds on the GSM-MIMO system capacity. We also derive an upper bound on the BER performance of maximum likelihood detection in GSM-MIMO systems. This bound is shown to be tight at moderate to high signal-to-noise ratios. When the number of transmit and receive antennas are large, the complexity of en-coding and decoding of GSM-MIMO signals can be prohibitively high. To alleviate this problem, we propose a low complexity GSM-MIMO encoding technique that utilizes com-binatorial number system for bits-to-symbol mapping. We also propose a novel layered message passing (LaMP) algorithm for decoding GSM-MIMO signals. Low computational complexity is achieved in the LaMP algorithm by detecting the modulation bits and the antenna index bits in two deferent layers. We then consider large-scale multiuser GSM-MIMO systems, where multiple users employ GSM at their transmitters to communicate with a BS having a large number of receive antennas. For this system, we develop computationally efficient message passing algorithms for signal detection using vector Gaussian approximation of interference. The performance results of these algorithms show that the GSM-MIMO system outperforms the massive MIMO system by several dBs for the same spectral efficiency. Precoder index modulation: It is known that the performance of a communication link can be enhanced by exploiting time diversity without reducing the rate of transmission using pseudo random phase preceding (PRPP). In order to further improve the performance of GSM-MIMO, we apply PRPP technique to GSM-MIMO systems. PRPP provides additional diversity advantage at the receiver and further improves the performance of GSM-MIMO systems. For PRPP-GSM systems, we propose methods to simultaneously precode both the antenna index bits and the modulation symbols using rectangular precoder matrices. Finally, we extend the idea of index modulation to pre-coding and propose a new modulation scheme referred to as precoder index modulation (PIM). In PIM, information bits are conveyed through the index of a prehared PRPP matrix, in addition to the information bits conveyed through the modulation symbols. PIM is shown to increase the achieved spectral efficiency, in addition to providing diver-sity advantages.