The search for the closest lattice point arises in many communication problems, and is known to be NP-hard. The Maximum Likelihood (ML) Detector is the optimal detector which yields an optimal solution to this problem, but at the expense of high computational complexity. Existing near-optimal methods used to solve the problem are based on the Sphere Decoder (SD), which searches for lattice points confined in a hyper-sphere around the received point. The SD has emerged as a powerful means of finding the solution to the ML detection problem for MIMO systems. However the bottleneck lies in the determination of the initial radius. This thesis is concerned with the detection of transmitted wireless signals in Multiple-Input Multiple-Output (MIMO) digital communication systems as efficiently and effectively as possible. The main objective of this thesis is to design efficient ML detection algorithms for MIMO systems based on the depth-first search (DFS) algorithms whilst taking into account complexity and bit error rate performance requirements for advanced digital communication systems. The increased capacity and improved link reliability of MIMO systems without sacrificing bandwidth efficiency and transmit power will serve as the key motivation behind the study of MIMO detection schemes. The fundamental principles behind MIMO systems are explored in Chapter 2. A generic framework for linear and non-linear tree search based detection schemes is then presented Chapter 3. This paves way for different methods of improving the achievable performance-complexity trade-off for all SD-based detection algorithms. The suboptimal detection schemes, in particular the Minimum Mean Squared Error-Successive Interference Cancellation (MMSE-SIC), will also serve as pre-processing as well as comparison techniques whilst channel capacity approaching Low Density Parity Check (LDPC) codes will be employed to evaluate the performance of the proposed SD. Numerical and simulation results show that non-linear detection schemes yield better performance compared to linear detection schemes, however, at the expense of a slight increase in complexity. The first contribution in this thesis is the design of a near ML-achieving SD algorithm for MIMO digital communication systems that reduces the number of search operations within the sphere-constrained search space at reduced detection complexity in Chapter 4. In this design, the distance between the ML estimate and the received signal is used to control the lower and upper bound radii of the proposed SD to prevent NP-complete problems. The detection method is based on the DFS algorithm and the Successive Interference Cancellation (SIC). The SIC ensures that the effects of dominant signals are effectively removed. Simulation results presented in this thesis show that by employing pre-processing detection schemes, the complexity of the proposed SD can be significantly reduced, though at marginal performance penalty. The second contribution is the determination of the initial sphere radius in Chapter 5. The new initial radius proposed in this thesis is based on the variable parameter α which is commonly based on experience and is chosen to ensure that at least a lattice point exists inside the sphere with high probability. Using the variable parameter α, a new noise covariance matrix which incorporates the number of transmit antennas, the energy of the transmitted symbols and the channel matrix is defined. The new covariance matrix is then incorporated into the EMMSE model to generate an improved EMMSE estimate. The EMMSE radius is finally found by computing the distance between the sphere centre and the improved EMMSE estimate. This distance can be fine-tuned by varying the variable parameter α. The beauty of the proposed method is that it reduces the complexity of the preprocessing step of the EMMSE to that of the Zero-Forcing (ZF) detector without significant performance degradation of the SD, particularly at low Signal-to-Noise Ratios (SNR). More specifically, it will be shown through simulation results that using the EMMSE preprocessing step will substantially improve performance whenever the complexity of the tree search is fixed or upper bounded. The final contribution is the design of the LRAD-MMSE-SIC based SD detection scheme which introduces a trade-off between performance and increased computational complexity in Chapter 6. The Lenstra-Lenstra-Lovasz (LLL) algorithm will be utilised to orthogonalise the channel matrix H to a new near orthogonal channel matrix H ̅.The increased computational complexity introduced by the LLL algorithm will be significantly decreased by employing sorted QR decomposition of the transformed channel H ̅ into a unitary matrix and an upper triangular matrix which retains the property of the channel matrix. The SIC algorithm will ensure that the interference due to dominant signals will be minimised while the LDPC will effectively stop the propagation of errors within the entire system. Through simulations, it will be demonstrated that the proposed detector still approaches the ML performance while requiring much lower complexity compared to the conventional SD.

[pt] Na primeira parte desta tese, são desenvolvidos dois esquemas de detecção por listas para sistemas MIMO multiusuário. As técnicas propostas usam uma única transformação de redução de reticulado (LR) para modificar a matriz de canal entre os usuários e a estação base (BS). Após a transformação LR, um candidato confiável do sinal transmitido é obtido usando um detector de cancelamento sucessivo de interferências (SIC). No detector em múltiplos ramos com redução de reticulado e cancelamento sucessivo de interferências (MB-LR-SIC) proposto, um número fixo de diferentes ordenamentos para o detector SIC gera uma lista de possíveis candidatos para a informação transmitida. O melhor candidato é escolhido usando o critério maximum likelihood (ML). No detector por listas de tamanho variável (VLD) proposto, um algoritmo que decide se o candidato atual tem uma boa qualidade ou se é necessário continuar procurando por um candidato melhor nos ordenamentos restantes é utilizado. Os resultados numéricos mostram que os esquemas propostos têm um desempenho quase ótimo com uma complexidade computacional bem abaixo do detector ML. Um esquema de detecção e decodificação iterativa (IDD) baseado no algoritmo VLD é também desenvolvido, produzindo um desempenho próximo a um sistema mono usuário (SU) livre de interferências. Na segunda parte desta tese, uma técnica de detecção desacoplada de sinais (DSD) para sistemas MIMO massivo é proposta. Esta técnica permite que o sinal composto recebido na BS seja separado em sinais independentes, correspondentes a diferentes classes de usuários, viabilizando assim o uso dos procedimentos de detecção propostos na primeira parte desta tese em sistemas MIMO massivos. Um modelo de sinais para sistemas MIMO massivo com antenas centralizadas e/ou antenas distribuídas operando em redes heterogêneas de quinta geração é proposto. Uma análise baseada na soma das taxas e um estudo de custo computacional para DSD são apresentados. Os resultados numéricos ilustram o excelente compromisso desempenho versus complexidade obtido com a técnica DSD quando comparada com o esquema de detecção conjunta tradicional. / [en] In the first part of this thesis, we introduce two list detection schemes for the uplink scenario of multiuser multiple-input multiple-output (MUMIMO) systems. The proposed techniques employ a single lattice reduction (LR) transformation to modify the channel matrix between the users and the base station (BS). After the LR transformation, a reliable candidate for the transmitted signal vector, provided by successive interference cancellation (SIC) detection is obtained. In the proposed multi-branch lattice reduction SIC (MB-LR-SIC) detector, a fixed number of different orderings, generates a list of SIC detection candidates. The best candidate is chosen according to the maximum likelihood (ML) selection criterion. For the proposed variable list detection (VLD) scheme, an algorithm to decide if the current candidate has good quality or if it is necessary to further explore different orderings to improve the detection performance is employed. Simulation results indicate that the proposed schemes have a near-optimal performance while keeping its computational complexity well below that of the ML detector. An iterative detection and decoding (IDD) scheme based on the VLD algorithm is also developed, producing an excellent performance that approaches the single user (SU) scenario. In the second part of this thesis, a decoupled signal detection (DSD) technique which allows the separation of uplink signals, for each user class, at the base station (BS) for massive MIMO systems is proposed. The proposed DSD allows to implement the detection procedures proposed in the first part of this thesis in massive MIMO scenarios. A mathematical signal model for massive MIMO systems with centralized and distributed antennas in the future fifth generation (5G) heterogeneous cellular networks is also developed. A sum-rate analysis and a study of computational cost for DSD are also presented. Simulation results show excellent performance of the proposed DSD algorithm when combined with linear and SIC-based detectors.