La thèse s'inscrit dans la perspective de l'explosion du trafic de données générée par l'augmentation du nombre d'utilisateurs ainsi que la croissance du débit qui doivent être prises en compte dans la définition des futures générations de communications radiocellulaires. Une solution est la technologie «large-scale MIMO » (systèmes MIMO de grande dimension) qui pose plusieurs défis. La conception des nouveaux algorithmes de détection de faible complexité est indispensable vu que les algorithmes classiques ne sont plus adaptés à cette configuration à cause de leurs mauvaises performances de détection ou de leur complexité trop élevée fonction du nombre d'antennes. Une première contribution de la thèse est un algorithme basé sur la technique de l'acquisition comprimée en exploitant les propriétés des signaux à alphabet fini. Appliqué à des systèmes MIMO de grande dimension, déterminés et sous-déterminés, cet algorithme réalise des performances (qualité de détection, complexité) prometteuses et supérieures comparé aux algorithmes de l'état de l'art. Une étude théorique approfondie a été menée pour déterminer les conditions optimales de fonctionnement et la distribution statistique des sorties. Une seconde contribution est l'intégration de l'algorithme original dans un récepteur itératif en différenciant les cas codé (code correcteur d'erreurs présent) et non codé. Un autre défi pour tenir les promesses des systèmes large scale MIMO (efficacité spectrale élevée) est l'estimation de canal. Une troisième contribution de la thèse est la proposition d'algorithmes d'estimation semi-aveugles qui fonctionnent avec une taille minimale des séquences d'apprentissage (égale au nombre d'utilisateurs) et atteignent des performances très proches de la borne théorique. / The thesis is part of the prospect of the explosion of data traffic generated by the increase of the number of users as well as the growth of the bit rate which must be taken into account in the definition of future generations of radio-cellular communications. A solution is the large-scale MIMO technology (MIMO systems oflarge size) which poses several challenges. The design of the new low complexity detection algorithms is indispensable since the conventional algorithms are no longer adapted to this configuration because of their poor detection performance or their too high complexity depending on the number of antennas. A first contribution of the thesis is an algorithm based on the technique of compressed sensing by exploiting the propertiesof the signals with finite alphabet. Applied to large-scale, determined and under-determined MIMO systems, this algorithm achieves promising and superior performance (quality ofdetection, complexity) compared to state-ofthe-art algorithms. A thorough theoretical study was conducted to determine the optimal operating conditions and the statistical distribution of outputs. A second contribution is the integration of the original algorithm into an iterative receiver by differentiating the coded and uncoded cases. Another challenge to keeping the promise of large- scale MIMO systems (high spectral efficiency) is channel estimation. A third contribution of the thesis is the proposal of semi-blind channel estimation algorithms that work with a minimum size of pilot sequences (equal to the number of users) and reach performances very close to the theoretical bound.

Advanced mobile communication systems are characterized by a dense deployment of different types of wireless access points. Since these systems are primarily limited by interference, multiple-input multiple-output (MIMO) techniques as well as coordinated transmission and detection schemes are necessary to mitigate this limitation. Thus, mobile communication systems become more complex which requires that also the mathematical tools for their theoretical analysis must evolve. These must be able to take the most important system characteristics into account, such as fading, path loss, and interference. The aim of this thesis is to develop such tools based on large random matrix theory and to demonstrate their usefulness with the help of several practical applications, such as the performance analysis of network MIMO and large-scale MIMO systems, the design of low-complexity polynomial expansion detectors, and the study of random beamforming techniques as well as multi-hop relay and double-scattering channels. The methods developed in this work provide deterministic approximations of the system performance which become arbitrarily tight in the large system regime with an unlimited number of transmitting and receiving devices. This leads in many cases to simple and close approximations of the finite-size system performance and allows one to draw relevant conclusions about the most significant parameters. One can think of these methods as a way to provide a deterministic abstraction of the physical layer which substantially reduces the system complexity. Due to this complexity reduction, it is possible to carry out a system optimization which would be otherwise intractable.

[pt] O emprego conjunto de Redes de Acesso por Rádio em Nuvem (CRANs) e sistemas de múltiplas entradas e múltiplas saídas (MIMO) de larga escala é uma solução chave para atender aos requisitos da quinta geração (5G) de redes sem fio. No entanto, alguns desafios ainda precisam ser superados como a redução do consumo de energia do sistema, a capacidade limitada dos links fronthaul e a redução dos custos de implantação e operação. Embora seja prejudicial para o desempenho do sistema, a quantização em baixa resolução é proposta como uma solução para estes desafios. Portanto, técnicas que melhoram o desempenho de sistemas quantizados grosseiramente são necessárias. Em sistemas móveis, os ADCs geralmente são precedidos por um controle de ganho automático (AGC). O AGC trabalha moldando a amplitude do sinal recebido dentro do intervalo do quantizador para usar eficientemente a resolução. A fim de solucionar esses problemas, esta dissertação apresenta uma otimização conjunta do AGC, que funciona nas cabeças de rádio remotas (RRHs), e um filtro de recepção linear de baixa resolução consciente (LRA) baseado no mínimo erro quadrático médio (MMSE), que funciona na unidade de nuvem (CU), para sistemas quantizados grosseiramente. Desenvolvemos receptores de cancelamento de interferência lineares e sucessivos (SIC) com base na proposta conjunta de AGC e LRA MMSE (AGC-LRA-MMSE). Uma análise da soma das taxas alcançáveis juntamente com um estudo de complexidade computacional também são realizadas. As simulações mostram que o projeto proposto fornece taxas de erro reduzidas e taxas alcançáveis mais altas do que as técnicas existentes. / [en] The joint employment of Cloud Radio Access Networks (C-RANs) and large-scale multiple-input multiple-output (MIMO) systems is a key solution to fulfill the requirements of the fifth generation (5G) of wireless networks. However, some challenges are still open to be overcome such as the high power consumption of large-scale MIMO systems, which employ a large number of analog-to-digital converters (ADCs), the capacity bottleneck of the fronthaul links and the system cost reduction. Although it often affects the system performance, the low-resolution quantization is a possible solution for these problems. Therefore, techniques that improve the performance of coarsely quantized systems are needed. In mobile applications, the ADCs are usually preceded by an automatic gain control (AGC). The AGC works shaping the received signal amplitude within the quantizer range to efficiently use the ADC resolution. Then, the optimization of an AGC is especially important. In order to present possible solutions for these issues, this thesis presents a joint optimization of the AGC, which works in the remote radio heads (RRHs), and a low-resolution aware (LRA) linear receive filter based on the minimum mean square error (MMSE), which works in the cloud unit (CU), for coarsely quantized large-scale MIMO with CRAN systems. We develop linear and successive interference cancellation (SIC) receivers based on the proposed joint AGC and LRA MMSE (AGCLRAMMSE) approach. An analysis of the achievable sum rates along with a computational complexity study is also carried out. Simulations show that the proposed design provides improved error rates and higher achievable rates than existing techniques.

[pt] Este trabalho tem como finalidade estudar o problema da detecção de usuários no canal reverso de sistemas MIMO de larga escala, que são caracterizados pelo elevado número de elementos de transmissão e recepção, com foco na complexidade computacional e no desempenho em termos de taxa de erro destes sistemas. Inicialmente, os algoritmos de detecção da família Likelihood Ascent Search (LAS) são investigados e é desenvolvido um novo algoritmo de detecção, denominado de Random-List Based LAS (RLBLAS), capaz de atingir melhores taxas de erros com menor complexidade computacional do que os demais detectores considerados. Posteriormente, técnicas de detecção e decodificação iterativas (Iterative Detection and Decoding - IDD) em sistemas MIMO foram analisadas de forma a propor uma estratégia IDD de complexidade computacional reduzida a fim de viabilizar a sua aplicação em cenários massivos. Finalmente, o problema da contaminação por pilotos em sistemas MIMO multicelulares de larga escala, um dos principais limitadores do desempenho desse tipo de sistema, é estudado e estratégias de detecção com cooperação parcial entre as estações base componentes do sistema que visam mitigar os efeitos da contaminação por pilotos são propostas. As análises e afirmações realizadas durante a presente tese são sustentadas por resultados de simulações de Monte Carlo dos sistemas de comunicações em diversos cenários distintos, incluindo os casos em que são considerados os efeitos de correlação entre as antenas de transmissão/recepção, os efeitos de sombreamento e os erros de estimação dos estados dos canais de comunicações envolvidos. / [en] This work focuses on the multi-user multi-cellular large-scale MIMO reverse channel detection problem, where the number of transmitting and receiving antenna elements grows to the order of hundreds. In these scenarios, one major issue is the computational complexity of such systems. Therefore, this thesis aims to propose low-complexity techniques with good BER performance for the reverse channel detection of MIMO systems. Initially, the detection algorithms of the Likelihood Ascent Search (LAS) family are investigated and a new LAS based detector is proposed. This new detector, named Random-List Based LAS (RLB-LAS), is capable of achieving better BER with lower complexity then the other considered detectors. Next, iterative detection and decoding (IDD) techniques are analyzed in order to propose an IDD strategy applied to the detection and decoding of the reverse MIMO channel with reduced complexity to make possible its application to massive scenarios. Finally, the pilot contamination problem in multi-cellular large-scale MIMO systems, one of the major bounds on BER performance of these systems, are studied and some cooperative strategies are proposed in order to reduce the effects of this type of impairments. The analysis and statements of this thesis are supported by Monte Carlo simulation results of the considered systems in different scenarios, including the cases where the effects of transmitting and receiving antenna correlation, log-normal shadowing, and the estimation errors on the channel state information acquisition are considered.

Les futurs systèmes de communication mobile sont caractérisés par un déploiement de plus en plus dense de différents types de points d'accès sans fil. Afin d’atténuer les interférences dans ces systèmes, les techniques aux entrées multiples-sorties multiples (MIMO) ainsi que la coopération entre les émetteurs et/ou les récepteurs sont nécessaires. Les systèmes de communication mobile en deviennent plus complexes, ce qui impose une évolution des outils mathématiques permettant leur analyse. Ceux-ci doivent être capables de prendre en compte les caractéristiques les plus importantes du système, telles que l'affaiblissement de propagation, les interférences et l'information imparfaite d'état du canal. Le but de cette thèse est de développer de tels outils basés sur la théorie des grandes matrices aléatoires et de démontrer leur utilité à l'aide de plusieurs applications pratiques, telles que l'analyse des performances des systèmes « network MIMO » et des systèmes MIMO à grande échelle, la conception de détecteurs de faible complexité à expansion polynomiale, l'étude des techniques de précodage unitaire aléatoire, ainsi que l'analyse de canaux à relais multiples et de canaux à double diffusion. En résumé, les méthodes développées dans ce travail fournissent des approximations déterministes de la performance du système qui deviennent exactes en régime asymptotique avec un nombre illimité d'émetteurs et de récepteurs. Cette approche conduit souvent à des approximations de la performance du système étonnamment simples et précises et permet de tirer d’importantes conclusions sur les paramètres les plus pertinents. / Advanced mobile communication systems are characterized by a dense deployment of different types of wireless access points. Since these systems are primarily limited by interference, multiple-input multiple-output (MIMO) techniques as well as coordinated transmission and detection schemes are necessary to mitigate this limitation. Thus, mobile communication systems become more complex which requires that also the mathematical tools for their theoretical analysis must evolve. These must be able to take the most important system characteristics into account, such as fading, path loss, and interference. The aim of this thesis is to develop such tools based on large random matrix theory and to demonstrate their usefulness with the help of several practical applications, such as the performance analysis of network MIMO and large-scale MIMO systems, the design of low-complexity polynomial expansion detectors, and the study of random beamforming techniques as well as multi-hop relay and double-scattering channels. The methods developed in this work provide deterministic approximations of the system performance which become arbitrarily tight in the large system regime with an unlimited number of transmitting and receiving devices. This leads in many cases to simple and close approximations of the finite-size system performance and allows one to draw relevant conclusions about the most significant parameters. One can think of these methods as a way to provide a deterministic abstraction of the physical layer which substantially reduces the system complexity. Due to this complexity reduction, it is possible to carry out a system optimization which would be otherwise intractable.