Dans le cadre du traitement statistique du signal, la plupart des algorithmes couramment utilisés reposent sur l'utilisation de la matrice de covariance des signaux étudiés. En pratique, ce sont les versions adaptatives de ces traitements, obtenues en estimant la matrice de covariance à l'aide d'échantillons du signal, qui sont utilisés. Ces algorithmes présentent un inconvénient : ils peuvent nécessiter un nombre d'échantillons important pour obtenir de bons résultats. Lorsque la matrice de covariance possède une structure rang faible, le signal peut alors être décomposé en deux sous-espaces orthogonaux. Les projecteurs orthogonaux sur chacun de ces sous espaces peuvent alors être construits, permettant de développer des méthodes dites rang faible. Les versions adaptatives de ces méthodes atteignent des performances équivalentes à celles des traitements classiques tout en réduisant significativement le nombre d'échantillons nécessaire. Par ailleurs, l'accroissement de la taille des données ne fait que renforcer l'intérêt de ce type de méthode. Cependant, cet accroissement s'accompagne souvent d'un accroissement du nombre de dimensions du système. Deux types d'approches peuvent être envisagées pour traiter ces données : les méthodes vectorielles et les méthodes tensorielles. Les méthodes vectorielles consistent à mettre les données sous forme de vecteurs pour ensuite appliquer les traitements classiques. Cependant, lors de la mise sous forme de vecteur, la structure des données est perdue ce qui peut entraîner une dégradation des performances et/ou un manque de robustesse. Les méthodes tensorielles permettent d'éviter cet écueil. Dans ce cas, la structure est préservée en mettant les données sous forme de tenseurs, qui peuvent ensuite être traités à l'aide de l'algèbre multilinéaire. Ces méthodes sont plus complexes à utiliser puisqu'elles nécessitent d'adapter les algorithmes classiques à ce nouveau contexte. En particulier, l'extension des méthodes rang faible au cas tensoriel nécessite l'utilisation d'une décomposition tensorielle orthogonale. Le but de cette thèse est de proposer et d'étudier des algorithmes rang faible pour des modèles tensoriels. Les contributions de cette thèse se concentrent autour de trois axes. Un premier aspect concerne le calcul des performances théoriques d'un algorithme MUSIC tensoriel basé sur la Higher Order Singular Value Decomposition (HOSVD) et appliqué à un modèle de sources polarisées. La deuxième partie concerne le développement de filtres rang faible et de détecteurs rang faible dans un contexte tensoriel. Ce travail s'appuie sur une nouvelle définition de tenseur rang faible et sur une nouvelle décomposition tensorielle associée : l'Alternative Unfolding HOSVD (AU-HOSVD). La dernière partie de ce travail illustre l'intérêt de l'approche tensorielle basée sur l'AU-HOSVD, en appliquant ces algorithmes à configuration radar particulière: le Traitement Spatio-Temporel Adaptatif ou Space-Time Adaptive Process (STAP). / Most of statistical signal processing algorithms, are based on the use of signal covariance matrix. In practical cases this matrix is unknown and is estimated from samples. The adaptive versions of the algorithms can then be applied, replacing the actual covariance matrix by its estimate. These algorithms present a major drawback: they require a large number of samples in order to obtain good results. If the covariance matrix is low-rank structured, its eigenbasis may be separated in two orthogonal subspaces. Thanks to the LR approximation, orthogonal projectors onto theses subspaces may be used instead of the noise CM in processes, leading to low-rank algorithms. The adaptive versions of these algorithms achieve similar performance to classic classic ones with less samples. Furthermore, the current increase in the size of the data strengthens the relevance of this type of method. However, this increase may often be associated with an increase of the dimension of the system, leading to multidimensional samples. Such multidimensional data may be processed by two approaches: the vectorial one and the tensorial one. The vectorial approach consists in unfolding the data into vectors and applying the traditional algorithms. These operations are not lossless since they involve a loss of structure. Several issues may arise from this loss: decrease of performance and/or lack of robustness. The tensorial approach relies on multilinear algebra, which provides a good framework to exploit these data and preserve their structure information. In this context, data are represented as multidimensional arrays called tensor. Nevertheless, generalizing vectorial-based algorithms to the multilinear algebra framework is not a trivial task. In particular, the extension of low-rank algorithm to tensor context implies to choose a tensor decomposition in order to estimate the signal and noise subspaces. The purpose of this thesis is to derive and study tensor low-rank algorithms. This work is divided into three parts. The first part deals with the derivation of theoretical performance of a tensor MUSIC algorithm based on Higher Order Singular Value Decomposition (HOSVD) and its application to a polarized source model. The second part concerns the derivation of tensor low-rank filters and detectors in a general low-rank tensor context. This work is based on a new definition of tensor rank and a new orthogonal tensor decomposition : the Alternative Unfolding HOSVD (AU-HOSVD). In the last part, these algorithms are applied to a particular radar configuration : the Space-Time Adaptive Process (STAP). This application illustrates the interest of tensor approach and algorithms based on AU-HOSVD.

Throughout stochastic data processing fields, mean and covariance matrices are commonly employed for purposes such as standardizing multivariate data through decorrelation. For practical applications, these matrices are usually estimated, and often, the data used for these estimates are non-Gaussian or may be corrupted by outliers or impulsive noise. To address this, robust estimators should be employed. However, in signal processing, where complex-valued data are common, the robust estimation techniques currently employed, such as M-estimators, provide limited robustness in the multivariate case. For this reason, this dissertation extends, to the complex-valued domain, the high-breakdown-point class of multivariate estimators called S-estimators. This dissertation defines S-estimators in the complex-valued context, and it defines their properties for complex-valued data. One major shortcoming of the leading high-breakdown-point multivariate estimators, such as the Rocke S-estimator and the smoothed hard rejection MM-estimator, is that they lack statistical efficiency at non-Gaussian distributions, which are common with real-world applications. This dissertation proposes a new tunable S-estimator, termed the Sq-estimator, for the general class of elliptically symmetric distributions—a class containing many common families such as the multivariate Gaussian, K-, W-, t-, Cauchy, Laplace, hyperbolic, variance gamma, and normal inverse Gaussian distributions. This dissertation demonstrates the diverse applicability and performance benefits of the Sq-estimator through theoretical analysis, empirical simulation, and the processing of real-world data. Through analytical and empirical means, the Sq-estimator is shown to generally provide higher maximum efficiency than the leading maximum-breakdown estimators, and it is also shown to generally be more stable with respect to initial conditions. To illustrate the theoretical benefits of the Sq for complex-valued applications, the efficiencies and influence functions of adaptive minimum variance distortionless response (MVDR) beamformers based on S- and M-estimators are compared. To illustrate the finite-sample performance benefits of the Sq-estimator, empirical simulation results of multiple signal classification (MUSIC) direction-of-arrival estimation are explored. Additionally, the optimal investment of real-world stock data is used to show the practical performance benefits of the Sq-estimator with respect to robustness to extreme events, estimation efficiency, and prediction performance. / Doctor of Philosophy / Throughout stochastic processing fields, mean and covariance matrices are commonly employed for purposes such as standardizing multivariate data through decorrelation. For practical applications, these matrices are usually estimated, and often, the data used for these estimates are non-normal or may be corrupted by outliers or large sporadic noise. To address this, estimators should be employed that are robust to these conditions. However, in signal processing, where complex-valued data are common, the robust estimation techniques currently employed provide limited robustness in the multivariate case. For this reason, this dissertation extends, to the complex-valued domain, the highly robust class of multivariate estimators called S-estimators. This dissertation defines S-estimators in the complex-valued context, and it defines their properties for complex-valued data. One major shortcoming of the leading highly robust multivariate estimators is that they may require unreasonably large numbers of samples (i.e. they may have low statistical efficiency) in order to provide good estimates at non-normal distributions, which are common with real-world applications. This dissertation proposes a new tunable S-estimator, termed the Sq-estimator, for the general class of elliptically symmetric distributions—a class containing many common families such as the multivariate Gaussian, K-, W-, t-, Cauchy, Laplace, hyperbolic, variance gamma, and normal inverse Gaussian distributions. This dissertation demonstrates the diverse applicability and performance benefits of the Sq-estimator through theoretical analysis, empirical simulation, and the processing of real-world data. Through analytical and empirical means, the Sq-estimator is shown to generally provide higher maximum efficiency than the leading highly robust estimators, and its solutions are also shown to generally be less sensitive to initial conditions. To illustrate the theoretical benefits of the Sq-estimator for complex-valued applications, the statistical efficiencies and robustness of adaptive beamformers based on various estimators are compared. To illustrate the finite-sample performance benefits of the Sq-estimator, empirical simulation results of signal direction-of-arrival estimation are explored. Additionally, the optimal investment of real-world stock data is used to show the practical performance benefits of the Sq-estimator with respect to robustness to extreme events, estimation efficiency, and prediction performance.