Dans le cadre du traitement statistique du signal, la plupart des algorithmes couramment utilisés reposent sur l'utilisation de la matrice de covariance des signaux étudiés. En pratique, ce sont les versions adaptatives de ces traitements, obtenues en estimant la matrice de covariance à l'aide d'échantillons du signal, qui sont utilisés. Ces algorithmes présentent un inconvénient : ils peuvent nécessiter un nombre d'échantillons important pour obtenir de bons résultats. Lorsque la matrice de covariance possède une structure rang faible, le signal peut alors être décomposé en deux sous-espaces orthogonaux. Les projecteurs orthogonaux sur chacun de ces sous espaces peuvent alors être construits, permettant de développer des méthodes dites rang faible. Les versions adaptatives de ces méthodes atteignent des performances équivalentes à celles des traitements classiques tout en réduisant significativement le nombre d'échantillons nécessaire. Par ailleurs, l'accroissement de la taille des données ne fait que renforcer l'intérêt de ce type de méthode. Cependant, cet accroissement s'accompagne souvent d'un accroissement du nombre de dimensions du système. Deux types d'approches peuvent être envisagées pour traiter ces données : les méthodes vectorielles et les méthodes tensorielles. Les méthodes vectorielles consistent à mettre les données sous forme de vecteurs pour ensuite appliquer les traitements classiques. Cependant, lors de la mise sous forme de vecteur, la structure des données est perdue ce qui peut entraîner une dégradation des performances et/ou un manque de robustesse. Les méthodes tensorielles permettent d'éviter cet écueil. Dans ce cas, la structure est préservée en mettant les données sous forme de tenseurs, qui peuvent ensuite être traités à l'aide de l'algèbre multilinéaire. Ces méthodes sont plus complexes à utiliser puisqu'elles nécessitent d'adapter les algorithmes classiques à ce nouveau contexte. En particulier, l'extension des méthodes rang faible au cas tensoriel nécessite l'utilisation d'une décomposition tensorielle orthogonale. Le but de cette thèse est de proposer et d'étudier des algorithmes rang faible pour des modèles tensoriels. Les contributions de cette thèse se concentrent autour de trois axes. Un premier aspect concerne le calcul des performances théoriques d'un algorithme MUSIC tensoriel basé sur la Higher Order Singular Value Decomposition (HOSVD) et appliqué à un modèle de sources polarisées. La deuxième partie concerne le développement de filtres rang faible et de détecteurs rang faible dans un contexte tensoriel. Ce travail s'appuie sur une nouvelle définition de tenseur rang faible et sur une nouvelle décomposition tensorielle associée : l'Alternative Unfolding HOSVD (AU-HOSVD). La dernière partie de ce travail illustre l'intérêt de l'approche tensorielle basée sur l'AU-HOSVD, en appliquant ces algorithmes à configuration radar particulière: le Traitement Spatio-Temporel Adaptatif ou Space-Time Adaptive Process (STAP). / Most of statistical signal processing algorithms, are based on the use of signal covariance matrix. In practical cases this matrix is unknown and is estimated from samples. The adaptive versions of the algorithms can then be applied, replacing the actual covariance matrix by its estimate. These algorithms present a major drawback: they require a large number of samples in order to obtain good results. If the covariance matrix is low-rank structured, its eigenbasis may be separated in two orthogonal subspaces. Thanks to the LR approximation, orthogonal projectors onto theses subspaces may be used instead of the noise CM in processes, leading to low-rank algorithms. The adaptive versions of these algorithms achieve similar performance to classic classic ones with less samples. Furthermore, the current increase in the size of the data strengthens the relevance of this type of method. However, this increase may often be associated with an increase of the dimension of the system, leading to multidimensional samples. Such multidimensional data may be processed by two approaches: the vectorial one and the tensorial one. The vectorial approach consists in unfolding the data into vectors and applying the traditional algorithms. These operations are not lossless since they involve a loss of structure. Several issues may arise from this loss: decrease of performance and/or lack of robustness. The tensorial approach relies on multilinear algebra, which provides a good framework to exploit these data and preserve their structure information. In this context, data are represented as multidimensional arrays called tensor. Nevertheless, generalizing vectorial-based algorithms to the multilinear algebra framework is not a trivial task. In particular, the extension of low-rank algorithm to tensor context implies to choose a tensor decomposition in order to estimate the signal and noise subspaces. The purpose of this thesis is to derive and study tensor low-rank algorithms. This work is divided into three parts. The first part deals with the derivation of theoretical performance of a tensor MUSIC algorithm based on Higher Order Singular Value Decomposition (HOSVD) and its application to a polarized source model. The second part concerns the derivation of tensor low-rank filters and detectors in a general low-rank tensor context. This work is based on a new definition of tensor rank and a new orthogonal tensor decomposition : the Alternative Unfolding HOSVD (AU-HOSVD). In the last part, these algorithms are applied to a particular radar configuration : the Space-Time Adaptive Process (STAP). This application illustrates the interest of tensor approach and algorithms based on AU-HOSVD.

Modern big data often emerge as tensors. Standard statistical methods are inadequate to deal with datasets of large volume, high dimensionality, and complex structure. Therefore, it is important to develop algorithms such as low-rank tensor decomposition for data compression, dimensionality reduction, and approximation. With the advancement in technology, high-dimensional images are becoming ubiquitous in the medical field. In lung radiation therapy, the respiratory motion of the lung introduces variabilities during treatment as the tumor inside the lung is moving, which brings challenges to the precise delivery of radiation to the tumor. Several approaches to quantifying this uncertainty propose using a model to formulate the motion through a mathematical function over time. [Li et al., 2011] uses principal component analysis (PCA) to propose one such model using each image as a long vector. However, the images come in a multidimensional arrays, and vectorization breaks the spatial structure. Driven by the needs to develop low-rank tensor decomposition and provided the 4DCT and Displacement Vector Field (DVF), we introduce two tensor decompositions, Population Value Decomposition (PVD) and Population Tucker Decomposition (PTD), to estimate the respiratory lung motion with high levels of accuracy and data compression. The first algorithm is a generalization of PVD [Crainiceanu et al., 2011] to higher order tensor. The second algorithm generalizes the concept of PVD using Tucker decomposition. Both algorithms are tested on clinical and phantom DVFs. New metrics for measuring the model performance are developed in our research. Results of the two new algorithms are compared to the result of the PCA algorithm.

Multi-instance data, in which each object (e.g., a document) is a collection of instances (e.g., word), are widespread in machine learning, signal processing, computer vision, bioinformatic, music, and social sciences. Existing probabilistic models, e.g., latent Dirichlet allocation (LDA), probabilistic latent semantic indexing (pLSI), and discrete component analysis (DCA), have been developed for modeling and analyzing multiinstance data. Such models introduce a generative process for multi-instance data which includes a low dimensional latent structure. While such models offer a great freedom in capturing the natural structure in the data, their inference may present challenges. For example, the sensitivity in choosing the hyper-parameters in such models, requires careful inference (e.g., through cross-validation) which results in large computational complexity. The inference for fully Bayesian models which contain no hyper-parameters often involves slowly converging sampling methods. In this work, we develop approaches for addressing such challenges and further enhancing the utility of such models. This dissertation demonstrates a unified convex framework for probabilistic modeling of multi-instance data. The three main aspects of the proposed framework are as follows. First, joint regularization is incorporated into multiple density estimation to simultaneously learn the structure of the distribution space and infer each distribution. Second, a novel confidence constraints framework is used to facilitate a tuning-free approach to control the amount of regularization required for the joint multiple density estimation with theoretical guarantees on correct structure recovery. Third, we formulate the problem using a convex framework and propose efficient optimization algorithms to solve it. This work addresses the unique challenges associated with both discrete and continuous domains. In the discrete domain we propose a confidence-constrained rank minimization (CRM) to recover the exact number of topics in topic models with theoretical guarantees on recovery probability and mean squared error of the estimation. We provide a computationally efficient optimization algorithm for the problem to further the applicability of the proposed framework to large real world datasets. In the continuous domain, we propose to use the maximum entropy (MaxEnt) framework for multi-instance datasets. In this approach, bags of instances are represented as distributions using the principle of MaxEnt. We learn basis functions which span the space of distributions for jointly regularized density estimation. The basis functions are analogous to topics in a topic model. We validate the efficiency of the proposed framework in the discrete and continuous domains by extensive set of experiments on synthetic datasets as well as on real world image and text datasets and compare the results with state-of-the-art algorithms. / Graduation date: 2013

In fields across science and engineering, we are increasingly faced with problems where the number of variables or features we need to estimate is much larger than the number of observations. Under such high-dimensional scaling, for any hope of statistically consistent estimation, it becomes vital to leverage any potential structure in the problem such as sparsity, low-rank structure or block sparsity. However, data may deviate significantly from any one such statistical model. The motivation of this thesis is: can we simultaneously leverage more than one such statistical structural model, to obtain consistency in a larger number of problems, and with fewer samples, than can be obtained by single models? Our approach involves combining via simple linear superposition, a technique we term dirty models. The idea is very simple: while any one structure might not capture the data, a superposition of structural classes might. Dirty models thus searches for a parameter that can be decomposed into a number of simpler structures such as (a) sparse plus block-sparse, (b) sparse plus low-rank and (c) low-rank plus block-sparse. In this thesis, we propose dirty model based algorithms for different problems such as multi-task learning, graph clustering and time-series analysis with latent factors. We analyze these algorithms in terms of the number of observations we need to estimate the variables. These algorithms are based on convex optimization and sometimes they are relatively slow. We provide a class of low-complexity greedy algorithms that not only can solve these optimizations faster, but also guarantee the solution. Other than theoretical results, in each case, we provide experimental results to illustrate the power of dirty models. / text

Quantifying uncertainties in large-scale forward and inverse PDE simulations has emerged as a central challenge facing the field of computational science and engineering. The promise of modeling and simulation for prediction, design, and control cannot be fully realized unless uncertainties in models are rigorously quantified, since this uncertainty can potentially overwhelm the computed result. While statistical inverse problems can be solved today for smaller models with a handful of uncertain parameters, this task is computationally intractable using contemporary algorithms for complex systems characterized by large-scale simulations and high-dimensional parameter spaces. In this dissertation, I address issues regarding the theoretical formulation, numerical approximation, and algorithms for solution of infinite-dimensional Bayesian statistical inverse problems, and apply the entire framework to a problem in global seismic wave propagation. Classical (deterministic) approaches to solving inverse problems attempt to recover the “best-fit” parameters that match given observation data, as measured in a particular metric. In the statistical inverse problem, we go one step further to return not only a point estimate of the best medium properties, but also a complete statistical description of the uncertain parameters. The result is a posterior probability distribution that describes our state of knowledge after learning from the available data, and provides a complete description of parameter uncertainty. In this dissertation, a computational framework for such problems is described that wraps around the existing forward solvers, as long as they are appropriately equipped, for a given physical problem. Then a collection of tools, insights and numerical methods may be applied to solve the problem, and interrogate the resulting posterior distribution, which describes our final state of knowledge. We demonstrate the framework with numerical examples, including inference of a heterogeneous compressional wavespeed field for a problem in global seismic wave propagation with 10⁶ parameters.

A restarted symplectic Lanczos method for the Hamiltonian eigenvalue problem is presented. The Lanczos vectors are constructed to form a symplectic basis. Breakdowns and near-breakdowns are overcome by inexpensive implicit restarts. The method is used to compute eigenvalues, eigenvectors and invariant subspaces of large and sparse Hamiltonian matrices and low rank approximations to the solution of continuous-time algebraic Riccati equations with large and sparse coefficient matrices.

Dans ce manuscrit de thèse, nous introduisons les avancées récentes sur la décomposition en matrices (et tenseurs) de rang faible et parcimonieuse ainsi que les contributions pour faire face aux principaux problèmes dans ce domaine. Nous présentons d’abord un aperçu des méthodes matricielles et tensorielles les plus récentes ainsi que ses applications sur la modélisation d’arrière-plan et la segmentation du premier plan. Ensuite, nous abordons le problème de l’initialisation du modèle de fond comme un processus de reconstruction à partir de données manquantes ou corrompues. Une nouvelle méthodologie est présentée montrant un potentiel intéressant pour l’initialisation de la modélisation du fond dans le cadre de VSI. Par la suite, nous proposons une version « double contrainte » de l’ACP robuste pour améliorer la détection de premier plan en milieu marin dans des applications de vidéo-surveillance automatisées. Nous avons aussi développé deux algorithmes incrémentaux basés sur tenseurs afin d’effectuer une séparation entre le fond et le premier plan à partir de données multidimensionnelles. Ces deux travaux abordent le problème de la décomposition de rang faible et parcimonieuse sur des tenseurs. A la fin, nous présentons un travail particulier réalisé en conjonction avec le Centre de Vision Informatique (CVC) de l’Université Autonome de Barcelone (UAB). / This thesis introduces the recent advances on decomposition into low-rank plus sparse matrices and tensors, as well as the main contributions to face the principal issues in moving object detection. First, we present an overview of the state-of-the-art methods for low-rank and sparse decomposition, as well as their application to background modeling and foreground segmentation tasks. Next, we address the problem of background model initialization as a reconstruction process from missing/corrupted data. A novel methodology is presented showing an attractive potential for background modeling initialization in video surveillance. Subsequently, we propose a double-constrained version of robust principal component analysis to improve the foreground detection in maritime environments for automated video-surveillance applications. The algorithm makes use of double constraints extracted from spatial saliency maps to enhance object foreground detection in dynamic scenes. We also developed two incremental tensor-based algorithms in order to perform background/foreground separation from multidimensional streaming data. These works address the problem of low-rank and sparse decomposition on tensors. Finally, we present a particular work realized in conjunction with the Computer Vision Center (CVC) at Autonomous University of Barcelona (UAB).

In many areas of science one often has a given matrix, representing for example a measured data set and is required to find a matrix that is closest in a suitable norm to the matrix and possesses additionally a structure, inherited from the model used or coming from the application. We call these problems structured matrix nearness problems. We look at three different groups of these problems that come from real applications, analyze the properties of the corresponding matrix structure, and propose algorithms to solve them efficiently. The first part of this thesis concerns the nearness problem of finding the nearest k factor correlation matrix C(X) = diag(I_n -XX T)+XX T to a given symmetric matrix, subject to natural nonlinear constraints on the elements of the n x k matrix X, where distance is measured in the Frobenius norm. Such problems arise, for example, when one is investigating factor models of collateralized debt obligations (CDOs) or multivariate time series. We examine several algorithms for solving the nearness problem that differ in whether or not they can take account of the nonlinear constraints and in their convergence properties. Our numerical experiments show that the performance of the methods depends strongly on the problem, but that, among our tested methods, the spectral projected gradient method is the clear winner. In the second part we look at two two-sided optimization problems where the matrix of unknowns Y ε R {n x p} lies in the Stiefel manifold. These two problems come from an application in atomic chemistry where one is looking for atomic orbitals with prescribed occupation numbers. We analyze these two problems, propose an analytic optimal solution of the first and show that an optimal solution of the second problem can be found by solving a convex quadratic programming problem with box constraints and p unknowns. We prove that the latter problem can be solved by the active-set method in at most 2p iterations. Subsequently, we analyze the set of optimal solutions C}= {Y ε R n x p:Y TY=I_p,Y TNY=D} of the first problem for N symmetric and D diagonal and find that a slight modification of it is a Riemannian manifold. We derive the geometric objects required to make an optimization over this manifold possible. We propose an augmented Lagrangian-based algorithm that uses these geometric tools and allows us to optimize an arbitrary smooth function over C. This algorithm can be used to select a particular solution out of the latter set C by posing a new optimization problem. We compare it numerically with a similar algorithm that ,however, does not apply these geometric tools and find that our algorithm yields better performance. The third part is devoted to low rank nearness problems in the Q-norm, where the matrix of interest is additionally of linear structure, meaning it lies in the set spanned by s predefined matrices U₁,..., U_s ε {0,1} n x p. These problems are often associated with model reduction, for example in speech encoding, filter design, or latent semantic indexing. We investigate three approaches that support any linear structure and examine further the geometric reformulation by Schuermans et al. (2003). We improve their algorithm in terms of reliability by applying the augmented Lagrangian method and show in our numerical tests that the resulting algorithm yields better performance than other existing methods.

Dans les applications modernes des statistiques et de l'apprentissage, il est courant que les données récoltées présentent un certain nombre d'imperfections. En particulier, les données sont souvent hétérogènes, c'est-à-dires qu'elles contiennent à la fois des informations quantitatives et qualitatives, incomplètes, lorsque certaines informations sont inaccessibles ou corrompues, et multi-sources, c'est-à-dire qu'elles résultent de l'agrégation de plusieurs jeux de données indépendant. Dans cette thèse, nous développons plusieurs méthodes pour l'analyse de données hétérogènes, incomplètes et multi-source. Nous nous attachons à étudier tous les aspects de ces méthodes, en fournissant des études théoriques précises, ainsi que des implémentations disponibles au public, et des évaluations empiriques. En particulier, nous considérons en détail deux applications issues de l'écologie pour la première et de la médecine pour la seconde. / In modern applications of statistics and machine learning, one often encounters many data imperfections. In particular, data are often heterogeneous, i.e. combine quantitative and qualitative information, incomplete, with missing values caused by machine failure or nonresponse phenomenons, and multi-source, when the data result from the compounding of diverse sources. In this dissertation, we develop several methods for the analysis of multi-source, heterogeneous and incomplete data. We provide a complete framework, and study all the aspects of the different methods, with thorough theoretical studies, open source implementations, and empirical evaluations. We study in details two particular applications from ecology and medical sciences.

A restarted symplectic Lanczos method for the Hamiltonian eigenvalue problem is presented. The Lanczos vectors are constructed to form a symplectic basis. Breakdowns and near-breakdowns are overcome by inexpensive implicit restarts. The method is used to compute eigenvalues, eigenvectors and invariant subspaces of large and sparse Hamiltonian matrices and low rank approximations to the solution of continuous-time algebraic Riccati equations with large and sparse coefficient matrices.