Reconstructing sparse signals from undersampled measurements is a challenging problem that arises in many areas of data science, such as signal processing, circuit design, optical engineering and image processing. The most natural way to formulate such problems is by searching for sparse, or parsimonious, solutions in which the underlying phenomena can be represented using just a few parameters. Accordingly, a natural way to phrase such problems revolves around L0 minimization in which the sparsity of the desired solution is controlled by directly counting the number of non-zero parameters. However, due to the nonconvexity and discontinuity of the L0 norm such optimization problems can be quite difficult. One modern tactic to treat such problems is to leverage convex relaxations, such as exchanging the L0 norm for its convex analog, the L1 norm. However, to guarantee accurate reconstructions for L1 minimization, additional conditions must be imposed, such as the restricted isometry property. Accordingly, in this thesis, we propose a novel extension to current approaches revolving around truncated L1 minimization and demonstrate that such approach can, in important cases, provide a better approximation of L0 minimization. Considering that the nonconvexity of the truncated L1 norm makes truncated l1 minimization unreliable in practice, we further generalize our method to partial L1 minimization to combine the convexity of L1 minimization and the robustness of L0 minimization. In addition, we provide a tractable iterative scheme via the augmented Lagrangian method to solve both optimization problems. Our empirical study on synthetic data and image data shows encouraging results of the proposed partial L1 minimization in comparison to L1 minimization.

High resolution 3D images are of high interest in military operations, where data can be used to classify and identify targets. The Swedish defence research agency (FOI) is interested in the latest research and technologies in this area. A draw- back with normal 3D-laser systems are the lack of high resolution for long range measurements. One technique for high long range resolution laser radar is based on time correlated single photon counting (TCSPC). By repetitively sending out short laser pulses and measure the time of flight (TOF) of single reflected pho- tons, extremely accurate range measurements can be done. A drawback with this method is that it is hard to create single photon detectors with many pixels and high temporal resolution, hence a single detector is used. Scanning an entire scene with one detector is very time consuming and instead, as this thesis is all about, the entire scene can be measured with less measurements than the number of pixels. To do this a technique called compressed sensing (CS) is introduced. CS utilizes that signals normally are compressible and can be represented sparse in some basis representation. CS sets other requirements on the sampling compared to the normal Shannon-Nyquist sampling theorem. With a digital micromirror device (DMD) linear combinations of the scene can be reflected onto the single photon detector, creating scalar intensity values as measurements. This means that fewer DMD-patterns than the number of pixels can reconstruct the entire 3D-scene. In this thesis a computer model of the laser system helps to evaluate different CS reconstruction methods with different scenarios of the laser system and the scene. The results show how many measurements that are required to reconstruct scenes properly and how the DMD-patterns effect the results. CS proves to enable a great reduction, 85 − 95 %, of the required measurements com- pared to pixel-by-pixel scanning system. Total variation minimization proves to be the best choice of reconstruction method. / Högupplösta 3D-bilder är väldigt intressanta i militära operationer där data kan utnyttjas för klassificering och identifiering av mål. Det är av stort intresse hos Totalförsvarets forskningsinstitut (FOI) att undersöka de senaste teknikerna in- om detta område. Ett stort problem med vanliga 3D-lasersystem är att de saknar hög upplösning för långa mätavstånd. En teknik som har hög avståndsupplös- ning är tidskorrelerande enfotonräknare, som kan räkna enstaka fotoner med extremt bra noggrannhet. Ett sådant system belyser en scen med laserljus och mäter sedan reflektionstiden för enstaka fotoner och kan på så sätt mäta avstånd. Problemet med denna metod är att göra detektion av många pixlar när man bara kan använda en detektor. Att skanna en hel scen med en detektor tar väldigt lång tid och istället handlar det här exjobbet om att göra färre mätningar än antalet pixlar, men ändå återskapa hela 3D-scenen. För att åstadkomma detta används en ny teknik kallad Compressed Sensing (CS). CS utnyttjar att mätdata normalt är komprimerbar och skiljer sig från det traditionella Shannon-Nyquists krav på sampling. Med hjälp av ett Digital Micromirror Device (DMD) kan linjärkombi- nationer av scenen speglas ner på enfotondetektorn och med färre DMD-mönster än antalet pixlar kan hela 3D-scenen återskapas. Med hjälp av en egenutvecklad lasermodell evalueras olika CS rekonstruktionsmetoder och olika scenarier av la- sersystemet. Arbetet visar att basrepresentationen avgör hur många mätningar som behövs och hur olika uppbyggnader av DMD-mönstren påverkar resultatet. CS visar sig möjliggöra att 85 − 95 % färre mätningar än antalet pixlar behövs för att avbilda hela 3D-scener. Total variation minimization visar sig var det bästa valet av rekonstruktionsmetod.

Compressed sensing (CS) is a relatively new branch of mathematics with very interesting applications in signal processing, statistics and computer science. This thesis presents some theory of compressed sensing, which allows us to recover (high-dimensional) sparse vectors from (low-dimensional) compressed measurements by solving the L1-minimization problem. A possible application of CS to the problem of error correction is also presented, where sparse vectors are that of arbitrary noise. Successful sparse recovery by L1-minimization relies on certain properties of rectangular matrices. But these matrix properties are extremely subtle and difficult to numerically verify. Therefore, to get an idea of how sparse (or dense) errors can be, numerical simulation of error correction was done. These simulations show the performance of error correction with respect to various levels of error sparsity and matrix dimensions. It turns out that error correction degrades slower for low matrix dimensions than for high matrix dimensions, while for sufficiently sparse errors, high matrix dimensions offer a higher likelihood of guaranteed error correction.

This thesis investigates the computation of piecewise polynomial solutions to ill- conditioned linear systems of equations when noise on the linear measurements is observed. Specifically, we enhance our understanding of and provide qualifications on when such ill-conditioned systems of equations can be solved to a satisfactory accuracy. We show that the conventional condition number of the coefficient matrix is not sufficiently informative in this regard and propose a more relevant conditioning measure that takes into account the decay rate of singular values. We also discuss interactions of several factors affecting the solvability of such systems, including the number of discontinuities in solutions, as well as the distribution of nonzero entries in sparse matrices. In addition, we construct and test an approach for computing piecewise polynomial solutions of highly ill-conditioned linear systems using a randomized, SVD-based truncation, and L1-norm regularization. The randomized truncation is a stabilization technique that helps reduce the cost of the traditional SVD truncation for large and severely ill-conditioned matrices. For L1-minimization, we apply a solver based on the Alternating Direction Method. Numerical results are presented to compare our approach that is faster and can solve larger problems, called RTL1 (randomized truncation L1-minimization), with a well-known solver PP-TSVD.

The objective of the proposed research is to develop a new algorithm for range and Doppler ambiguity resolution in radar detection data using L1 minimization methods for sparse signals and to investigate the properties of such techniques. This novel approach to ambiguity resolution makes use of the sparse measurement structure of the post-detection data in multiple pulse repetition frequency radars and the resulting equivalence of the computationally intractable L0 minimization and the surrogate L1 minimization methods. The ambiguity resolution problem is cast as a linear system of equations which is then solved for the unique sparse solution in the absence of errors. It is shown that the new technique successfully resolves range and Doppler ambiguities and the recovery is exact in the ideal case of no errors in the system. The behavior of the technique is then investigated in the presence of real world data errors encountered in radar measurement and detection process. Examples of such errors include blind zone effects, collisions, false alarms and missed detections. It is shown that the mathematical model consisting of a linear system of equations developed for the ideal case can be adjusted to account for data errors. Empirical results show that the L1 minimization approach also works well in the presence of errors with minor extensions to the algorithm. Several examples are presented to demonstrate the successful implementation of the new technique for range and Doppler ambiguity resolution in pulse Doppler radars.

Le codage parcimonieux permet la reconstruction d'un signal à partir de quelques projections linéaires de celui-ci, sous l'hypothèse que le signal se décompose de manière parcimonieuse, c'est-à-dire avec peu de coefficients, sur un dictionnaire connu. Le codage est simple, et la complexité est déportée sur la reconstruction. Après une explication détaillée du fonctionnement du codage parcimonieux, une présentation de quelques résultats théoriques et quelques simulations pour cerner les performances envisageables, nous nous intéressons à trois problèmes : d'abord, l'étude de conception d'un système permettant le codage d'un signal par une matrice binaire, et des avantages apportés par une telle implémentation. Ensuite, nous nous intéressons à la détermination du dictionnaire de représentation parcimonieuse du signal par des méthodes d'apprentissage. Enfin, nous discutons la possibilité d'effectuer des opérations comme la classification sur le signal sans le reconstruire. / Compressed sensing allows to reconstruct a signal from a few linear projections, under the assumption that the signal can be sparsely represented, that is, with only a few coefficients, on a known dictionary. Coding is very simple and all the complexity is gathered on the reconstruction. After more detailed explanations of the principle of compressed sensing, some theoretic resultats from literature and a few simulations allowing to get an idea of expected performances, we focusson three problems: First, the study for the building of a system using compressed sensing with a binary matrix and the obtained benefits. Then, we have a look at the building of a dictionary for sparse representations of the signal. And lastly, we discuss the possibility of processing signal without reconstruction, with an example in classification.

With the increasing pervasiveness of digital cameras, the Internet, and social networking, there is a growing need to catalog and analyze large collections of photos and videos. In this dissertation, we explore unconstrained still-image and video-based face recognition in real-world scenarios, e.g. social photo sharing and movie trailers, where people of interest are recognized and all others are ignored. In such a scenario, we must obtain high precision in recognizing the known identities, while accurately rejecting those of no interest. Recent advancements in face recognition research has seen Sparse Representation-based Classification (SRC) advance to the forefront of competing methods. However, its drawbacks, slow speed and sensitivity to variations in pose, illumination, and occlusion, have hindered its wide-spread applicability. The contributions of this dissertation are three-fold: 1. For still-image data, we propose a novel Linearly Approximated Sparse Representation-based Classification (LASRC) algorithm that uses linear regression to perform sample selection for l1-minimization, thus harnessing the speed of least-squares and the robustness of SRC. On our large dataset collected from Facebook, LASRC performs equally to standard SRC with a speedup of 100-250x. 2. For video, applying the popular l1-minimization for face recognition on a frame-by-frame basis is prohibitively expensive computationally, so we propose a new algorithm Mean Sequence SRC (MSSRC) that performs video face recognition using a joint optimization leveraging all of the available video data and employing the knowledge that the face track frames belong to the same individual. Employing MSSRC results in a speedup of 5x on average over SRC on a frame-by-frame basis. 3. Finally, we make the observation that MSSRC sometimes assigns inconsistent identities to the same individual in a scene that could be corrected based on their visual similarity. Therefore, we construct a probabilistic affinity graph combining appearance and co-occurrence similarities to model the relationship between face tracks in a video. Using this relationship graph, we employ random walk analysis to propagate strong class predictions among similar face tracks, while dampening weak predictions. Our method results in a performance gain of 15.8% in average precision over using MSSRC alone.

A surface reconstruction and image enhancement non-linear finite element technique based on minimization of L1 norm of the total variation of the gradient is introduced. Since minimization in the L1 norm is computationally expensive, we seek to improve the performance of this algorithm in two fronts: first, local L1- minimization, which allows parallel implementation; second, application of the Augmented Lagrangian method to solve the minimization problem. We show that local solution of the minimization problem is feasible. Furthermore, the Augmented Lagrangian method can successfully be used to solve the L1 minimization problem. This result is expected to be useful for improving algorithms computing digital elevation maps for natural and urban terrain, fitting surfaces to point-cloud data, and image super-resolution.

One of the key areas that have attracted attention in the construction industry today is Structural Health Monitoring, more commonly known as SHM. It is a concept developed to monitor the quality and longevity of various engineering structures. The incorporation of such a system would help to continuously track health of the structure, indicate the occurrence/presence of any damage in real time and give us an idea of the number of useful years for the same. Being a recently conceived idea, the state of the art technique in the field is straight forward - populating a given structure with sensors and extracting information from them. In this regard, instrumenting with too many sensors may be inefficient as this could lead to superfluous data that is expensive to capture and process. This research aims to explore an alternate SHM technique that optimizes the data acquisition process by eliminating the amount of redundant data that is sensed and uses this sufficient data to detect and locate the fault present in the structure. Efficient data acquisition requires a mechanism that senses just the necessary amount of data for detection and location of fault. For this reason Compressive Sensing (CS) is explored as a plausible idea. CS claims that signals can be reconstructed from what was previously believed to be incomplete information by Shannon's theorem, taking only a small amount of random and linear non - adaptive measurements. As responses of many physical systems contain a finite basis, CS exploits this feature and determines the sparse solution instead of the traditional least - squares type solution.As a first step, CS is demonstrated by successfully recovering the frequency components of a simple sinusoid. Next, the question of how CS compares with the conventional Fourier transform is analyzed. For this, recovery of temporal frequencies and signal reconstruction is performed using the same number of samples for both the approaches and the errors are compared. On the other hand, the FT error is gradually minimized to match that of CS by increasing the number of regularly placed samples. Once the advantages are established, feasibility of using CS to detect damage in a single degree of freedom system is tested under unforced and forced conditions. In the former scenario, damage is indicated when there is a change in natural frequency of vibration of the system after an impact. In the latter, the system is excited harmonically and damage is detected by a change in amplitude of the system's vibration. As systems in real world applications are predominantly multi-DOF, CS is tested on a 2-DOF system excited with a harmonic forcing. Here again, damage detection is achieved by observing the change in the amplitude of vibration of the system. In order to employ CS for detecting either a change in frequency or amplitude of vibration of a structure subjected to realistic forcing conditions, it would be prudent to explore the reconstruction of a signal which contains multiple frequencies. This is accomplished using CS on a chirp signal. Damage detection is clearly a spatio-temporal problem. Hence it is important to additionally explore the extension of CS to spatial reconstruction. For this reason, mode shape reconstruction of a beam with standard boundary conditions is performed and validated with standard/analytical results from literature. As the final step, the operation deflection shapes (ODS) are reconstructed for a simply supported beam using CS to establish that it is indeed a plausible approach for a less expensive SHM. While experimenting with the idea of spatio-temporal domain, the mode shape as well as the ODS of the given beam are examined under two conditions - undamaged and damaged. Damage in the beam is simulated as a decrease in the stiffness coefficient over a certain number of elements. Although the range of modes to be examined heavily depends on the structure in question, literature suggests that for most practical applications, lower modes are more dominant in indicating damage. For ODS on the other hand, damage is indicated by observing the shift in the recovered spatial frequencies and it is confirmed by the reconstructed response.

This thesis presents the mathematical analysis of a continuous-times system for sparse signal recovery. Sparse recovery arises in Compressed Sensing (CS), where signals of large dimension must be recovered from a small number of linear measurements, and can be accomplished by solving a complex optimization program. While many solvers have been proposed and analyzed to solve such programs in digital, their high complexity currently prevents their use in real-time applications. On the contrary, a continuous-time neural network implemented in analog VLSI could lead to significant gains in both time and power consumption. The contributions of this thesis are threefold. First, convergence results for neural networks that solve a large class of nonsmooth optimization programs are presented. These results extend previous analysis by allowing the interconnection matrix to be singular and the activation function to have many constant regions and grow unbounded. The exponential convergence rate of the networks is demonstrated and an analytic expression for the convergence speed is given. Second, these results are specialized to the L1-minimization problem, which is the most famous approach to solving the sparse recovery problem. The analysis relies on standard techniques in CS and proves that the network takes an efficient path toward the solution for parameters that match results obtained for digital solvers. Third, the convergence rate and accuracy of both the continuous-time system and its discrete-time equivalent are derived in the case where the underlying sparse signal is time-varying and the measurements are streaming. Such a study is of great interest for practical applications that need to operate in real-time, when the data are streaming at high rates or the computational resources are limited. As a conclusion, while existing analysis was concentrated on discrete-time algorithms for the recovery of static signals, this thesis provides convergence rate and accuracy results for the recovery of static signals using a continuous-time solver, and for the recovery of time-varying signals with both a discrete-time and a continuous-time solver.