The field of compressive sensing deals with the recovery of a sparse signal from a small set of measurements or linear projections of the signal. In this thesis, we introduce a stochastic framework that allows a collection of correlated sparse signals to be recovered by exploiting both intra and inter signal correlation. Our approach differs from others by not assuming that the collection of sparse signals have a common support or a common component; in some cases, this assumption does not hold true. Imagine a simplified cognitive radio problem, where users can send a single tone (sine-wave) in a finite number of frequencies; it is desired to find the used frequencies over a large area (creation of a radio map). This is a sparse problem; however, as we move spatially, the occuppied frequencies change, thus voiding the assumption of a common support/component. Our solution to multi sparse signal recovery addresses this problem, where signals that are close geographically are highly correlated and their support gradually changes as the distance between signals grow. Our approach consists of the creation of a probabilistic model that accounts for inter and intra signal correlation and then using belief propagation to calculate the posterior distribution of the signals and perform recovery.

The field of compressive sensing deals with the recovery of a sparse signal from a small set of measurements or linear projections of the signal. In this thesis, we introduce a stochastic framework that allows a collection of correlated sparse signals to be recovered by exploiting both intra and inter signal correlation. Our approach differs from others by not assuming that the collection of sparse signals have a common support or a common component; in some cases, this assumption does not hold true. Imagine a simplified cognitive radio problem, where users can send a single tone (sine-wave) in a finite number of frequencies; it is desired to find the used frequencies over a large area (creation of a radio map). This is a sparse problem; however, as we move spatially, the occuppied frequencies change, thus voiding the assumption of a common support/component. Our solution to multi sparse signal recovery addresses this problem, where signals that are close geographically are highly correlated and their support gradually changes as the distance between signals grow. Our approach consists of the creation of a probabilistic model that accounts for inter and intra signal correlation and then using belief propagation to calculate the posterior distribution of the signals and perform recovery.

We propose sparse-matrix belief propagation, which executes loopy belief propagation in Markov random fields by replacing indexing over graph neighborhoods with sparse-matrix operations. This abstraction allows for seamless integration with optimized sparse linear algebra libraries, including those that perform matrix and tensor operations on modern hardware such as graphical processing units (GPUs). The sparse-matrix abstraction allows the implementation of belief propagation in a high-level language (e.g., Python) that is also able to leverage the power of GPU parallelization. We demonstrate sparse-matrix belief propagation by implementing it in a modern deep learning framework (PyTorch), measuring the resulting massive improvement in running time, and facilitating future integration into deep learning models. / Master of Science

As wireless communication systems continue to push the limits of energy and spectral efficiency, increased demands are placed on the capabilities of the receiver. At the same time, the computational resources available for processing received signals will continue to grow. This opens the door for iterative algorithms to play an increasing role in the next generation of communication receivers. In the context of receivers, the goal of iterative probabilistic processing is to approximate maximum a posteriori (MAP) symbol-by-symbol detection of the information bits and estimation of the unknown channel or signal parameters. The sum-product algorithm is capable of efficiently approximating the marginal posterior probabilities desired for MAP detection and provides a unifying framework for the development of iterative receiver algorithms. However, in some applications the sum-product algorithm is computationally infeasible. Specifically, this is the case when both continuous and discrete parameters are present within the model. Also, the complexity of the sum-product algorithm is exponential in the number of variables connected to a particular factor node and can be prohibitive in multi-user and multi-antenna applications. In this dissertation we identify three key problems which can benefit from iterative probabilistic processing, but for which the sum-product algorithm is too complex. They are (1) joint synchronization and detection in multipath channels with emphasis on frame timing, (2) detection in co-channel interference and non-Gaussian noise, and (3) joint channel estimation and multi-signal detection. This dissertation presents the advances we have made in iterative probabilistic processing in order to tackle these problems. The motivation behind the work is to (a) compromise as little as possible on the performance that is achieved while limiting the computational complexity and (b) maintain good theoretical justification to the algorithms that are developed. / Ph. D.

Accurate and efficient localization of facial features is a crucial first step in many face-related computer vision tasks. Some of these tasks include, but not limited to: identity recognition, expression recognition, and head-pose estimation. Most effort in the field has been exerted towards developing better ways of modeling prior appearance knowledge and image observations. Modeling prior shape knowledge, on the other hand, has not been explored as much. In this dissertation I primarily focus on the limitations of the existing methods in terms of modeling the prior shape knowledge. I first introduce a new pose-constrained shape model. I describe my shape model as being "highly flexible yet sufficiently strict". Existing pose-constrained shape models are either too strict, and have questionable generalization power, or they are too loose, and have questionable localization accuracies. My model tries to find a good middle-ground by learning which shape constraints are more "informative" and should be kept, and which ones are not-so-important and may be omitted. I build my pose-constrained facial feature localization approach on this new shape model using a probabilistic graphical model framework. Within this framework, observed and unobserved variables are defined as the local image observations, and the feature locations, respectively. Feature localization, or "probabilistic inference", is then achieved by nonparametric belief propagation. I show that this approach outperforms other popular pose-constrained methods through qualitative and quantitative experiments. Next, I expand my pose-constrained localization approach to unconstrained setting using a multi-model strategy. While doing so, once again I identify and address the two key limitations of existing multi-model methods: 1) semantically and manually defining the models or "guiding" their generation, and 2) not having efficient and effective model selection strategies. First, I introduce an approach based on unsupervised clustering where the models are automatically learned from training data. Then, I complement this approach with an efficient and effective model selection strategy, which is based on a multi-class naive Bayesian classifier. This way, my method can have many more models, each with a higher level of expressive power, and consequently, provides a more effective partitioning of the face image space. This approach is validated through extensive experiments and comparisons with state-of-the-art methods on state-of-the-art datasets. In the last part of this dissertation I discuss a particular application of the previously introduced techniques; facial feature localization in unconstrained videos. I improve the frame-by-frame localization results, by estimating the actual head-movement from a sequence of noisy head-pose estimates, and then using this information for detecting and fixing the localization failures. / text

ITC/USA 2010 Conference Proceedings / The Forty-Sixth Annual International Telemetering Conference and Technical Exhibition / October 25-28, 2010 / Town and Country Resort & Convention Center, San Diego, California / Decoding and Turbo Equalization (TEQ) algorithms based on the Sum-Product Algorithm (SPA) are well established for LDPC codes. However there is increasing interest in linear and nonlinear programming (NLP)-based decoders which may offer computational and performance advantages over the SPA. We present NLP decoders and Turbo equalizers based on an Augmented Lagrangian formulation of the decoding problem. The decoders update estimates of both the Lagrange multipliers and transmitted codeword while solving an approximate quadratic programming problem. Simulation results show that the NLP decoder performance is intermediate between the SPA and bit-flipping algorithms. The NLP may thus be attractive in some applications as it eliminates the tanh/atanh computations in the SPA.

"In modern wireless communication systems, intersymbol interference (ISI) introduced by frequency selective fading is one of the major impairments to reliable data communication. In ISI channels, the receiver observes the superposition of multiple delayed reflections of the transmitted signal, which will result errors in the decision device. As the data rate increases, the effect of ISI becomes severe. To combat ISI, equalization is usually required for symbol detectors. The optimal maximum-likelihood sequence estimation (MLSE) based on the Viterbi algorithm (VA) may be used to estimate the transmitted sequence in the presence of the ISI. However, the computational complexity of the MLSE increases exponentially with the length of the channel impulse response (CIR). Even in channels which do not exhibit significant time dispersion, the length of the CIR will effectively increase as the sampling rate goes higher. Thus the optimal MLSE is impractical to implement in the majority of practical wireless applications. This dissertation is devoted to exploring practically implementable symbol detectors with near-optimal performance in wireless ISI channels. Particularly, we focus on the design and implementation of an iterative detector based on the belief propagation (BP) algorithm. The advantage of the BP detector is that its complexity is solely dependent on the number of nonzero coefficients in the CIR, instead of the length of the CIR. We also extend the work of BP detector design for various wireless applications. Firstly, we present a partial response BP (PRBP) symbol detector with near-optimal performance for channels which have long spanning durations but sparse multipath structure. We implement the architecture by cascading an adaptive linear equalizer (LE) with a BP detector. The channel is first partially equalized by the LE to a target impulse response (TIR) with only a few nonzero coefficients remaining. The residual ISI is then canceled by a more sophisticated BP detector. With the cascaded LE-BP structure, the symbol detector is capable to achieve a near-optimal error rate performance with acceptable implementation complexity. Moreover, we present a pipeline high-throughput implementation of the detector for channel length 30 with quadrature phase-shift keying (QPSK) modulation. The detector can achieve a maximum throughput of 206 Mb/s with an estimated core area of 3.162 mm^{2} using 90-nm technology node. At a target frequency of 515 MHz, the dynamic power is about 1.096 W. Secondly, we investigate the performance of aforementioned PRBP detector under a more generic 3G channel rather than the sparse channel. Another suboptimal partial response maximum-likelihood (PRML) detector is considered for comparison. Similar to the PRBP detector, the PRML detector also employs a hybrid two-stage scheme, in order to allow a tradeoff between performance and complexity. In simulations, we consider a slow fading environment and use the ITU-R 3G channel models. From the numerical results, it is shown that in frequency-selective fading wireless channels, the PRBP detector provides superior performance over both the traditional minimum mean squared error linear equalizer (MMSE-LE) and the PRML detector. Due to the effect of colored noise, the PRML detector in fading wireless channels is not as effective as it is in magnetic recording applications. Thirdly, we extend our work to accommodate the application of Advanced Television Systems Committee (ATSC) digital television (DTV) systems. In order to reduce error propagation caused by the traditional decision feedback equalizer (DFE) in DTV receiver, we present an adaptive decision feedback sparsening filter BP (DFSF-BP) detector, which is another form of PRBP detector. Different from the aforementioned LE-BP structure, in the DFSF-BP scheme, the BP detector is followed by a nonlinear filter called DFSF as the partial response equalizer. In the first stage, the DFSF employs a modified feedback filter which leaves the strongest post-cursor ISI taps uncorrected. As a result, a long ISI channel is equalized to a sparse channel having only a small number of nonzero taps. In the second stage, the BP detector is applied to mitigate the residual ISI. Since the channel is typically time-varying and suffers from Doppler fading, the DFSF is adapted using the least mean square (LMS) algorithm, such that the amplitude and the locations of the nonzero taps of the equalized sparse channel appear to be fixed. As such, the channel appears to be static during the second stage of equalization which consists of the BP detector. Simulation results demonstrate that the proposed scheme outperforms the traditional DFE in symbol error rate, under both static channels and dynamic ATSC channels. Finally, we study the symbol detector design for cooperative communications, which have attracted a lot of attention recently for its ability to exploit increased spatial diversity available at distributed antennas on other nodes. A system framework employing non-orthogonal amplify-and-forward half-duplex relays through ISI channels is developed. Based on the system model, we first design and implement an optimal maximum-likelihood detector based on the Viterbi algorithm. As the relay period increases, the effective CIR between the source and the destination becomes long and sparse, which makes the optimal detector impractical to implement. In order to achieve a balance between the computational complexity and performance, several sub-optimal detectors are proposed. We first present a multitrellis Viterbi algorithm (MVA) based detector which decomposes the original trellis into multiple parallel irregular sub-trellises by investigating the dependencies between the received symbols. Although MVA provides near-optimal performance, it is not straightforward to decompose the trellis for arbitrary ISI channels. Next, the decision feedback sequence estimation (DFSE) based detector and BP-based detector are proposed for cooperative ISI channels. Traditionally these two detectors are used with fixed, static channels. In our model, however, the effective channel is periodically time-varying, even when the component channels themselves are static. Consequently, we modify these two detector to account for cooperative ISI channels. Through simulations in frequency selective fading channels, we demonstrate the uncoded performance of the DFSE detector and the BP detector when compared to the optimal MLSE detector. In addition to quantifying the performance of these detectors, we also include an analysis of the implementation complexity as well as a discussion on complexity/performance tradeoffs."

"A comprehensive analysis of a novel detection scheme for SISO wireless transmission scenarios is presented in this dissertation. The scheme, which is based on Belief-Propagation (BP) detectors, is evaluated in both a computer simulation environment and a custom-built software-defined radio test-bed. In this dissertation, we address the design aspects of BP-based receivers, including several approaches to minimize the bit error rate of MAP detectors. We also present the development of an interface framework for a software defined radio platform that aims to implement complex communication transceivers capable of prototyping the hybrid structure with a pre-filter filter and BP detector. Numerical simulations compared the proposed schemes with an existing approaches and showed significant performance gains without requiring great computational cost at the receiver. Furthermore, experiments using GNU Radio Companion and the FMCOMMS software defined radio hardware platform confirm the correct functionality of the proposed interface, and stress tests are conducted to assess the functionality of the interface and how it deteriorates across a range of operating conditions. Finally, we present several experiments using the FMCOMMS software defined radio platform that implement the proposed BP-based receiver scheme and discuss its capabilities and limitations."

In many applications of graphical models arising in computer vision, the hidden variables of interest are most naturally specified by continuous, non-Gaussian distributions. There exist inference algorithms for discrete approximations to these continuous distributions, but for the high-dimensional variables typically of interest, discrete inference becomes infeasible. Stochastic methods such as particle filters provide an appealing alternative. However, existing techniques fail to exploit the rich structure of the graphical models describing many vision problems. Drawing on ideas from regularized particle filters and belief propagation (BP), this paper develops a nonparametric belief propagation (NBP) algorithm applicable to general graphs. Each NBP iteration uses an efficient sampling procedure to update kernel-based approximations to the true, continuous likelihoods. The algorithm can accomodate an extremely broad class of potential functions, including nonparametric representations. Thus, NBP extends particle filtering methods to the more general vision problems that graphical models can describe. We apply the NBP algorithm to infer component interrelationships in a parts-based face model, allowing location and reconstruction of occluded features.

This memo describes the initial results of a project to create a self-supervised algorithm for learning object segmentation from video data. Developmental psychology and computational experience have demonstrated that the motion segmentation of objects is a simpler, more primitive process than the detection of object boundaries by static image cues. Therefore, motion information provides a plausible supervision signal for learning the static boundary detection task and for evaluating performance on a test set. A video camera and previously developed background subtraction algorithms can automatically produce a large database of motion-segmented images for minimal cost. The purpose of this work is to use the information in such a database to learn how to detect the object boundaries in novel images using static information, such as color, texture, and shape. This work was funded in part by the Office of Naval Research contract #N00014-00-1-0298, in part by the Singapore-MIT Alliance agreement of 11/6/98, and in part by a National Science Foundation Graduate Student Fellowship.