Estimation of nonlinear models finds applications in every field of engineering and the sciences. Much work has been done to build solid statistical theories for its use and interpretation. However, there has been little analysis of the tolerance of nonlinear model estimators to deviations from assumptions and normality. We focus on analyzing the robustness properties of M-estimators of nonlinear models by studying the effects of deviations from assumptions and normality on these estimators. We discuss St. Laurent and Cook's Jacobian Leverage and identify the relationship of the technique to the robustness concept of influence. We derive influence functions for M-estimators of nonlinear models and show that influence of position becomes, more generally, influence of model. The result shows that, for M-estimators, we must bound not only influence of residual but also influence of model. Several examples highlight the unique problems of nonlinear model estimation and demonstrate the utility of the influence function. / Master of Science

The presence of noise and corruption in real-world data can be inevitably caused by accidental outliers, transmission loss, or even adversarial data attacks. Unlike traditional random noise usually assume a specific distribution with low corruption ratio, the data collected from crowdsourcing or labeled by weak annotators can contain adversarial data corruption. More challenge, the adversarial data corruption can be arbitrary, unbounded and do not follow any specific distribution. In addition, in the era of data explosion, the fast-growing amount of data makes the robust models more difficult to handle large-scale data sets. This thesis focuses on the development of methods for scalable robust models under the adversarial data corruption assumptions. Four methods are proposed, including robust regression via heuristic hard-thresholding, online and distributed robust regression with adversarial noises, self-paced robust learning for leveraging clean labels in noisy data, and robust regression via online feature selection with adversarial noises. Moreover, I extended the self-paced robust learning method to its distributed version for the scalability of the proposed algorithm, named distributed self-paced learning in alternating direction method of multiplier. Last, a robust multi-factor personality prediction model is proposed to hand the correlated data noises. For the first method, existing solutions for robust regression lack rigorous recovery guarantee of regression coefficients under the adversarial data corruption with no prior knowledge of corruption ratio. The proposed contributions of our work include: (1) Propose efficient algorithms to address the robust least-square regression problem; (2) Design effective approaches to estimate the corruption ratio; (3) Provide a rigorous robustness guarantee for regression coefficient recovery; and (4) Conduct extensive experiments for performance evaluation. For the second method, existing robust learning methods typically focus on modeling the entire dataset at once; however, they may meet the bottleneck of memory and computation as more and more datasets are becoming too large to be handled integrally. The proposed contributions of our work for this task include: (1) Formulate a framework for the scalable robust least-squares regression problem; (2) Propose online and distributed algorithms to handle the adversarial corruption; (3) Provide a rigorous robustness guarantee for regression coefficient recovery; and (4) Conduct extensive experiments for performance evaluations. For the third method, leveraging the prior knowledge of clean labels in noisy data is actually a crucial issue in practice, but existing robust learning methods typically focus more on eliminating noisy data. However, the data collected by ``weak annotator" or crowd-sourcing can be too noisy for existing robust methods to train an accurate model. Moreover, existing work that utilize additional clean labels are usually designed for some specific problems such as image classification. These methods typically utilize clean labels in large-scale noisy data based on their additional domain knowledge; however, these approaches are difficult to handle extremely noisy data and relied on their domain knowledge heavily, which makes them difficult be used in more general problems. The proposed contributions of our work for this task include: (1) Formulating a framework to leverage the clean labels in noisy data; (2) Proposing a self-paced robust learning algorithm to train models under the supervision of clean labels; (3) Providing a theoretical analysis for the convergence of the proposed algorithm; and (4) Conducting extensive experiments for performance evaluations. For the fourth method, the presence of data corruption in user-generated streaming data, such as social media, motivates a new fundamental problem that learns reliable regression coefficient when features are not accessible entirely at one time. Until now, several important challenges still cannot be handled concurrently: 1) corrupted data estimation when only partial features are accessible; 2) online feature selection when data contains adversarial corruption; and 3) scaling to a massive dataset. This paper proposes a novel RObust regression algorithm via Online Feature Selection (textit{RoOFS}) that concurrently addresses all the above challenges. Specifically, the algorithm iteratively updates the regression coefficients and the uncorrupted set via a robust online feature substitution method. We also prove that our algorithm has a restricted error bound compared to the optimal solution. Extensive empirical experiments in both synthetic and real-world data sets demonstrated that the effectiveness of our new method is superior to that of existing methods in the recovery of both feature selection and regression coefficients, with very competitive efficiency. For the fifth method, existing self-paced learning approaches typically focus on modeling the entire dataset at once; however, this may introduce a bottleneck in terms of memory and computation, as today's fast-growing datasets are becoming too large to be handled integrally. The proposed contributions of our work for this task include: (1) Reformulate the self-paced problem into a distributed setting.; (2) A distributed self-paced learning algorithm based on consensus ADMM is proposed to solve the textit{SPL} problem in a distributed setting; (3) A theoretical analysis is provided for the convergence of our proposed textit{DSPL} algorithm; and (4) Extensive experiments have been conducted utilizing both synthetic and real-world data based on a robust regression task. For the last method, personality prediction in multiple factors, such as openness and agreeableness, is growing in interest especially in the context of social media, which contains massive online posts or likes that can potentially reveal an individual's personality. However, the data collected from social media inevitably contains massive amounts of noise and corruption. To address it, traditional robust methods still suffer from several important challenges, including 1) existence of correlated corruption among multiple factors, 2) difficulty in estimating the corruption ratio in multi-factor data, and 3) scalability to massive datasets. This paper proposes a novel robust multi-factor personality prediction model that concurrently addresses all the above challenges by developing a distributed robust regression algorithm. Specifically, the algorithm optimizes regression coefficients of each factor in parallel with a heuristically estimated corruption ratio and then consolidates the uncorrupted set from multiple factors in two strategies: global consensus and majority voting. We also prove that our algorithm benefits from strong guarantees in terms of convergence rates and coefficient recovery, which can be utilized as a generic framework for the multi-factor robust regression problem with correlated corruption property. Extensive experiment on synthetic and real dataset demonstrates that our algorithm is superior to those of existing methods in both effectiveness and efficiency. / Doctor of Philosophy / Social media has experienced a rapid growth during the past decade. Millions of users of sites such as Twitter have been generating and sharing a wide variety of content including texts, images, and other metadata. In addition, social media can be treated as a social sensor that reflects different aspects of our society. Event analytics in social media have enormous significance for applications like disease surveillance, business intelligence, and disaster management. Social media data possesses a number of important characteristics including dynamics, heterogeneity, noisiness, timeliness, big volume, and network properties. These characteristics cause various new challenges and hence invoke many interesting research topics, which will be addressed here. This dissertation focuses on the development of five novel methods for social media-based spatiotemporal event detection and forecasting. The first of these is a novel unsupervised approach for detecting the dynamic keywords of spatial events in targeted domains. This method has been deployed in a practical project for monitoring civil unrest events in several Latin American regions. The second builds on this by discovering the underlying development progress of events, jointly considering the structural contexts and spatiotemporal burstiness. The third seeks to forecast future events using social media data. The basic idea here is to search for subtle patterns in specific cities as indicators of ongoing or future events, where each pattern is defined as a burst of context features (keywords) that are relevant to a specific event. For instance, an initial expression of discontent gas price increases could actually be a potential precursor to a more general protest about government policies. Beyond social media data, in the fourth method proposed here, multiple data sources are leveraged to reflect different aspects of the society for event forecasting. This addresses several important problems, including the common phenomena that different sources may come from different geographical levels and have different available time periods. The fifth study is a novel flu forecasting method based on epidemics modeling and social media mining. A new framework is proposed to integrate prior knowledge of disease propagation mechanisms and real-time information from social media.

This thesis presents a new robust filtering technique that suppresses impulsive noise in speech signals. The method makes use of Projection Statistics based on medians to detect segments of speech with impulses. The autoregressive model employed to smooth out the speech signal is identified by means of a robust nonlinear estimator known as the Schweppe-type Huber GM-estimator. Simulation results are presented that demonstrate the effectiveness of the filter. Another contribution of the work is the development of a robust version of the Kalman filter based on the Huber M-estimator. The performances of this filter are evaluated for a simple autoregressive process. / Master of Science

In this thesis we demonstrate a technique for robust controller design for an autonomous underwater vehicle (AUV) that explicitly handles the trade-off between reference tracking, agility, and energy efficient performance. AUVs have many sources of modeling uncertainty that impact the uncertainty in maneuvering performance. A robust control design process is proposed to handle these uncertainties while meeting control system performance objectives. We investigate the relationships between linear system design parameters and the control performance of our vehicle in order to inform an H∞ controller synthesis problem with the objective of balancing these tradeoffs. We evaluate the controller based on its reference tracking performance, agility and energy efficiency, and show the efficacy of our control design strategy. / Master of Science / In this thesis we demonstrate a technique for autopilot design for an autonomous underwater vehicle (AUV) that explicitly handles the trade-off between three performance metrics. Mathematical models of AUVs are often unable to fully describe their many physical properties. The discrepancies between the mathematical model and reality impact how certain we can be about an AUV's behavior. Robust controllers are a class of controller that are designed to handle uncertainty. A robust control design process is proposed to handle these uncertainties while meeting vehicle performance objectives. We investigate the relationships between design parameters and the performance of our vehicle. We then use this relationship to inform the design of a controller. We evaluate this controller based on its energy efficiency, agility and ability to stay on course, and thus show the effectiveness of our control design strategy.

This thesis concerns optimization of structures considering various uncertainties. The overall objective is to find methods to create solutions that are optimal both in the sense of handling the typical load case and minimising the variability of the response, i.e. robust optimal designs. Traditionally optimized structures may show a tendency of being sensitive to small perturbations in the design or loading conditions, which of course are inevitable. To create robust designs, it is necessary to account for all conceivable variations (or at least the influencing ones) in the design process. The thesis is divided in two parts. The first part serves as a theoretical background to the second part, the two appended articles. This first part includes the concept of robust design, basic statistics, optimization theory and meta modelling. The first appended paper is an application of existing methods on a large industrial example problem. A sensitivity analysis is performed on a Scania truck cab subjected to impact loading in order to identify the most influencing variables on the crash responses. The second paper presents a new method that may be used in robust optimizations, that is, optimizations that account for variations and uncertainties. The method is demonstrated on both an analytical example and a Finite Element example of an aluminium extrusion subjected to axial crushing. / ROBDES

There are two main approaches to robust stability analysis: the input-output stability framework with scaling or multiplier, and the Lyapunov functions. Analysis methods in these two directions are usually developed independently, and the relationship between the two is not clear except for some special cases. This motivates us to study the relationship between the two approaches. The generalized performance problem refers to certain frequency-domain conditions on a transfer matrix. We prove the equivalent relationship between generalized performance and robust stability under certain assumptions. The definition of generalized performance requires the internal stability of a transfer matrix, which is not a necessity for robust stability. In view of this, we derive new frequency-domain conditions for robust stability without this requirement. Our result contains a version of the circle criterion as a special case. To tackle the generalized performance problem, we propose a version of the Kalman-Yakubovich-Popov (KYP) lemma to transform the frequency-domain conditions into linear matrix inequalities (LMIs). The proposed LMI condition is then connected to the quadratic stability of an uncertain linear system. Combining the derived results gives a clear picture of the relationships between robust stability, generalized performance, quadratic stability, and KYP lemma. The connections not only unify some previous results but also extend those results to more general stability regions and types of uncertainty. In addition to robust stability analysis, we also tackle the corresponding synthesis problem, i.e. robust pole placement. The desired region for robust pole placement can be the intersection or the union of simple regions. (Simple regions are the half plane, the disk, and the outside of a disk.) One contribution of our synthesis result is that the desired region can be non-convex¡Xmost results on robust pole placement focus on convex regions only. Two examples of the longitudinal control of a combat aircraft and the attitude control of a satellite demonstrate the effectiveness of our result.

<p>This thesis concerns optimization of structures considering various uncertainties. The overall objective is to find methods to create solutions that are optimal both in the sense of handling the typical load case and minimising the variability of the response, i.e. robust optimal designs.</p><p>Traditionally optimized structures may show a tendency of being sensitive to small perturbations in the design or loading conditions, which of course are inevitable. To create robust designs, it is necessary to account for all conceivable variations (or at least the influencing ones) in the design process.</p><p>The thesis is divided in two parts. The first part serves as a theoretical background to the second part, the two appended articles. This first part includes the concept of robust design, basic statistics, optimization theory and meta modelling.</p><p>The first appended paper is an application of existing methods on a large industrial example problem. A sensitivity analysis is performed on a Scania truck cab subjected to impact loading in order to identify the most influencing variables on the crash responses.</p><p>The second paper presents a new method that may be used in robust optimizations, that is, optimizations that account for variations and uncertainties. The method is demonstrated on both an analytical example and a Finite Element example of an aluminium extrusion subjected to axial crushing.</p> / ROBDES

In wireless communication systems, relays are widely used to extend coverage. Over the past years, relays have evolved from simple repeaters to more sophisticated units that perform signal processing to improve signal to interference plus noise ratio (SINR) or throughput (or both) at the destination receiver. There are various types of relays such as amplify and forward (AF), decode and forward (DF), and compress and forward (CF) (or estimate and forward (EF)) relays. In addition, recently there has been a growing interest in two-way relays (TWR). By utilizing the concept of analog network coding (ANC), TWRs can improve the throughput of a wireless sys- tem by reducing the number of time slots needed to complete a bi-directional message exchange between two destination nodes. It’s well known that the performance of a TWR system greatly depends on its ability to apply signal processing techniques to effectively mitigate the self-interference and noise accumulation, thereby improving the SINR. We study a TWR system that is equipped with multiple antennas at the relay node and a single antenna at the two destination nodes. Different from traditional work on TWR, we focus on the case with imperfect knowledge of channel state information (CSI). For such a TWR, we formulate a robust optimization problem that takes into ac- count norm-bounded estimation errors in CSI and designs an optimal beamforming matrix. Realizing the fact that this problem is extremely hard to solve globally, we derive two different methods to obtain either optimal or efficient suboptimal beam- forming matrix solutions. The first method involves solving the robust optimization problem using the S-procedure and semidefinite programming (SDP) with rank-one relaxation. This method provides an optimal solution when the rank-one relaxation condition for the SDP is satisfied. In cases where the rank-one condition cannot be satisfied, it’s necessary to resort to sub-optimal techniques. The second approach presented here reformulates the robust non-convex quadratically constrained quadratic programming (QCQP) into a robust linear programming (LP) problem by using first-order perturbation of the optimal non-robust beamforming solution (which assumes no channel estimation error). Finally, we view the TWR robust beamforming problem from a practical standpoint and develop a set of iterative algorithms based on Newton’s method or the steepest descent method that are practical for hardware implementation.

The design exploration method for adaptive design systems is developed to facilitate the pursuit of a balance between the efficiency and accuracy in systems engineering design. The proposed method is modified from an existing multiscale material robust design method, the Inductive Design Exploration Method (IDEM). The IDEM is effective in managing uncertainty propagation in the model chain. However, it is not an appropriate method in other systems engineering design outside of original design domain due to its high computational cost. In this thesis, the IDEM is augmented with more efficient solution search methods to improve its capability for efficiently exploring robust design solutions in systems engineering design. The accuracy of the meta-model in engineering design is one uncertainty source. In current engineering design, response surface model is widely used. However, this method is shown as inaccurate in fitting nonlinear models. In this thesis, the local regression method is introduced as an alternative of meta-modeling technique to reduce the computational cost of simulation models. It is proposed as an appropriate method in systems design with nonlinear simulations models. The proposed methods are tested and verified by application to a Multifunctional Energetic Materials design and a Photonic Crystal Coupler and Waveguide design. The methods are demonstrated through the better accuracy of the local regression model in comparison to the response surface model and the better efficiency of the design exploration method for adaptive design systems in comparison to the IDEM. The proposed methods are validated theoretically and empirically through application of the validation square.

Machine learning algorithms are invented to learn from data and to use data to perform predictions and analyses. Many agencies are now using machine learning algorithms to present services and to perform tasks that used to be done by humans. These services and tasks include making high-stake decisions. Determining the right decision strongly relies on the correctness of the input data. This fact provides a tempting incentive for criminals to try to deceive machine learning algorithms by manipulating the data that is fed to the algorithms. And yet, traditional machine learning algorithms are not designed to be safe when confronting unexpected inputs. In this dissertation, we address the problem of adversarial machine learning; i.e., our goal is to build safe machine learning algorithms that are robust in the presence of noisy or adversarially manipulated data. Many complex questions -- to which a machine learning system must respond -- have complex answers. Such outputs of the machine learning algorithm can have some internal structure, with exponentially many possible values. Adversarial machine learning will be more challenging when the output that we want to predict has a complex structure itself. In this dissertation, a significant focus is on adversarial machine learning for predicting structured outputs. In this thesis, first, we develop a new algorithm that reliably performs collective classification: It jointly assigns labels to the nodes of graphed data. It is robust to malicious changes that an adversary can make in the properties of the different nodes of the graph. The learning method is highly efficient and is formulated as a convex quadratic program. Empirical evaluations confirm that this technique not only secures the prediction algorithm in the presence of an adversary, but it also generalizes to future inputs better, even if there is no adversary. While our robust collective classification method is efficient, it is not applicable to generic structured prediction problems. Next, we investigate the problem of parameter learning for robust, structured prediction models. This method constructs regularization functions based on the limitations of the adversary in altering the feature space of the structured prediction algorithm. The proposed regularization techniques secure the algorithm against adversarial data changes, with little additional computational cost. In this dissertation, we prove that robustness to adversarial manipulation of data is equivalent to some regularization for large-margin structured prediction, and vice versa. This confirms some of the previous results for simpler problems. As a matter of fact, an ordinary adversary regularly either does not have enough computational power to design the ultimate optimal attack, or it does not have sufficient information about the learner's model to do so. Therefore, it often tries to apply many random changes to the input in a hope of making a breakthrough. This fact implies that if we minimize the expected loss function under adversarial noise, we will obtain robustness against mediocre adversaries. Dropout training resembles such a noise injection scenario. Dropout training was initially proposed as a regularization technique for neural networks. The procedure is simple: At each iteration of training, randomly selected features are set to zero. We derive a regularization method for large-margin parameter learning based on dropout. Our method calculates the expected loss function under all possible dropout values. This method results in a simple objective function that is efficient to optimize. We extend dropout regularization to non-linear kernels in several different directions. We define the concept of dropout for input space, feature space, and input dimensions, and we introduce methods for approximate marginalization over feature space, even if the feature space is infinite-dimensional. Empirical evaluations show that our techniques consistently outperform the baselines on different datasets.