Deep neural networks (DNNs) have achieved remarkable success in data-intense applications, while such success relies heavily on massive and carefully labeled data. In practice, obtaining large-scale datasets with correct labels is often expensive, time-consuming, and sometimes even impossible. Common approaches of constructing datasets involve some degree of error-prone processes, such as automatic labeling or crowdsourcing, which inherently introduce noisy labels. It has been observed that noisy labels severely degrade the generalization performance of classifiers, especially the overparameterized (deep) neural networks. Therefore, studying noisy labels and developing techniques for training accurate classifiers in the presence of noisy labels is of great practical significance. In this thesis, we conduct a thorough study to fully understand LNL and provide a comprehensive error decomposition to reveal the core issue of LNL. We then point out that the core issue in LNL is that the empirical risk minimizer is unreliable, i.e., the DNNs are prone to overfitting noisy labels during training. To reduce the learning errors, we propose five different methods, 1) Co-matching: a framework consists of two networks to prevent the model from memorizing noisy labels; 2) SELC: a simple method to progressively correct noisy labels and refine the model; 3) NAL: a regularization method that automatically distinguishes the mislabeled samples and prevents the model from memorizing them; 4) EM-enhanced loss: a family of robust loss functions that not only mitigates the influence of noisy labels, but also avoids underfitting problem; 5) MixNN: a framework that trains the model with new synthetic samples to mitigate the impact of noisy labels. Our experimental results demonstrate that the proposed approaches achieve comparable or better performance than the state-of-the-art approaches on benchmark datasets with simulated label noise and large-scale datasets with real-world label noise. / Dissertation / Doctor of Philosophy (PhD) / Machine Learning has been highly successful in data-intensive applications but is often hampered when datasets contain noisy labels. Recently, Learning with Noisy Labels (LNL) is proposed to tackle this problem. By using techniques from LNL, the models can still generalize well even when trained on the data containing noisy supervised information. In this thesis, we study this crucial problem and provide a comprehensive analysis to reveal the core issue of LNL. We then propose five different methods to effectively reduce the learning errors in LNL. We show that our approaches achieve comparable or better performance compared to the state-of-the-art approaches on benchmark datasets with simulated label noise and real-world noisy datasets.

Top-N recommender systems automatically recommend N items for users from huge amounts of products. Personalized Top-N recommender systems have great impact on many real world applications such as E-commerce platforms and social networks. Sometimes there is no rating information in user-item feedback matrix but only implicit purchase or browsing history, that means the user-item feedback matrix is a binary matrix, we call such feedbacks as implicit feedbacks. In our work we try to learn Top-N recommender systems with implicit feedbacks. First, we design a heterogeneous loss function to learn the model. Second, we incorporate item side information into recommender systems. We formulate a low-rank constraint minimization problem and give a closed-form solution for it. Third, we also use item side information to learn recommender systems. We use gradient descent method to learn our model. Most existing methods produce personalized top-N recommendations by minimizing a specific uniform loss such as pairwise ranking loss or pointwise recovery loss. In our first model, we propose a novel personalized Top-N recommendation approach that minimizes a combined heterogeneous loss based on linear self-recovery models. The heterogeneous loss integrates the strengths of both pairwise ranking loss and pointwise recovery loss to provide more informative recommendation predictions. We formulate the learning problem with heterogeneous loss as a constrained convex minimization problem and develop a projected stochastic gradient descent optimization algorithm to solve it. Most previous systems are only based on the user-item feedback matrix. In many applications, in addition to the user-item rating/purchase matrix, item-based side information such as product reviews, book reviews, item comments, and movie plots can be easily collected from the Internet. This abundant item-based information can be used for recommendation systems. In the second model, we propose a novel predictive collaborative filtering approach that exploits both the partially observed user-item recommendation matrix and the item-based side information to produce top-N recommender systems. The proposed approach automatically identifies the most interesting items for each user from his or her non-recommended item pool by aggregating over his or her recommended items via a low-rank coefficient matrix. Moreover, it also simultaneously builds linear regression models from the item-based side information such as item reviews to predict the item recommendation scores for the users. The proposed approach is formulated as a rank constrained joint minimization problem with integrated least squares losses, for which an efficient analytical solution can be derived. In the third model, we also propose a joint discriminative prediction model that exploits both the partially observed user-item recommendation matrix and the item-based side information to build top-N recommender systems. This joint model aggregates observed user-item recommendation activities to predict the missing/new user-item recommendation scores while simultaneously training a linear regression model to predict the user-item recommendation scores from auxiliary item features. We evaluate the proposed approach on a variety of recommendation tasks. The experimental results show that the proposed joint model is very effective for producing top-N recommendation systems. / Computer and Information Science

According to the Public Health Agency of Canada, falls account for 95% of all hip fractures in Canada; 20% of fall-related injury cases end in death. This thesis evaluates the predictive power of many variables to predict fall-related injuries. The dataset chosen was CCHS which is high dimensional and diverse. The use of Principal Component Analysis (PCA) and random forest was employed to determine the highest priority risk factors to include in the predictive model. The results show that it is possible to predict fall-related injuries with a sensitivity of 80% or higher using four predictors (frequency of consultations with medical doctor, food and vegetable consumption, height and monthly physical activity level of over 15 minutes). Alternatively, the same sensitivity can be reached using age, frequency of walking for exercise per 3 months, alcohol consumption and personal income. None of the predictive models reached an accuracy of 70% or higher. Further work in studying nutritional diets that offer protection from incurring a fall related injury are also recommended. Since the predictors are behavioral determinants of health and have a high sensitivity but a low accuracy, population health interventions are recommended rather than individual-level interventions. Suggestions to improve accuracy of built models are also proposed. / Thesis / Master of Science (MSc)

Applications employing very large datasets are increasingly common in this age of Big Data. While these applications provide great benefits in various domains, their usage can be hampered by real-world privacy and security risks. In this work we propose algorithms which aim to provide privacy and security protection in different aspects of these applications. First, we address the problem of data privacy. When the datasets used contain personal information, they must be properly anonymized in order to protect the privacy of the subjects to which the records pertain. A popular privacy preservation technique is the k-anonymity model which guarantees that any record in the dataset must be indistinguishable from at least k-1 other records in terms of quasi-identifiers (i.e. the subset of attributes that can be used to deduce the identity of an individual). Achieving k-anonymity while considering the competing goal of data utility can be a challenge, especially for datasets containing large numbers of records. We formulate k-anonymization as an optimization problem with the objective to maximize data utility, and propose two practical algorithms for solving this problem. Secondly, we address the problem of application security; specifically, for predictive models using Deep Learning, where adversaries can use minimally perturbed inputs (a.k.a. adversarial examples) to cause a neural network to produce incorrect outputs. We propose an approach which protects against adversarial examples in image classification-type networks. The approach relies on two mechanisms: 1) a mechanism that increases robustness at the expense of accuracy; and, 2) a mechanism that improves accuracy. We show that an approach combining the two mechanisms can provide protection against adversarial examples while retaining accuracy. We provide experimental results to demonstrate the effectiveness of our algorithms for both problems. / Thesis / Master of Science (MSc) / Applications employing very large datasets are increasingly common in this age of Big Data. While these applications provide great benefits in various domains, their usage can be hampered by real-world privacy and security risks. In this work we propose algorithms which aim to provide privacy and security protection in different aspects of these applications. We address the problem of data privacy; when the datasets used contain personal information, they must be properly anonymized in order to protect the privacy of the subjects to which the records pertain. We propose two practical algorithms for anonymization which are also utility-centric. We address the problem of application security, specifically for Deep Learning applications where adversaries can use minimally perturbed inputs to cause a neural network to produce incorrect outputs. We propose an approach which protects against these attacks. We provide experimental results to demonstrate the effectiveness of our algorithms for both problems.

Using computational approaches utilizing large datasets to investigate public health information is an important mechanism for institutions seeking to identify strategies for improving public health. The art in computational approaches, for example in health research, is managing the trade-offs between the two perspectives: first, inference and second, prediction. Many techniques from statistical methods (SM) and machine learning (ML) may, in principle, be used for both perspectives. However, SM has a well established focus on inference by building probabilistic models which allows us to determine a quantitative measure of confidence about the magnitude of the effect. Simulation-based validation approaches can be used in conjunction with SM to explicitly verify assumptions and redefine the specified model, if necessary. On the other hand, ML uses general-purpose algorithms to find patterns that best predict the outcome and makes minimal assumptions about the data-generating process; and may be more effective in a number of situations. My work employs both SM- and ML- based computational approaches to investigate particular public health problems. Chapter One provides philosophical background and compares the application of the two approaches in public health. Chapter Two describes and implements penalized Cox proportional hazard models for time-varying covariates time-to-event data. Chapter Three applies traditional survival models and machine learning algorithms to predict survival times of cancer patients, while incorporating the information about the time-varying covariates. Chapter Four discusses and implements various approaches for computing predictions and effects for generalized linear (mixed) models. Finally, Chapter Five implements and compares various statistical models for handling univariate and multivariate binary outcomes for water, sanitation and hygiene (WaSH) data. / Thesis / Doctor of Philosophy (PhD)

In an attempt to thwart global warming in a concerted way, more than 130 countries have committed to becoming carbon neutral around 2050. In the United States, the Biden ad- ministration has called for 100% clean energy by 2035. It is estimated that in order to meet that target, the energy production from solar and wind should increase to 50-70% from the current 11% share. Under higher penetration of solar and wind, the intermittency of the energy source poses critical problems in forecasting, uncertainty quantification, reserve man- agement, unit commitment, and economic dispatch, and presents unique challenges to the distribution system, including predicting solar adoption by the user as well as forecasting end-use load profiles. While these problems are complex, advances in machine learning and artificial intelligence provide opportunities for novel paradigms for addressing the challenges. The overall aim of the dissertation is to harness data-driven and model-based techniques and develop computationally efficient tools for improved power systems operation under high re- newables penetration in the next-generation electric grid. Some of the salient contributions of this work are the reduction in the number of uncertain scenarios by 99%; dramatic reduc- tion in the computational overhead to simulate stochastic unit commitment and economic dispatch on a single-node electric-grid system to merely 10 seconds from 24 hours; reduc- tion in the total monthly operating cost of two-stage stochastic economic dispatch by an average of 5%, and reduction in average overall reserve due to intermittency in renewables by 50%; and improvement in the existing end-use load prediction and rooftop PV adopter identification tools by a considerable margin. / Doctor of Philosophy / In an attempt to thwart global warming in a concerted way, more than 130 countries have committed to becoming carbon neutral around 2050. In the United States, the Biden ad- ministration has called for 100% clean energy by 2035. It is estimated that in order to meet that target, the energy production from solar and wind should increase to 50-70% from the current 11% share. Under higher penetration of solar and wind, the intermittency of the energy source poses critical problems in forecasting, uncertainty quantification, reserve man- agement, unit commitment, and economic dispatch, and presents unique challenges to the distribution system, including predicting solar adoption by the user as well as forecasting end-use load profiles. While these problems are complex, advances in machine learning and artificial intelligence provide opportunities for novel paradigms for addressing the challenges. The overall aim of the dissertation is to harness data-driven and model-based techniques and develop computationally efficient tools for improved power systems operation under high re- newables penetration in the next-generation electric grid. Some of the salient contributions of this work are the reduction in the number of uncertain scenarios by 99%; dramatic reduc- tion in the computational overhead to simulate stochastic unit commitment and economic dispatch on a single-node electric-grid system to merely 10 seconds from 24 hours; reduc- tion in the total monthly operating cost of two-stage stochastic economic dispatch by an average of 5%, and reduction in average overall reserve due to intermittency in renewables by 50%; and improvement in the existing end-use load prediction and rooftop PV adopter identification tools by a considerable margin.

Modern AI systems, particularly with the rise of big data and deep learning in the last decade, have greatly improved our daily life and at the same time created a long list of controversies. AI systems are often subject to malicious and stealthy subversion that jeopardizes their efficacy. Many of these issues stem from the data-driven nature of machine learning. While big data and deep models significantly boost the accuracy of machine learning models, they also create opportunities for adversaries to tamper with models or extract sensitive data. Malicious data providers can compromise machine learning systems by supplying false data and intermediate computation results. Even a well-trained model can be deceived to misbehave by an adversary who provides carefully designed inputs. Furthermore, curious parties can derive sensitive information of the training data by interacting with a machine-learning model. These adversarial scenarios, known as poisoning attack, adversarial example attack, and inference attack, have demonstrated that security, privacy, and robustness have become more important than ever for AI to gain wider adoption and societal trust. To address these problems, we proposed the following solutions: (1) FLARE, which detects and mitigates stealthy poisoning attacks by leveraging latent space representations; (2) MANDA, which detects adversarial examples by utilizing evaluations from diverse sources, i.e, model-based prediction and data-based evaluation; (3) FeCo which enhances the robustness of machine learning-based network intrusion detection systems by introducing a novel representation learning method; and (4) DP-FedMeta, which preserves data privacy and improves the privacy-accuracy trade-off in machine learning systems through a novel adaptive clipping mechanism. / Doctor of Philosophy / Over the past few decades, machine learning (ML) has become increasingly popular for enhancing efficiency and effectiveness in data analytics and decision-making. Notable applications include intelligent transportation, smart healthcare, natural language generation, intrusion detection, etc. While machine learning methods are often employed for beneficial purposes, they can also be exploited for malicious intents. Well-trained language models have demonstrated generalizability deficiencies and intrinsic biases; generative ML models used for creating art have been repurposed by fraudsters to produce deepfakes; and facial recognition models trained on big data have been found to leak sensitive information about data owners. Many of these issues stem from the data-driven nature of machine learning. While big data and deep models significantly improve the accuracy of ML models, they also enable adversaries to corrupt models and infer sensitive data. This leads to various adversarial attacks, such as model poisoning during training, adversarially crafted data in testing, and data inference. It is evident that security, privacy, and robustness have become more important than ever for AI to gain wider adoption and societal trust. This research focuses on building trustworthy machine-learning systems in adversarial environments from a data perspective. It encompasses two themes: securing ML systems against security or privacy vulnerabilities (security of AI) and using ML as a tool to develop novel security solutions (AI for security). For the first theme, we studied adversarial attack detection in both the training and testing phases and proposed FLARE and MANDA to secure matching learning systems in the two phases, respectively. Additionally, we proposed a privacy-preserving learning system, dpfed, to defend against privacy inference attacks. We achieved a good trade-off between accuracy and privacy by proposing an adaptive data clipping and perturbing method. In the second theme, the research is focused on enhancing the robustness of intrusion detection systems through data representation learning.

The application of feedback to enhance motivation is beneficial across various life contexts. While both feedback and motivation have been studied widely in psychological science, most of this research has used close-ended approaches to study feedback empirically, which limits the scope of investigation. The present study was one of the first applications of text-analysis to assess the impact of feedback on the recipient's motivation. A transformer machine-learning model was used to create a tool that can predict the average motivating influence of a particular feedback statement, as perceived by a recipient within an academic context. Feedback was defined and evaluated from the perspective of Feedback Intervention Theory (FIT). Both research hypotheses were supported, given that the model's motivation predictions were positively associated with the actual motivation scores of feedback statements, and the model was closer to estimating the true motivation scores than expected by chance. These findings, paired with additional exploratory analyses, demonstrated the utility and effectiveness of the model in predicting perceived student motivation from feedback statements. Thus, this research provided a reliable tool researchers and practitioners in academia could use to evaluate the motivating influence of feedback for students, and it might inspire future studies in this domain. / Doctor of Philosophy / The use of feedback to enhance motivation is beneficial across various life domains. While both feedback and motivation have been studied widely in psychological science, most of this research has used close-ended (not text-analytic) approaches to study feedback empirically, which limits the scope of investigation. The present study was one of the first applications of text-analysis to assess the impact of feedback on the recipient's motivation. A machine-learning model was used to create a tool that can predict the average motivating influence of a particular feedback statement, as perceived by a recipient within an academic context. Both research hypotheses were supported. The motivation predictions were positively associated with the actual motivation scores of feedback statements, and the model was closer to estimating the true motivation scores than would be expected by chance. These findings, paired with additional exploratory analyses, demonstrated the utility and effectiveness of the model in predicting perceived student motivation from feedback statements. Additionally, based on this study it is recommended that professors include specific behaviors to be modified when delivering feedback. Thus, this research provided a tool that researchers and practitioners in academia could use to evaluate the motivating influence of feedback for students, and it might certainly inspire future studies in this domain.

Bats are famous for their capability of navigating in dense forests for hundreds of kilometers within one night by using their sonar system. Airborne sonar hasn't been heavily used in the industrial world compared to other sensors such as lidar, radar, and cameras. In this study, we applied a biosonar robot to navigate in a dense forest with bat-like FM-CF ultrasonic signals with deep learning. The results presented show that airborne biosonar can classify different areas' plants, in addition to achieving a similar level of navigation granularity compared to GPS, which is about 6 meters of radius resolution. The time- frequency representations of echoes from the forest are used as input data to explore the biosonar navigation ability, and the state-of-the-art CNN deep network (Resnet 152) is used as the brain to do the echolocation in the dense forest. The navigation ability can be improved significantly by combining multiple 10 ms long echoes, however, the data size of the reflected waves is much smaller than the other popularly used sensors, as echo can be collected at a rate of 40 echoes per second. The results can prove that airborne sonar can be used to navigate in GPS-denied environments, and can be an important sensor used in a scenario when other sensors meet constraints, like in the sensor fusion applications. / Doctor of Philosophy / The ability to identify natural landmarks could contribute to the navigation skills of echolo- cating bats and also advance the quest for autonomy in natural environments with man- made systems. The critical sensors used in autonomous robot navigation are camera array, radar, and lidar, airborne sonar hasn't been verified for its navigation efficiency. However, recognizing natural landmarks based on biosonar echoes has to deal with the unpredictable nature of echoes that are typically superpositions of contributions from many different reflec- tors with unknown properties. This dissertation intends to explore the bioinspired airborne sonar navigation ability in dense natural forests. The first part of this project is to use reflected echoes to navigate on a large scale, data was collected from different mountains which are dozens of kilometers away from each other, and we achieved the use of one single navigator in those locations. The second part is to explore the navigation granularity of airborne sonar sensors, data were collected from a small dense forest area, we try to classify which part of the foliage was based on the echo, and in the end, we achieved GPS accuracy for navigation. The finding in this work proves that the sonar sensor can play an important role in the sensing system, with the help of a deep neural network, with a 10 ms long echo, it can have a similar navigation ability to GPS.

In this work, we present different deep learning-based digital pre-distorters and compare them based on their performance towards improving the linearity of highly non-linear power amplifiers. The simulation results show that BiLSTM based DPDs work the best in terms of improving the linearity performance. We also compare two methodologies of direct learning and indirect learning to develop deep learning-based digital pre-distorters (DL-DPDs) models and evaluate their improvement on the linearity of Power Amplifiers (PA). We carry out a theoretical analysis on the differences between these training methodologies and verify their performance with simulation results on class-AB and class-F⁻¹ PAs. The simulation results show that both the learning methods lead to an improvement of more than 12 dB and 11dB in the linearity of class-AB and class-F⁻¹ PAs respectively, with indirect learning DL-DPD offering marginally better performance. Moreover, we compare the DL-DPD with memory polynomial models and show that using the former gives a significant improvement over the memory polynomials. Furthermore, we discuss the advantages of exploiting a BiLSTM based neural network architecture for designing direct/indirect DPDs. We demonstrate that BiLSTM DPD can be used to pre distort signals of any size without the drop in linearity. Moreover, based on the insights we develop a frequency domain loss using which further increased the linearity of the PA. / Master of Science / Wireless communication devices have fundamentally changed the way we interact with people. This increased the user's reliance on communication devices and significantly grew the need for higher data rates and faster internet speeds. But one major obstacle inside the transmitter chain (antenna) with increasing the data rates is the power amplifier, which distorts the signals at these higher powers. This distortion will reduce the efficiency and reliability of communication systems, greatly decreasing the quality of communication. So, we developed a high-performance DPD using deep learning to combat this issue. In this paper, we compare different deep learning-based DPDs and analyze which offers better performance. We also contrast two training methodologies to learn these DL-DPDs, theoretically and with simulation to arrive at which method offers better performing DPDs. We do these experiments on two different types of power amplifiers, and signals of any length. We design a new loss function, such that optimizing it leads to better DL-DPDs.