Indiana University-Purdue University Indianapolis (IUPUI) / Current research in deep learning is primarily focused on using Python as a support language. Go, an emerging language, that has many benefits including native support for concurrency has seen a rise in adoption over the past few years. However, this language is not widely used to develop learning models due to the lack of supporting libraries and frameworks for model development. In this thesis, the use of Go for the development of neural network models in general and convolution neural networks is explored. The proposed study is based on a Go-CUDA implementation of neural network models called GoCuNets. This implementation is then compared to a Go-CPU deep learning implementation that takes advantage of Go's built in concurrency called ConvNetGo. A comparison of these two implementations shows a significant performance gain when using GoCuNets compared to ConvNetGo.

<p>  </p> <p>The abundance of heterogeneous data produced and collected each day via multimodal sources may contain hidden events of interest, but in order to extract them the streams of data need to be analyzed with appropriate algorithms, so these events are presented to the end user at the right moment and at the right time. This dissertation proposes a series of algorithms that shape a comprehensive framework for situational knowledge on demand to address this problem. The framework consists of several modules and approaches, each of them is presented in a separate chapter: I begin with video data analysis in streaming video and video at rest for enhanced object detection of real-life surveillance video. For detecting the rare events of interest, I develop a semantic video analysis algorithm which uses an overlay knowledge graph and a semantical network. I show that the usage of the external knowledge for understanding the semantic analysis outperforms other techniques such as transfer learning. </p> <p>The semantical outliers can be used further for improving the algorithm of detecting new objects in the stream of different modalities. I extend the framework with additional modules for natural language data and apply the extended version of the semantic analysis algorithm to define the events of interest from multimodal streaming data. I present a way of combining several feature extractors which can be extended to multiple heterogeneous streams of data in order to efficiently fuse the data based on its semantical similarity, and then show how the serverless architecture of the framework outperforms conventional cloud software architecture. </p> <p>Besides detecting the rare and semantically incompatible events, the semantic analysis can be used for improving the neural networks performance with the data augmentation. The algorithm presented for augmenting the data with the potentially novel objects to circumvent the data drift problem uses the knowledge graph and generative adversarial networks to present the objects to augment the training datasets for supervised learning. I extend the presented framework with a pipeline for generating synthetic novelties to improve the performance of feature extractors and provide the empirical evaluation of the developed method.</p>

Current research in deep learning is primarily focused on using Python as a support language. Go, an emerging language, that has many benefits including native support for concurrency has seen a rise in adoption over the past few years. However, this language is not widely used to develop learning models due to the lack of supporting libraries and frameworks for model development. In this thesis, the use of Go for the development of neural network models in general and convolution neural networks is explored. The proposed study is based on a Go-CUDA implementation of neural network models called GoCuNets. This implementation is then compared to a Go-CPU deep learning implementation that takes advantage of Go's built in concurrency called ConvNetGo. A comparison of these two implementations shows a significant performance gain when using GoCuNets compared to ConvNetGo.<br>

Autism Spectrum Disorder (ASD) is part of the developmental disabilities. There are numerous symptoms for ASD patients, including lack of abilities in social interaction, communication obstacle and repeatable behaviors. Meanwhile, the rate of ASD prevalence has kept rising by the past 20 years from 1 out of 150 in 2000 to 1 out of 54 in 2016. In addition, the ASD population is quite large. Specifically, 3.5 million Americans live with ASD in the year of 2014, which will cost U.S. citizens $236-$262 billion dollars annually for autism services. So, it is critical to make an accurate diagnosis for preschool age children with ASD, in order to give them a better life. Instead of using traditional ASD behavioral tests, such as ADI-R, ADOS, and DSM-IV, we applied brain MRI images as input to make diagnosis. We revised 3D-ResNet structure to fit 110 preschool children's brain MRI data, along with Convolution 3D and VGG model. The prediction accuracy with raw data is 65.22%. The accuracy is significantly improved to 82.61% by removing the noise around the brain. We also showed the speed of ML prediction is 308 times faster than behavior tests.

<p>Over the past decade, distributed representation learning has emerged as a popular alternative to conventional centralized machine learning training. The increasing interest in distributed representation learning, specifically federated learning, can be attributed to its fundamental property that promotes data privacy and communication savings. While conventional ML encourages aggregating data at a central location (e.g., data centers), distributed representation learning advocates keeping data at the source and instead transmitting model parameters across the network. However, since the advent of deep learning, model sizes have become increasingly large often comprising million-billions of parameters, which leads to the problem of communication latency in the learning process. In this thesis, we propose to tackle the problem of communication latency in two different ways: (i) learning private representation of data to enable its sharing, and (ii) reducing the communication latency by minimizing the corresponding long-range communication requirements.</p> <p><br></p> <p>To tackle the former goal, we first start by studying the problem of learning representations that are private yet informative, i.e., providing information about intended ''ally'' targets while hiding sensitive ''adversary'' attributes. We propose Exclusion-Inclusion Generative Adversarial Network (EIGAN), a generalized private representation learning (PRL) architecture that accounts for multiple ally and adversary attributes, unlike existing PRL solutions. We then address the practical constraints of the distributed datasets by developing Distributed EIGAN (D-EIGAN), the first distributed PRL method that learns a private representation at each node without transmitting the source data. We theoretically analyze the behavior of adversaries under the optimal EIGAN and D-EIGAN encoders and the impact of dependencies among ally and adversary tasks on the optimization objective. Our experiments on various datasets demonstrate the advantages of EIGAN in terms of performance, robustness, and scalability. In particular, EIGAN outperforms the previous state-of-the-art by a significant accuracy margin (47% improvement), and D-EIGAN's performance is consistently on par with EIGAN under different network settings.</p> <p><br></p> <p>We next tackle the latter objective - reducing the communication latency - and propose two timescale hybrid federated learning (TT-HF), a semi-decentralized learning architecture that combines the conventional device-to-server communication paradigm for federated learning with device-to-device (D2D) communications for model training. In TT-HF, during each global aggregation interval, devices (i) perform multiple stochastic gradient descent iterations on their individual datasets, and (ii) aperiodically engage in consensus procedure of their model parameters through cooperative, distributed D2D communications within local clusters. With a new general definition of gradient diversity, we formally study the convergence behavior of TT-HF, resulting in new convergence bounds for distributed ML. We leverage our convergence bounds to develop an adaptive control algorithm that tunes the step size, D2D communication rounds, and global aggregation period of TT-HF over time to target a sublinear convergence rate of O(1/t) while minimizing network resource utilization. Our subsequent experiments demonstrate that TT-HF significantly outperforms the current art in federated learning in terms of model accuracy and/or network energy consumption in different scenarios where local device datasets exhibit statistical heterogeneity. Finally, our numerical evaluations demonstrate robustness against outages caused by fading channels, as well favorable performance with non-convex loss functions.</p>