<p>The most important networks from different domains—such as Computing, Organization, Economic, Social, Academic, and Biology—are networks that change over time. For example, in an organization there are email and collaboration networks (e.g., different people or teams working on a document). Apart from the connectivity of the networks changing over time, they can contain attributes such as the topic of an email or message, contents of a document, or the interests of a person in an academic citation or a social network. Analyzing these dynamic networks can be critical in decision-making processes. For instance, in an organization, getting insight into how people from different teams collaborate, provides important information that can be used to optimize workflows.</p> <p><br></p> <p>Network generative models provide a way to study and analyze networks. For example, benchmarking model performance and generalization in tasks like node classification, can be done by evaluating models on synthetic networks generated with varying structure and attribute correlation. In this work, we begin by presenting our systemic study of the impact that graph structure and attribute auto-correlation on the task of node classification using collective inference. This is the first time such an extensive study has been done. We take advantage of a recently developed method that samples attributed networks—although static—with varying network structure jointly with correlated attributes. We find that the graph connectivity that contributes to the network auto-correlation (i.e., the local relationships of nodes) and density have the highest impact on the performance of collective inference methods.</p> <p><br></p> <p>Most of the literature to date has focused on static representations of networks, partially due to the difficulty of finding readily-available datasets of dynamic networks. Dynamic network generative models can bridge this gap by generating synthetic graphs similar to observed real-world networks. Given that motifs have been established as building blocks for the structure of real-world networks, modeling them can help to generate the graph structure seen and capture correlations in node connections and activity. Therefore, we continue with a study of motif evolution in <em>dynamic</em> temporal graphs. Our key insight is that motifs rarely change configurations in fast-changing dynamic networks (e.g. wedges intotriangles, and vice-versa), but rather keep reappearing at different times while keeping the same configuration. This finding motivates the generative process of our proposed models, using temporal motifs as building blocks, that generates dynamic graphs with links that appear and disappear over time.</p> <p><br></p> <p>Our first proposed model generates dynamic networks based on motif-activity and the roles that nodes play in a motif. For example, a wedge is sampled based on the likelihood of one node having the role of hub with the two other nodes being the spokes. Our model learns all parameters from observed data, with the goal of producing synthetic graphs with similar graph structure and node behavior. We find that using motifs and node roles helps our model generate the more complex structures and the temporal node behavior seen in real-world dynamic networks.</p> <p><br></p> <p>After observing that using motif node-roles helps to capture the changing local structure and behavior of nodes, we extend our work to also consider the attributes generated by nodes’ activities. We propose a second generative model for attributed dynamic networks that (i) captures network structure dynamics through temporal motifs, and (ii) extends the structural roles of nodes in motifs to roles that generate content embeddings. Our new proposed model is the first to generate synthetic dynamic networks and sample content embeddings based on motif node roles. To the best of our knowledge, it is the only attributed dynamic network model that can generate <em>new</em> content embeddings—not observed in the input graph, but still similar to that of the input graph. Our results show that modeling the network attributes with higher-order structures (e.g., motifs) improves the quality of the networks generated.</p> <p><br></p> <p>The generative models proposed address the difficulty of finding readily-available datasets of dynamic networks—attributed or not. This work will also allow others to: (i) generate networks that they can share without divulging individual’s private data, (ii) benchmark model performance, and (iii) explore model generalization on a broader range of conditions, among other uses. Finally, the evaluation measures proposed will elucidate models, allowing fellow researchers to push forward in these domains.</p>

<p>The introduction of artificial neural networks (ANNs) to speech recognition applications has sparked the rapid development and popularization of digital assistants. These digital assistants perform keyword spotting (KWS), constantly monitoring the audio captured by a microphone for a small set of words or phrases known as keywords. Upon recognizing a keyword, a larger audio recording is saved and processed by a separate, more complex neural network. More broadly, neural networks in speech recognition have popularized voice as means of interacting with electronic devices, sparking an interest in individuals using speech recognition in their own projects. However, while large companies have the means to develop custom neural network architectures alongside proprietary hardware platforms, such development precludes those lacking similar resources from developing efficient and effective neural networks for embedded systems. While small, low-power embedded systems are widely available in the hobbyist space, a clear process is needed for developing a neural network that accounts for the limitations of these resource-constrained systems. In contrast, a wide variety of neural network architectures exists, but often little thought is given to deploying these architectures on edge devices. </p> <p><br></p> <p>This thesis first presents an overview of audio processing techniques, artificial neural network fundamentals, and machine learning tools. A summary of a set of specific neural network architectures is also discussed. Finally, the process of implementing and modifying these existing neural network architectures and training specific models in Python using TensorFlow is demonstrated. The trained models are also subjected to post-training quantization to evaluate the effect on model performance. The models are evaluated using metrics relevant to deployment on resource-constrained systems, such as memory consumption, latency, and model size, in addition to the standard comparisons of accuracy and parameter count. After evaluating the models and architectures, the process of deploying one of the trained and quantized models is explored on an Arduino Nano 33 BLE using TensorFlow Lite for Microcontrollers and on a Digilent Nexys 4 FPGA board using CFU Playground.</p>

<p>Drug discovery and development has experienced a tremendous growth in the recent</p> <p>years, and methods to accelerate the process are necessary as the demand for effective drugs</p> <p>to treat a wide range of diseases continue to increase. Nevertheless, the majority of conventional</p> <p>techniques are labor-intensive or have relatively low yields. As a result, academia</p> <p>and the pharmaceutical industry are continuously seeking for rapid and efficient methods to</p> <p>accelerate the drug discovery pipeline. Therefore, in order to expedite the drug discovery</p> <p>process, recent developments in physical and artificial intelligence models have been utilized</p> <p>extensively. However, the overarching problem is how to use these cutting-edge advancements</p> <p>in artificial intelligence to enhance drug discovery? Therefore, this dissertation work</p> <p>focused on developing and applying artificial intelligence and physical models to accelerate</p> <p>the drug discovery pipeline at different stages. As the first study reported in the dissertation,</p> <p>the potential to apply graph neural network-based machine learning architectures</p> <p>with the assistance of molecular modeling features to identify plausible drug leads out of</p> <p>structurally similar chemical databases is assessed. Then, the capability of applying molecular</p> <p>modeling methods including molecular docking and molecular dynamics simulations to</p> <p>identify prospective targets and biological pathways for small molecular drugs is discussed</p> <p>and evaluated in the following chapter. Further, the capability of applying state-of-the-art</p> <p>deep learning architectures such as multi-layer perceptron and recurrent neural networks</p> <p>to optimize the formulation development stage has been assessed. Moreover, this dissertation</p> <p>has contributed to assist functionality identification of unknown compounds using</p> <p>simple machine learning based computational frameworks. The developed omics data analysis</p> <p>pipeline is then discussed in order to comprehend the effects of a particular treatment</p> <p>on the proteome and lipidome levels of cells. In conclusion, the potential for developing and</p> <p>utilizing various artificial intelligence-based approaches to accelerate the drug discovery and</p> <p>development process is explored in this research. Thus, these collaborative studies intend</p> <p>to contribute to ongoing acceleration efforts and advancements in the drug discovery and</p> <p>development field.</p>

<p>Uncertainty analysis is of great use both for calculating outputs that are more akin to real<br> flight, and for optimization to more robust shapes. However, implementation of uncertainty<br> has been a longstanding challenge in the field of aerodynamics due to the computational cost<br> of simulations. Geometric uncertainty in particular is often left unexplored in favor of uncer-<br> tainties in freestream parameters, turbulence models, or computational error. Therefore, this<br> work proposes a method of geometric uncertainty analysis for aerodynamic shapes that miti-<br> gates the barriers to its feasible computation. The process takes a two- or three-dimensional<br> shape and utilizes a combination of multifidelity meshes and Gaussian process regression<br> (GPR) surrogates in a multifidelity Monte Carlo (MFMC) algorithm. Multifidelity meshes<br> allow for finer sampling with a given budget, making the surrogates more accurate. GPR<br> surrogates are made practical to use by parameterizing major factors in geometric uncer-<br> tainty with only four variables in 2-D and five in 3-D. In both cases, two parameters control<br> the heights of steps that occur on the top and bottom of airfoils where leading and trailing<br> edge devices are attached. Two more parameters control the height and length of waves<br> that can occur in an ideally smooth shape during manufacturing. A fifth parameter controls<br> the depth of span-wise skin buckling waves along a 3-D wing. Parameters are defined to<br> be uniformly distributed with a maximum size of 0.4 mm and 0.15 mm for steps and waves<br> to remain within common manufacturing tolerances. The analysis chain is demonstrated<br> with two test cases. The first, the RAE2822 airfoil, uses transonic freestream parameters<br> set by the ADODG Benchmark Case 2. The results show a mean drag of nearly 10 counts<br> above the deterministic case with fixed lift, and a 2 count increase for a fixed angle of attack<br> version of the case. Each case also has small variations in lift and angle of attack of about<br> 0.5 counts and 0.08◦, respectively. Variances for each of the three tracked outputs show that<br> more variability is possible, and even likely. The ONERA M6 transonic wing, popular due<br> to the extensive experimental data available for computational validation, is the second test<br> case. Variation is found to be less substantial here, with a mean drag increase of 0.5 counts,<br> and a mean lift increase of 0.1 counts. Furthermore, the MFMC algorithm enables accurate<br> results with only a few hours of wall time in addition to GPR training. </p>

<p>Nuclear engineering hosts diverse domains including, but not limited to, power plant automation, human-machine interfacing, detection and identification of special nuclear materials, modeling of reactor kinetics and dynamics that most frequently are described by systems of differential equations (DEs), either ordinary (ODEs) or partial ones (PDEs). In this work we study multiple problems related to safety and Special Nuclear Material detection, and numerical solutions for partial differential equations using neural networks. More specifically, this work is divided in six chapters. Chapter 1 is the introduction, in Chapter</p> <p>2 we discuss the development of a gamma-ray radionuclide library for the characterization</p> <p>of gamma-spectra. In Chapter 3, we present a new approach, the ”Variance Counterbalancing”, for stochastic</p> <p>large-scale learning. In Chapter 4, we introduce a systematic approach for constructing proper trial solutions to partial differential equations (PDEs) of up to second order, using neural forms that satisfy prescribed initial, boundary and interface conditions. Chapter 5 is about an alternative, less imposing development of neural-form trial solutions for PDEs, inside rectangular and non-rectangular convex boundaries. Chapter 6 presents an ensemble method that avoids the multicollinearity issue and provides</p> <p>enhanced generalization performance that could be suitable for handling ”few-shots”- problems frequently appearing in nuclear engineering.</p>

<p>High-performance computing (HPC) and machine learning (ML) have been widely adopted by both academia and industries to address enormous data problems at extreme scales. While research has reported on the interactions of HPC and ML, achieving high performance and scalability for parallel and distributed ML algorithms is still a challenging task. This dissertation first summarizes the major challenges for applying HPC to ML applications: 1) poor performance and scalability, 2) loss of the convergence rate, 3) lower quality of the trained model, and 4) a lack of performance optimization techniques designed for specific applications. Researchers can address the four challenges in new ML applications. This dissertation shows how to solve them for two specific applications: 1) a clustering algorithm and 2) graph optimization algorithms that use reinforcement learning (RL).</p> <p>As to the clustering algorithm, we first propose an algorithm called the simulated-annealing clustering algorithm. By combining a blocked data layout and asynchronous local optimization within each thread, the simulated-annealing enhanced clustering algorithm has a convergence rate that is comparable to the K-means algorithm but with much higher performance. Experiments with synthetic and real-world datasets show that the simulated-annealing enhanced clustering algorithm is significantly faster than the MPI K-means library using up to 1024 cores. However, the optimization costs (Sum of Square Error (SSE)) of the simulated-annealing enhanced clustering algorithm became higher than the original costs. To tackle this problem, we devise a new algorithm called the full-step feel-the-way clustering algorithm. In the full-step feel-the-way algorithm, there are L local steps within each block of data points. We use the first local step’s results to compute accurate global optimization costs. Our results show that the full-step algorithm can significantly reduce the global number of iterations needed to converge while obtaining low SSE costs. However, the time spent on the local steps is greater than the benefits of the saved iterations. To improve this performance, we next optimize the local step time by incorporating a sampling-based method called reassignment-history-aware sampling. Extensive experiments with various synthetic and real world datasets (e.g., MNIST, CIFAR-10, ENRON, and PLACES-2) show that our parallel algorithms can outperform the fastest open-source MPI K-means implementation by up to 110% on 4,096 CPU cores with comparable SSE costs.</p> <p>Our evaluations of the sampling-based feel-the-way algorithm establish the effectiveness of the local optimization strategy, the blocked data layout, and the sampling methods for addressing the challenges of applying HPC to ML applications. To explore more parallel strategies and optimization techniques, we focus on a more complex application: graph optimization problems using reinforcement learning (RL). RL has proved successful for automatically learning good heuristics to solve graph optimization problems. However, the existing RL systems either do not support graph RL environments or do not support multiple or many GPUs in a distributed setting. This has compromised RL’s ability to solve large scale graph optimization problems due to the lack of parallelization and high scalability. To address the challenges of parallelization and scalability, we develop OpenGraphGym-MG, a high performance distributed-GPU RL framework for solving graph optimization problems. OpenGraphGym-MG focuses on a class of computationally demanding RL problems in which both the RL environment and the policy model are highly computation intensive. In this work, we distribute large-scale graphs across distributed GPUs and use spatial parallelism and data parallelism to achieve scalable performance. We compare and analyze the performance of spatial and data parallelism and highlight their differences. To support graph neural network (GNN) layers that take data samples partitioned across distributed GPUs as input, we design new parallel mathematical kernels to perform operations on distributed 3D sparse and 3D dense tensors. To handle costly RL environments, we design new parallel graph environments to scale up all RL-environment-related operations. By combining the scalable GNN layers with the scalable RL environment, we are able to develop high performance OpenGraphGym-MG training and inference algorithms in parallel.</p> <p>To summarize, after proposing the major challenges for applying HPC to ML applications, this thesis explores several parallel strategies and performance optimization techniques using two ML applications. Specifically, we propose a local optimization strategy, a blocked data layout, and sampling methods for accelerating the clustering algorithm, and we create a spatial parallelism strategy, a parallel graph environment, agent, and policy model, and an optimized replay buffer, and multi-node selection strategy for solving large optimization problems over graphs. Our evaluations prove the effectiveness of these strategies and demonstrate that our accelerations can significantly outperform the state-of-the-art ML libraries and frameworks without loss of quality in trained models.</p>

<p dir="ltr">For turbocharged engine simulation analysis, manufacturer-provided data are typically used to predict the mass flow and efficiency of the turbine. To create a turbine map, physical tests are performed in labs at various turbine speeds and expansion ratios. These tests can be very expensive and time-consuming. Current testing methods can have limitations that result in errors in the turbine map. As such, only a modest set of data can be generated, all of which have to be interpolated and extrapolated to create a smooth surface that can then be used for simulation analysis.</p><p><br></p><p dir="ltr">The current method used by the manufacturer is a physics-informed polynomial regression model that depends on the Blade Speed Ratio (BSR ) in the polynomial function to model the efficiency and MFP. This method is memory-consuming and provides a lower-than-desired accuracy. This model is decades old and must be updated with new state-of-the-art Machine Learning models to be more competitive. Currently, CTT is facing up to +/-2% error in most turbine maps for efficiency and MFP and the aim is to decrease the error to 0.5% while interpolating the data points in the available region. The current model also extrapolates data to regions where experimental data cannot be measured. Physical tests cannot validate this extrapolation and can only be evaluated using CFD analysis.</p><p><br></p><p dir="ltr">The thesis focuses on investigating different AI techniques to increase the accuracy of the model for interpolation and evaluating the models for extrapolation. The data was made available by CTT. The available data consisted of various turbine parameters including ER, turbine speeds, efficiency, and MFP which were considered significant in turbine modeling. The AI models developed contained the above 4 parameters where ER and turbine speeds are predictors and, efficiency and MFP are the response. Multiple supervised ML models such as SVM, GPR, LMANN, BRANN, and GBPNN were developed and evaluated. From the above 5 ML models, BRANN performed the best achieving an error of 0.5% across multiple turbines for efficiency and MFP. The same model was used to demonstrate extrapolation, where the model gave unreliable predictions. Additional data points were inputted in the training data set at the far end of the testing regions which greatly increased the overall look of the map.</p><p><br></p><p dir="ltr">An additional contribution presented here is to completely predict an expansion ratio line and evaluate with CTT test data points where the model performed with an accuracy of over 95%. Since physical testing in a lab is expensive and time-consuming, another goal of the project was to reduce the number of data points provided for ANN model training. Furthermore, strategically reducing the data points is of utmost importance as some data points play a major role in the training of ANN and can greatly affect the model's overall accuracy. Up to 50% of the data points were removed for training inputs and it was found that BRANN was able to predict a satisfactory turbine map while reducing 20% of the overall data points at various regions.</p>

<p dir="ltr">Detecting anomalies in spacecraft time-series data is a high priority, especially considering the harshness of the spacecraft operating environment. These anomalies often function as precursors for system failure. Traditionally, the time-series data channels are monitored manually by domain experts, which is time-consuming. Additionally, there are thousands of channels to monitor. Machine learning methods have proven to be useful for automatic anomaly detection, but a unique model must be trained from scratch for each time-series. This thesis proposes three approaches for reducing training costs. First, a transfer learning approach that finetunes a general pre-trained model to reduce training time and the number of unique models required for a given spacecraft. The second and third approaches both use online learning to reduce the amount of training data and time needed to identify anomalies. The second approach leverages an ensemble of extreme learning machines while the third approach uses deep learning models. All three approaches are shown to achieve reasonable anomaly detection performance with reduced training costs.</p><p dir="ltr">Measuring the phenotypes, or observable traits, of a plant enables plant scientists to understand the interaction between the growing environment and the genetic characteristics of a plant. Plant phenotyping is typically done manually, and often involves destructive sampling, making the entire process labor-intensive and difficult to replicate. In this thesis, we use image processing for characterizing two different disease progressions. Tar spot disease can be identified visually as it induces small black circular spots on the leaf surface. We propose using a Mask R-CNN to detect tar spots from RGB images of leaves, thus enabling rapid non-destructive phenotyping of afflicted plants. The second disease, bacteria-induced wilting, is measured using a visual assessment that is often subjective. We design several metrics that can be extracted from RGB images that can be used to generate consistent wilting measurements with a random forest. Both approaches ensure faster, replicable results, enabling accurate, high-throughput analysis to draw conclusions about effective disease treatments and plant breeds.</p>

<p dir="ltr">Neural operator learning is emerging as a prominent technique in scientific machine learn- ing for modeling complex nonlinear systems with multi-physics and multi-scale applications. A common drawback of such operators is that they are data-hungry and the results are highly dependent on the quality and quantity of the training data provided to the models. Moreover, obtaining high-quality data in sufficient quantity can be computationally prohibitive. Faster surrogate models are required to overcome this drawback which can be learned from datasets of variable fidelity and also quantify the uncertainty. In this work, we propose a Multi-Level Stacked Deep Operator Network (MLSDON) which can learn from datasets of different fidelity and is not dependent on the input function. Through various experiments, we demonstrate that the MLSDON can approximate the high-fidelity solution operator with better accuracy compared to a Vanilla DeepONet when sufficient high-fidelity data is unavailable. We also extend MLSDON to build robust confidence intervals by making conformalized predictions. This technique guarantees trajectory coverage of the predictions irrespective of the input distribution. Various numerical experiments are conducted to demonstrate the applicability of MLSDON to multi-fidelity, multi-scale, and multi-physics problems.</p>

<p dir="ltr">The primary purpose of this thesis is to propose and demonstrate BioGRNsemble, a modular and flexible approach for inferencing gene regulatory networks from RNA-Seq data. Integrating the GENIE3 and GRNBoost2 algorithms, this ensembles-of-ensembles method attempts to balance the outputs of both models through averaging, before providing a trimmed-down gene regulatory network consisting of transcription and target genes. Using a Drosophila Eye Dataset, we were able to successfully test this novel methodology, and our validation analysis using an online database determined over 3500 gene links correctly detected, albeit out of almost 530,000 predictions, leaving plenty of room for improvement in the future.</p>