Pattern formation, or the generation of coordinated, emergent behavior, is ubiquitous in nature. Researchers have long sought to understand the mechanisms behind such systems as zebra stripes, repeating flower petals, and fingers on hands, within fields such as physics and developmental biology. Notably, a diverse array of bacteria species naturally self-organize into durable macroscale patterns on solid surfaces via swarming motility—a highly coordinated, rapid movement of bacteria powered by flagella. Meanwhile, researchers in the synthetic biology field, which aims to rationally engineer living organisms for biotechnological applications, have been engineering synthetic pattern formation in microbes over the last several decades. Engineering swarming is an untapped opportunity to increase the scale and robustness of coordinated synthetic microbial systems. In this thesis, we expand the field of engineered pattern formation by applying the tools of synthetic biology and deep learning to engineer and characterize the swarming of Proteus mirabilis, which natively forms a centimeter-scale ring pattern. We engineer P. mirabilis to “write” external inputs into visible spatial records. Specifically, we engineer tunable expression of swarming-related genes that modify pattern features, and we develop quantitative approaches to decoding. Next, we develop a dual-input system that modulates two swarming-related genes simultaneously, and we apply convolutional neural networks (CNNs) to decode the resulting patterns with over 90% top-3 accuracy. We separately show growing colonies can record dynamic environmental changes which can be decoded with a U-Net model. We show the robustness of the engineered strains’ readout to fluctuations in temperature and environmental water samples. Lastly, we engineer strains which sense and respond to heavy metals. Our pCopA-flgM strain records the presence of 0 to 50 mM aqueous copper with decreased colony ring width. We conclude in this chapter that engineering native swarm patterns can thus be applied for building bacterial recorders with a visible macroscale readout. In parallel, to better characterize the swarm patterns of P. mirabilis, we develop a pipeline using deep learning approaches to segment colony images. We develop easy-to-use, semi-automated ground truth annotation and preprocessing methods. We separately segment the (1) colony background from agar and (2) the internal colony ring boundaries. The first task is achieved with a patch-classification approach; in the process, we find that the combination of the trained CNN and the “majority voting” method of label fusion achieves a test DICE score of 93% and correctly segments even faint outer swarm rings. The second task is accomplished with a U-Net which achieves over 83% test DICE. We show that our trained models easily segment a set of colonies generated at two relevant conditions, enabling automated analysis of features such as area and ring width. We apply our pipeline to analyze the more complex patterns of our engineered strains, such as the pCopA-flgM strain. The work in this chapter altogether advances the ability to analyze swarm patterns of P. mirabilis. We also aim to expand the use of our colony-characterization approaches beyond P. mirabilis to other microbes. Therefore, we present our work using deep learning to classify a set of Bacillus species isolated from soil samples. We generate datasets of the species grown under different conditions and apply transfer learning to train well-known CNN architectures such as ResNet and Inception to classify these datasets. This approach allows the models to easily learn these small datasets, and the models generalize to correctly predict a species which forms branching patterns regardless of exact growth condition. We visualize the attributions of the models with the integrated gradients method and find that model predictions are attributable to colony regions. This work sets the stage for classification, segmentation, and characterization of a wider array of microbial species with distinctive macroscale colony morphologies. Finally, we conclude by discussing ongoing efforts to expand upon the work presented in this thesis towards the sensing of dynamic inputs such as light, engineering of species other than P. mirabilis, and further optimization of the system of an engineered swarm pattern as a macroscale biosensor readout. Such work can contribute not only to the fields of synthetic pattern formation and the study of bacterial swarming, but also to the fields of engineered living materials and bio-inspired design.

Wearable technology offers the promise of decentralized and personalized healthcare, which can both alleviate current burdens on medical resources, and also help individuals to be more informed about their health. The heterogeneity of disease phenotypes necessitates adaptations to both diagnosing and surveilling disease, but to ensure user adoption and behavioral change, there needs to be a convenient way to amass such health information continuously. This can be in part accomplished by the development of continuously monitoring, compact wearable medical sensors and analytics technology that provide updates on analyte and biosignal measurements at regular intervals in situ. This dissertation investigates methods for collecting and analyzing information from wearable devices with these principles in mind. In Aim 1, we developed new methods for analysis of cardiovascular biosignals. Current methods of estimating left ventricular mass index (LVMI, a strong risk factor for cardiac outcomes), rely on the analysis of echocardiographic signals. Though still the gold standard, echocardiography can typically only be performed in the clinic, making it inconvenient to obtain frequent measurements of LVMI. Frequent measurements can be useful for monitoring cardiac risk, particularly for high-risk individuals, so we investigated the feasibility of predicting LVMI using a deep learning-based approach through ambulatory blood pressure readings, a one-time laboratory test and demographic information. We find that adding blood pressure waveform information in conjunction with multitask learning improved prediction errors (compared to baseline linear regression and neural network models), pointing to its potential as a clinical tool. Using transfer learning, we developed a model that does not require waveform data, but achieved similar prediction accuracies as methods that do require such data – opening the door to use cases that eliminate the need for wearing a blood pressure cuff continuously during the measurement period. Overall, such a technique has the potential to provide information to individuals who are at high risk of cardiac outcomes both inside and outside the clinic. In Aims 2 and 3, we developed a minimally invasive hydrogel patch for continuous monitoring of calcium, as proof-of-concept for wearable measurement of a wide variety of analytes typically assayed in the lab – a technology that can facilitate treatment and management of many prevalent diseases. Specifically, in Aim 2, we engineered a DNA polyacrylamide hydrogel microneedle array that sensed physiologically relevant calcium levels, for potential use by individuals who have hypoparathyroidism, a condition in which blood calcium levels are low and calcium supplements are needed. A negative mold was made using a CNC mill, the positive mold was cast in silicone, and the aptamer along with acrylamide and bis-acrylamide was seeded into the silicone mold. The DNA hydrogel was then fabricated using a simple UV curing protocol. The optimized DNA hydrogel was specific to calcium, used simple fabrication methods and had a fast, reversible signal response. Finally, in Aim 3, we developed the DNA hydrogel sensor into a wearable, integrated system with real-time fluorescence monitoring for testing in vivo. The microneedle array needed to be hydrated for the DNA aptamer to function, but polyacrylamide was too weak in its hydrated state to effectively pierce through skin epidermis. We demonstrated a method for strengthening our hydrogel system with polyethylene glycol diacrylate (PEGDA), while maintaining an optically translucent gel for detection purposes. We conducted piercing studies with a skin phantom on different microneedle array sizes and shapes, and determined that a 3x3 array of beveled microneedles required the least amount of force to pierce through a skin phantom. A custom complementary metal-oxide semiconductor (CMOS) system was developed to capture real-time fluorescence signals from the microneedle array, which correlated to calcium levels in vitro. This setup was then validated in a rat study. In this dissertation, we demonstrated methods for monitoring human biosignals using signal processing techniques, material innovations and integrated sensing platforms. While a work in progress, this dissertation is a step towards realizing the goal of decentralized, connected health for earlier detection and better management of disease.

This thesis encompasses two studies on the application of computational techniques, including deep learning and physics-based methods, in the exploration of protein structure and dynamics. In Chapter 1, I will introduce the background knowledge. Chapter 2 describes the development of a deep learning method for protein loop modeling. We introduce a fast and accurate method for protein loop structure modeling and refinement using deep learning. This method, which is both fast and accurate, integrates a protein language model, a graph neural network, and attention-based modules to predict all-atom protein loop structures from sequences. Its accuracy was validated on benchmark datasets CASP14 and CAMEO, showing performance comparable to or better than the state-of-the-art method, AlphaFold2. The model’s robustness against loop structures outside of the training set was confirmed by testing on datasets after removing high-identity templates and train- ing set homologs. Moreover, it demonstrated significantly lower computational costs compared to existing methods. Application of this method in real-world scenarios included predicting anti- body complementarity-determining regions (CDR) loop structures and refining loop structures in inexact side-chain environments. The method achieved sub-angstrom or near-angstrom accuracy for most CDR loops and notably enhanced the quality of many suboptimal loop predictions in in- exact environments, marking an advancement in protein loop structure prediction and its practical applications. Chapter 3 presents a collaborative study that employs nuclear magnetic resonance (NMR) experiments, molecular dynamics (MD), and hybrid quantum mechanics/molecular mechanics (QM/MM) calculations to investigate protein conformational dynamics across varying temperatures. NMR chemical shifts provide a sensitive probe of protein structure and dynamics. Prediction of shifts, and therefore interpretation of shifts, particularly for the frequently measured amidic 15N sites, remains a tall challenge. We demonstrate that protein ¹⁵N chemical shift prediction from QM/MM predictions can be improved if conformational variation is included via MD sampling, focusing on the antibiotic target, E. coli Dihydrofolate reductase (DHFR). Variations of up to 25 ppm in predicted ¹⁵N chemical shifts are observed over the trajectory. For solution shifts, the average of fluctuations on the low picosecond timescale results in a superior prediction to a single optimal conformation. For low-temperature solid-state measurements, the histogram of predicted shifts for locally minimized snapshots with specific solvent arrangements sampled from the trajectory explains the heterogeneous linewidths; in other words, the conformations and associated solvent are ‘frozen out’ at low temperatures and result in inhomogeneously broadened NMR peaks. We identified conformational degrees of freedom that contribute to chemical shift variation. Backbone torsion angles show high amplitude fluctuations during the trajectory on the low picosecond timescale. For a number of residues, including I60, 𝝍 varies by up to 60o within a conformational basin during the MD simulations, despite the fact that I60 (and other sites studied) are in a secondary structure element and remain well folded during the trajectory. Fluctuations in 𝝍 appear to be compensated by other degrees of freedom in the protein, including 𝝓 of the succeeding residue, resulting in “rocking” of the amide plane with changes in hydrogen bonding interactions. Good agreement for both room-temperature and low-temperature NMR spectra provides strong support for the specific approach to conformational averaging of computed chemical shifts.

This thesis investigates the problem of learning video representations for video understanding. Previous works have explored the use of data-driven deep learning approaches, which have been shown to be effective in learning useful video representations. However, obtaining large amounts of labeled data can be costly and time-consuming. We investigate self-supervised approach as for multimodal video data to overcome this challenge. Video data typically contains multiple modalities, such as visual, audio, transcribed speech, and textual captions, which can serve as pseudo-labels for representation learning without needing manual labeling. By utilizing these modalities, we can train deep representations over large-scale video data consisting of millions of video clips collected from the internet. We demonstrate the scalability benefits of multimodal self-supervision by achieving new state-of-the-art performance in various domains, including video action recognition, text-to-video retrieval, and text-to-video grounding. We also examine the limitations of these approaches, which often rely on the association assumption involving multiple modalities of data used in self-supervision. For example, the text transcript is often assumed to be about the video content, and two segments of the same video share similar semantics. To overcome this problem, we propose new methods for learning video representations with more intelligent sampling strategies to capture samples that share high-level semantics or consistent concepts. The proposed methods include a clustering component to address false negative pairs in multimodal paired contrastive learning, a novel sampling strategy for finding visually groundable video-text pairs, an investigation of object tracking supervision for temporal association, and a new multimodal task for demonstrating the effectiveness of the proposed model. We aim to develop more robust and generalizable video representations for real-world applications, such as human-to-robot interaction and event extraction from large-scale news sources.

No description available.

Nowadays, damage detection strategies built on the application of Artificial Neural Network tools to define models that mimic the dynamic behavior of structural systems are viral. However, a fundamental issue in developing these strategies for damage assessment is given by the unbalanced nature of the available databases for civil, mechanical, or aerospace applications, which commonly do not contain sufficient information from all the different classes that need to be identified. Unfortunately, when the aim is to classify between the healthy and damaged conditions in a structure or a generic dynamic system, it is extremely rare to have sufficient data for the unhealthy state since the system has already failed. At the same time, it is common to have plenty of data coming from the system under operational conditions. Consequently, the learning task, carried on with deep learning approaches, becomes case-dependent and tends to be specialized for a particular case and a very limited number of damage scenarios. This doctoral research presents a framework for damage classification in dynamic systems intended to overcome the limitations imposed by unbalanced datasets. In this methodology, the model's classification ability is enriched by using lower-level features derived through an improved extraction strategy that learns from a rich audio dataset how to characterize vibration traits starting from human voice recordings. This knowledge is then transferred to a target domain with much less data points, such as a structural system where the same discrimination approach is employed to classify and differentiate different health conditions. The goal is to enrich the model's ability to discriminate between classes on the audio records, presenting multiple different categories with more information to learn. The proposed methodology is validated both numerically and experimentally.

Building facades, serving mainly to protect occupants and structural components from natural forces, require periodic inspections for the detection and assessment of building façade anomalies. Over the past years, a growing trend of utilizing camera-equipped drones for periodical building facade inspection has emerged. Building façade anomalies, such as cracks and erosion, can be detected through analyzing drone-captured video, photographs, and infrared images. Such anomalies are known to have an impact on various building performance aspects, e.g., thermal, energy, moisture control issues. Current research efforts mainly focus on the design of drone flight schema for building inspection, 3D building model reconstruction through drone-captured images, and the detection of specific façade anomalies with these images. However, there are several research gaps impeding the improvement of automation level during the processes of building façade inspection with UAV (Unmanned Aerial Vehicle). These gaps are (1) lack effective ways to store multi-type data captured by drones with the connection to the spatial information of building facades, (2) lack high-performance tools for UAV-image analysis for the automated detection of building façade anomalies, and (3) lack a comprehensive management (i.e., storage, retrieval, analysis, and display) of large amounts and multi-media information for cyclic façade inspection. When seeking inspirations from nature, the process of drone-based facade inspection can be compared with caching birds' foraging food through spatial memory, visual sensing, and remarkable memories. This dissertation aims at investigating ways to improve the management of UAV-captured data and the automation level of drone-based façade anomaly inspection with inspirations from caching birds' foraging behavior. Firstly, a 2D spatial model of building façades was created in the geographic information system (GIS) for the registration and storage of UAV-images to assign façade spatial information to each image. Secondly, computational methods like computer vision and deep learning neural networks were applied to develop algorithms for automated extraction of visual features of façade anomalies within UAV-captured images. Thirdly, a GIS-based database was designed for the comprehensive management of heterogeneous inspection data, such as the spatial, multi-spectral, and temporal data. This research will improve the automation level of storage, retrieval, analysis, and documentation of drone-captured images to support façade inspection during a building's service lifecycle. It has promising potential for supporting the decision-making of early-intervention or maintenance strategies to prevent façade failures and improve building performance. / Doctor of Philosophy / Building facades require periodic inspections and maintenance to protect occupants and structures from natural forces like the sun, wind, rain, and snow. Over the past years, a growing trend of utilizing drones for periodical building facade inspection has emerged. Building façade anomalies, such as cracks and corrosion, can be detected from the drone-captured photographs or video. Such anomalies are known to have an impact on various building performance aspects, such as moisture issues, abnormal heat loss, and additional energy consumptions. Existing practices for detecting façade anomalies from drone-captured photographs mainly rely on manual checking by going through numerous façade images and repetitively zooming in and out these high-resolution images, which is time-consuming and labor-intensive with potential risks of human errors. Besides, this manual checking process impedes the management of drone-captured data and the documentation of façade inspection activities. At the same time, the emerging technologies of computer vision (CV) and artificial intelligence (AI) have provided many opportunities to improve the automation level of façade anomaly detection and documentation. Previous research efforts have explored the image-based generation of 3D building models using computer vision techniques, as well as image-based detection of specific anomalies using deep learning techniques. However, few studies have looked into the comprehensive management, including the storage, retrieval, analysis, and display, of drone-captured images with the spatial coordinate information of building facades; there is also a lack of high-performance image analytics tools for the automated detection of building façade anomalies. This dissertation aims at investigating ways to improve the automation level of analyzing and managing drone-captured images as well as documenting building façade inspection information. To achieve this goal, a building façade model was created in the geographic information system (GIS) for the semi-automated registration and storage of drone-captured images with spatial coordinates by using computer vision techniques. Secondly, deep learning was applied for automated detection of façade anomalies in drone-captured images. Thirdly, a GIS-based database was designed as the platform for the automated analysis and management of heterogeneous data for drone-captured images, façade model information, and detected façade anomalies. This research will improve the automation level of drone-based façade inspection throughout a building's service lifecycle. It has promising potential for supporting the decision-making of maintenance strategies to prevent façade failures and improve building performance.

In recent years, metasurfaces have shown promising abilities to control and manipulate electromagnetic (EM) waves through modified surface boundary conditions. These surfaces are electrically thin and comprise an array of spatially varying sub-wavelength scattering elements (or meta-atoms). Metasurfaces can transform an incident EM wave into an arbitrarily tailored transmitted or reflected wavefront through carefully engineering each meta-atom. Recent developments in metasurfaces have opened exciting new opportunities in antenna design, sensing, and communications systems. In particular, reconfigurable metasurfaces - wherein meta-atoms are embedded with active components - lead to the development of low-cost, lightweight, and compact systems capable of producing programmable radiation patterns and jointly performing multi-function communications, and enable advanced sensors for next-generation platforms. This research introduces reconfigurable metasurfaces and their various applications in designing simplified communications systems, wherein the RF aperture and transceiver are integrated within the metasurface. Finally, we will present our recent work on reconfigurable metasurfaces control, metasurface-enabled direct signal modulation, and deep learning-based metasurface design. / Doctor of Philosophy / Metasurfaces are a promising new technology to enhance the capacity and coverage of wireless communication networks by dynamically reconfiguring the wireless propagation environment. These low-profile artificial electromagnetic surfaces, consisting of subwavelength resonant elements, are capable of tailoring electromagnetic waves controllably. In this dissertation, we control the transmission or reflection properties of the surface using digital codes by embedding tunable elements within each subwavelength element. Furthermore, metasurface antennas are a promising candidate for reducing the cost and hardware footprint of wireless sensor systems, such as radar or imaging. Using a digital microcontroller, we program the metasurface to steer the antenna beam in the direction of interest, modulate the radio wave, or change the polarization of an incoming signal. In addition to dynamic beamforming capabilities, we program the metasurface to reduce the scattering of an incoming signal, thereby reducing its perturbations on the radio environment. Still, the design of metasurfaces for specific applications remains complex and technically challenging. Lastly, we present innovative deep learning techniques to simplify metasurface design.

A protein is a biopolymer of amino acids that encodes a particular function. Given that there are 20 amino acids possible at each site, even a short protein of 100 amino acids has $20^{100}$ possible variants, making it unrealistic to evaluate all possible sequences in sequence level space. This search space could be reduced by considering the fact that billions of years of evolution exerting a constant pressure has left us with only a small subset of protein sequences that carry out particular cellular functions. The portion of amino acid space occupied by actual proteins found in nature is therefore much smaller than that which is possible cite{kauffman1993origins}. By examining related proteins that share a conserved function and common evolutionary history (heretofore referred to as protein families), it is possible to identify common motifs that are shared. Examination of these motifs allows us to characterize protein families in greater depth and even generate new ``in silico" proteins that are not found in nature, but exhibit properties of a particular protein family. Using novel deep learning approaches and leveraging the large volume of genomic data that is now available due to high-throughput DNA sequencing, it is now possible to examine protein families in a scale and resolution that has never before been possible. By using this abundance of data to learn high dimensional representations of amino acids sequences, in this work, we show that it is possible to generate novel sequences from a particular protein family. Such a deep sequential model-based approach has great value for bioinformatics and biotechnological applications due to its rapid sampling abilities. / Master of Science / Proteins are one of the most important functional biological elements. These are composed of amino acids which link together to form different shapes which might encode a particular function. These proteins may act independently or might form ``complexes" to have a particular function. Therefore, understanding them is of utmost importance. Due to the fact that there are 20 amino acids even a protein sequence fragment of length 5 can have more than 3 million different combinations. Given, that proteins are generally 1000 amino acids long, looking at all the possibilities is next to impossible. In this work, by leveraging the ``deep learning" paradigm and the vast amount of data available, we try to model these proteins and generate new proteins belonging to a specific ``protein family." This approach has great value for bioinformatics and biotechnological applications due to its rapid sampling abilities.

The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane (CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. A dataset containing more than 16,000 images from 13 people, with a clear view of the neck area, and with CTM position labeled by a medical expert was built to train and validate the proposed model. It achieved a success rate of $99.6%$ to detect the position of the CTM with an error of less than 5mm. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. Finally, the detection neural network and the manipulation process are combined as an integrated system. The system was validated in real-life experiments on a human-sized medical manikin using a Kinect V2 camera and a MICO robot arm manipulator. / Master of Science / The thesis focuses on applying deep learning and reinforcement learning techniques on human keypoint detection and robot arm manipulation. Inspired by Semi-Autonomous Victim Extraction Robot (SAVER), an autonomous first-aid airway-management robotic system designed to perform Cricothyrotomy on patients is proposed. Perception, decision-making, and control are embedded in the system. In this system, first, the location of the cricothyroid membrane(CTM)-the incision site of Cricothyrotomy- is detected; then, the robot arm is controlled to reach the detected position on a medical manikin. A hybrid neural network (HNNet) that can balance both speed and accuracy is proposed. HNNet is an ensemble-based network architecture that consists of two ensembles: the region proposal ensemble and the keypoint detection ensemble. This architecture can maintain the original high resolution of the input image without heavy computation and can meet the high-precision and real-time requirements at the same time. Finally, the detection neural network and the manipulation process are combined as an integrated system. The robot arm manipulator was trained with the reinforcement learning model to reach the detected location. The system was validated in real-life experiments on a human-sized medical manikin using an RGB-D camera and a robot arm manipulator.