This project represents an important first step in determining the fundamental aesthetically relevant features of sound. Though there has been much effort in revealing the features learned by a deep neural network (DNN) trained on visual data, little effort in applying these techniques to a network trained on audio data has been performed. Importantly, these efforts in the audio domain often impose strong biases about relevant features (e.g., musical structure). In this project, a DNN is trained to mimic the acoustic aesthetic judgment of a professional composer. A unique corpus of sounds and corresponding professional aesthetic judgments is leveraged for this purpose. By applying a variation of Google's "DeepDream" algorithm to this trained DNN, and limiting the assumptions introduced, we can begin to listen to and examine the features of sound fundamental for aesthetic judgment. / Master of Science / The question of what makes a sound aesthetically “interesting” is of great importance to many, including biologists, philosophers of aesthetics, and musicians. This project serves as an important first step in determining the fundamental aesthetically relevant features of sound. First, a computer is trained to mimic the aesthetic judgments of a professional composer; if the composer would deem a sound “interesting,” then so would the computer. During this training, the computer learns for itself what features of sound are important for this classification. Then, a variation of Google’s “DeepDream” algorithm is applied to allow these learned features to be heard. By carefully considering the manner in which the computer is trained, this algorithmic “dreaming” allows us to begin to hear aesthetically salient features of sound.

It has been a while that computers have replaced our record books, and medical records are no exception. Electronic Health Records (EHR) are digital version of a patient's medical records. EHRs are available to authorized users, and they contain the medical records of the patient, which should help doctors understand a patient's condition quickly. In recent years, Deep Learning models have proved their value and have become state-of-the-art in computer vision, natural language processing, speech and other areas. The private nature of EHR data has prevented public access to EHR datasets. There are many obstacles to create a deep learning model with EHR data. Because EHR data are primarily consisting of huge sparse matrices, these challenges are mostly unique to this field. Due to this, research in this area is limited, and we can improve existing research substantially. In this study, we focus on high-performance synthetic data generation in EHR datasets. Artificial data generation can help reduce privacy leakage for dataset owners as it is proven that de-identification methods are prone to re-identification attacks. We propose a novel approach we call Improved Correlation Capturing Wasserstein Generative Adversarial Network (SCorGAN) to create EHR data. This work, leverages Deep Convolutional Neural Networks to extract and understand spatial dependencies in EHR data. To improve our model's performance, we focus on our Deep Convolutional AutoEncoder to better map our real EHR data to our latent space where we train the Generator. To assess our model's performance, we demonstrate that our generative model can create excellent data that are statistically close to the input dataset. Additionally, we evaluate our synthetic dataset against the original data using our previous work that focused on GAN Performance Evaluation. This work is publicly available at https://github.com/mohibeyki/SCorGAN / Master of Science / Artificial Intelligence (AI) systems have improved greatly in recent years. They are being used to understand all kinds of data. A practical use case for AI systems is to leverage their power to identify illnesses and find correlations between different conditions. To train AI and Machine Learning systems, we need to feed them huge datasets, and in the training process, we need to guide them so that they learn different features in our data. The more data an intelligent system has seen, the better it performs. However, health records are private, and we cannot share real people's health records with the public, whether they are a researcher or not. This study provides a novel approach to synthetic data generation that others can use with intelligent systems. Then these systems can work with actual health records can give us accurate feedback on people's health conditions. We then show that our synthetic dataset is a good substitute for real datasets to train intelligent systems. Lastly, we present an intelligent system that we have trained using synthetic datasets to identify illnesses in a real dataset with high accuracy and precision.

This work addresses the problem of automatically detecting human skin in images without reliance on color information. Unlike previous methods, we present a new approach that performs well in the absence of such information. A key aspect of the work is that color-space augmentation is applied strategically during the training, with the goal of reducing the influence of features that are based entirely on color and increasing more semantic understanding. The resulting system exhibits a dramatic improvement in performance for images in which color details are diminished. We have demonstrated the concept using the U-Net architecture, and experimental results show improvements in evaluations for all Fitzpatrick skin tones in the ECU dataset. We further tested the system with RFW dataset to show that the proposed method is consistent across different ethnicities and reduces bias to any skin tones. Therefore, this work has strong potential to aid in mitigating bias in automated systems that can be applied to many applications including surveillance and biometrics. / Master of Science / Skin segmentation deals with the classification of skin and non-skin pixels and regions in a image containing these information. Although most previous skin-detection methods have used color cues almost exclusively, they are vulnerable to external factors (e.g., poor or unnatural illumination and skin tones). In this work, we present a new approach based on U-Net that performs well in the absence of color information. To be specific, we apply a new color space augmentation into the training stage to improve the performance of skin segmentation system over the illumination and skin tone diverse. The system was trained and tested with both original and color changed ECU dataset. We also test our system with RFW dataset, a larger dataset with four human races with different skin tones. The experimental results show improvements in evaluations for skin tones and complex illuminations.

Society's newfound dependence on wireless transmission systems has driven demand for access to the electromagnetic (EM) spectrum to an all-time high. In particular, wireless applications related to the fifth generation (5G) of cellular technology along with statically allocated radar systems have contributed to the increasing scarcity of the sub 6 GHz frequency bands. As a result, development of Dynamic Spectrum Access (DSA) techniques for sharing these frequencies has become a critical research area for the greater wireless community. Since among incumbent systems, radars are the largest consumers of spectrum in the sub 6 GHz regime, and are being used increasingly for civilian applications such as traffic control, adaptive cruise control, and collision avoidance, the need for radars which can adaptively tune specific transmission parameters in an intelligent manner to promote coexistence with other systems has arisen. Thus, fully-aware, dynamic, cognitive radar has been proposed as target for radars to evolve towards. In this thesis, we extend current research thrusts towards cognitive radar to utilize Reinforcement Learning (RL) techniques which allow a radar system to learn desired behavior using information obtained from past transmissions. Since radar systems inherently interact with their electromagnetic environment, it is natural to view the use of reinforcement learning techniques as a straightforward extension to previous adaptive techniques. However, in designing learning algorithms for radar systems, we must carefully define goal-driven rewards, formalize the learning process, and consider an appropriate amount of environmental information. In this thesis, we apply well-established and emerging reinforcement learning approaches to meet the demands of modern radar coexistence problems. In particular, function estimation using deep neural networks is examined, as Deep RL presents a scalable learning framework which allows many environmental states to be considered in the decision-making process. We then show how these techniques can be used to improve traditional radar performance metrics, such as interference avoidance, spectral efficiency, and target detectibility with simulated and experimental results. We also compare the learning techniques to each other and naive approaches, such as fixed bandwidth radar and avoiding interference reactively. Finally, online learning strategies are considered which aim to balance the fundamental learning trade-off between exploration and exploitation. We show that online learning techniques can be used to select individual waveforms or applied as a high-level controller in a hierarchical learning scheme based on the biologically inspired concept of metacognition. The general use of RL techniques provides a robust framework for decision making under uncertainty that is more flexible than previously proposed cognitive radar strategies. Further, the wide array of RL models and algorithms allow the underlying structure to be applied to both small and large-scale radar scenarios. / Master of Science / Society's newfound dependence on wireless transmission systems has driven demand for control of the electromagnetic (EM) spectrum to an all-time high. In particular, federal spectrum auctions and the fifth generation of wireless technologies have contributed to the scarcity of frequency bands below 6GHz. These frequencies are widely used by both radar and communications systems due to favorable propagation characteristics. However, current radar systems typically occupy a fixed bandwidth and are tend to be poorly equipped to share their allocated spectrum with other users, which has become a necessity given the growth of wireless traffic. In this thesis, we study learning algorithms which enable a radar to optimize its electromagnetic pulses based on feedback from received signals. In particular, we are interested in reinforcement learning algorithms which allow a radar to learn optimal behavior based on rewards defined by a human. Using these algorithms, radar system designers can choose which metrics may be most important for a given radar application which can then be optimized for the given setting. However, scaling reinforcement learning to real-world problems such as radar optimization is often difficult due to the massive scope of the problem. Here we attempt to identify potential issues with implementation of each algorithm and narrow in on algorithms that are well-suited for real-time radar operation.

Melanoma is the most deadly form of skin cancer worldwide, which causes the 75% of deaths related to skin cancer. National Cancer Institute estimated that 91,270 new case and 9,320 deaths are expected in 2018 caused by melanoma. Early detection of melanoma is the key for the treatment. The image technique to diagnose skin cancer is dermoscopy, which leads to improved diagnose accuracy compared to traditional ABCD criteria. But reading and examining dermoscopic images is a time-consuming and complex process. Therefore, computerized analysis methods of dermoscopic images have been developed to assist the visual interpretation of dermoscopic images. The automatic segmentation of skin lesion attributes is a key step in computerized analysis of dermoscopic images. The International Skin Imaging Collaboration (ISIC) hosted the 2018 Challenges to help the diagnosis of melanoma based on dermoscopic images. In this thesis, I develop a deep learning based approach to automatically segment the attributes from dermoscopic skin lesion images. The approach described in the thesis achieved the Jaccard index of 0.477 on the official test dataset, which ranked 5th place in the challenge. / Master of Science / Melanoma is the most deadly form of skin cancer worldwide, which causes the 75% of deaths related to skin cancer. Early detection of melanoma is the key for the treatment. The image technique to diagnose skin cancer is called dermoscopy. It has become increasingly conveniently to use dermoscopic device to image the skin in recent years. Dermoscopic lens are available in the market for individual customer. When coupling the dermoscopic lens with smartphones, people are be able to take dermoscopic images of their skin even at home. However, reading and examining dermoscopic images is a time-consuming and complex process. It requires specialists to examine the image, extract the features, and compare with criteria to make clinical diagnosis. The time-consuming image examination process becomes the bottleneck of fast diagnosis of melanoma. Therefore, computerized analysis methods of dermoscopic images have been developed to promote the melanoma diagnosis and to increase the survival rate and save lives eventually. The automatic segmentation of skin lesion attributes is a key step in computerized analysis of dermoscopic images. In this thesis, I developed a deep learning based approach to automatically segment the attributes from dermoscopic skin lesion images. The segmentation result from this approach won 5th place in a public competition. It has the potential to be utilized in clinic application in the future.

The ability to understand occlusion and disocclusion is critical in analyzing motion and forecasting changes. For example, when we see a car gradually blocks our view of a human figure, we know that either the car or the human is moving. We also know that the human behind the car will be visible again if we move to other positions. As many vision-based intelligent systems need to handle and react to visual data with potentially intensive motions, it is therefore beneficial to incorporate the occlusion reasoning into such systems. In this thesis, we study how we can improve the performance of vision-based deep learning models by harnessing the power of occlusion handling. We first visit the problem of optical flow estimation for motion analysis. We present a deep learning module that builds upon occlusion handling methods in classic Computer Vision literature. Our results show performance improvement in occluded regions on standard benchmarks, as well as real-world applications. We then examine the problem of view synthesis for 3D photography. We propose an inpainting method that leverages local color and depth context for novel view synthesis. We validate the proposed inpainting approach with a series of quantitative and qualitative experiments, and demonstrate promising results in predicting plausible content in occluded regions. / Master of Science / Human has the ability to understand occlusion, and make use of such knowledge to make predictions about motions and occluded contents. For example, when we see a car gradually blocks our view of a human figure, we know that either the car or the human is moving. We also know that the human behind the car will be visible again if we move to other positions. In this thesis, we study how we can replicate such an ability to artificial intelligence systems. We first investigate the effect of occlusion reasoning in the task of predicting motion. Our experimental results show that a system equipped with our occlusion reasoning module can better capture the motions happening in image sequences. Next, we examine the problem of hallucinating visual contents that are blocked in an image. We develop a model that can produce plausible content in occluded regions. In our experiments, we show that given one single RGB image with an estimated depth map, our model can produce a corresponding 3D photo by hallucinating the structures that are not visible in the image.

The ability to extract figures and tables from scientific documents can solve key use-cases such as their semantic parsing, summarization, or indexing. Although a few methods have been developed to extract figures and tables from scientific documents, their performance on scanned counterparts is considerably lower than on born-digital ones. To facilitate this, we propose methods to effectively extract figures and tables from Electronic Theses and Dissertations (ETDs), that out-perform existing methods by a considerable margin. Our contribution towards this goal is three-fold. (a) We propose a system/model for improving the performance of existing methods on scanned scientific documents for figure and table extraction. (b) We release a new dataset containing 10,182 labelled page-images spanning across 70 scanned ETDs with 3.3k manually annotated bounding boxes for figures and tables. (c) Lastly, we release our entire code and the trained model weights to enable further research (https://github.com/SampannaKahu/deepfigures-open). / Master of Science / Portable Document Format (PDF) is one of the most popular document formats. However, parsing PDF files is not a trivial task. One use-case of parsing PDF files is the search functionality on websites hosting scholarly documents (i.e., IEEE Xplore, etc.). Having the ability to extract figures and tables from a scholarly document helps this use-case, among others. Methods using deep learning exist which extract figures from scholarly documents. However, a large number of scholarly documents, especially the ones published before the advent of computers, have been scanned from hard paper copies into PDF. In particular, we focus on scanned PDF versions of long documents, such as Electronic Theses and Dissertations (ETDs). No experiments have been done yet that evaluate the efficacy of the above-mentioned methods on this scanned corpus. This work explores and attempts to improve the performance of these existing methods on scanned ETDs. A new gold standard dataset is created and released as a part of this work for figure extraction from scanned ETDs. Finally, the entire source code and trained model weights are made open-source to aid further research in this field.

Data-efficient learning is crucial for building language models that can adapt to a wide variety of tasks with minimal annotations of labeled examples. Recently, the advent of large language models (LLMs) has given rise to a new ability called in-context learning (ICL), where LLMs can learn and perform a new task via inference on a prompt that consists of a few input-output pairs, all while keeping their parameters frozen. While ICL excels on academic benchmarks, it faces several challenges in real-world deployment, including sensitivity to prompt artifacts, poor calibration of model confidence, and inefficiency due to large model sizes. We conduct a systematic study of ICL sensitivity and find a negative correlation between ICL sensitivity and accuracy. To improve ICL calibration, we propose a sensitivity-based method that assigns the negative value of sensitivity as a confidence score, and demonstrate that our approach outperforms baselines in selective prediction tasks. Additionally, we propose to enhancing the efficiency of ICL through a new method called in-context tuning, which involves fine-tuning small language models on ICL prompts. We further augment the ICL capabilities of small LMs by incorporating distillation from larger LLMs into the in-context tuning process. Besides proposing new strategies to improve the reliability, accuracy, and efficiency of ICL, we also present a study on understanding how ICL emerges. The emergence of ICL is mysterious, as ICL prompts consisting of input-output concatenations are rare in natural text, yet pre-training on natural text alone is sufficient for ICL to emerge. We identify a structure called parallel structures, which capture pairs of phrases sampled from the same distribution, and verify through ablation experiments that these structures are a major source of ICL. Finally, we investigate the effectiveness of LLMs in explaining themselves when prompted with ICL demonstration examples. We propose a new metric called counterfactual simulatability, which measures whether humans can use LLM-generated explanations to construct precise and generalized mental models of the LLMs. Our results demonstrate that LLMs’ capacity to provide faithful explanations is significantly lower than that of humans, even with ICL examples. To address this, we propose explanation-precision fine-tuning, which uses data augmentation to generate synthetic fine-tuning data with explanations that are consistent with answers on relevant inputs. Our contributions advance the accuracy, reliability, efficiency and understanding of ICL of LLMs, offering methods to mitigate sensitivity, improve calibration, enhance efficiency, and strengthen the self-explaining power of LLMs. This work paves the way for more data-efficient, reliable and explainable language models for diverse real-world applications.

published_or_final_version / Industrial and Manufacturing Systems Engineering / Doctoral / Doctor of Philosophy

No description available.