This project represents an important first step in determining the fundamental aesthetically relevant features of sound. Though there has been much effort in revealing the features learned by a deep neural network (DNN) trained on visual data, little effort in applying these techniques to a network trained on audio data has been performed. Importantly, these efforts in the audio domain often impose strong biases about relevant features (e.g., musical structure). In this project, a DNN is trained to mimic the acoustic aesthetic judgment of a professional composer. A unique corpus of sounds and corresponding professional aesthetic judgments is leveraged for this purpose. By applying a variation of Google's "DeepDream" algorithm to this trained DNN, and limiting the assumptions introduced, we can begin to listen to and examine the features of sound fundamental for aesthetic judgment. / Master of Science / The question of what makes a sound aesthetically “interesting” is of great importance to many, including biologists, philosophers of aesthetics, and musicians. This project serves as an important first step in determining the fundamental aesthetically relevant features of sound. First, a computer is trained to mimic the aesthetic judgments of a professional composer; if the composer would deem a sound “interesting,” then so would the computer. During this training, the computer learns for itself what features of sound are important for this classification. Then, a variation of Google’s “DeepDream” algorithm is applied to allow these learned features to be heard. By carefully considering the manner in which the computer is trained, this algorithmic “dreaming” allows us to begin to hear aesthetically salient features of sound.

It has been a while that computers have replaced our record books, and medical records are no exception. Electronic Health Records (EHR) are digital version of a patient's medical records. EHRs are available to authorized users, and they contain the medical records of the patient, which should help doctors understand a patient's condition quickly. In recent years, Deep Learning models have proved their value and have become state-of-the-art in computer vision, natural language processing, speech and other areas. The private nature of EHR data has prevented public access to EHR datasets. There are many obstacles to create a deep learning model with EHR data. Because EHR data are primarily consisting of huge sparse matrices, these challenges are mostly unique to this field. Due to this, research in this area is limited, and we can improve existing research substantially. In this study, we focus on high-performance synthetic data generation in EHR datasets. Artificial data generation can help reduce privacy leakage for dataset owners as it is proven that de-identification methods are prone to re-identification attacks. We propose a novel approach we call Improved Correlation Capturing Wasserstein Generative Adversarial Network (SCorGAN) to create EHR data. This work, leverages Deep Convolutional Neural Networks to extract and understand spatial dependencies in EHR data. To improve our model's performance, we focus on our Deep Convolutional AutoEncoder to better map our real EHR data to our latent space where we train the Generator. To assess our model's performance, we demonstrate that our generative model can create excellent data that are statistically close to the input dataset. Additionally, we evaluate our synthetic dataset against the original data using our previous work that focused on GAN Performance Evaluation. This work is publicly available at https://github.com/mohibeyki/SCorGAN / Master of Science / Artificial Intelligence (AI) systems have improved greatly in recent years. They are being used to understand all kinds of data. A practical use case for AI systems is to leverage their power to identify illnesses and find correlations between different conditions. To train AI and Machine Learning systems, we need to feed them huge datasets, and in the training process, we need to guide them so that they learn different features in our data. The more data an intelligent system has seen, the better it performs. However, health records are private, and we cannot share real people's health records with the public, whether they are a researcher or not. This study provides a novel approach to synthetic data generation that others can use with intelligent systems. Then these systems can work with actual health records can give us accurate feedback on people's health conditions. We then show that our synthetic dataset is a good substitute for real datasets to train intelligent systems. Lastly, we present an intelligent system that we have trained using synthetic datasets to identify illnesses in a real dataset with high accuracy and precision.

This work addresses the problem of automatically detecting human skin in images without reliance on color information. Unlike previous methods, we present a new approach that performs well in the absence of such information. A key aspect of the work is that color-space augmentation is applied strategically during the training, with the goal of reducing the influence of features that are based entirely on color and increasing more semantic understanding. The resulting system exhibits a dramatic improvement in performance for images in which color details are diminished. We have demonstrated the concept using the U-Net architecture, and experimental results show improvements in evaluations for all Fitzpatrick skin tones in the ECU dataset. We further tested the system with RFW dataset to show that the proposed method is consistent across different ethnicities and reduces bias to any skin tones. Therefore, this work has strong potential to aid in mitigating bias in automated systems that can be applied to many applications including surveillance and biometrics. / Master of Science / Skin segmentation deals with the classification of skin and non-skin pixels and regions in a image containing these information. Although most previous skin-detection methods have used color cues almost exclusively, they are vulnerable to external factors (e.g., poor or unnatural illumination and skin tones). In this work, we present a new approach based on U-Net that performs well in the absence of color information. To be specific, we apply a new color space augmentation into the training stage to improve the performance of skin segmentation system over the illumination and skin tone diverse. The system was trained and tested with both original and color changed ECU dataset. We also test our system with RFW dataset, a larger dataset with four human races with different skin tones. The experimental results show improvements in evaluations for skin tones and complex illuminations.

Society's newfound dependence on wireless transmission systems has driven demand for access to the electromagnetic (EM) spectrum to an all-time high. In particular, wireless applications related to the fifth generation (5G) of cellular technology along with statically allocated radar systems have contributed to the increasing scarcity of the sub 6 GHz frequency bands. As a result, development of Dynamic Spectrum Access (DSA) techniques for sharing these frequencies has become a critical research area for the greater wireless community. Since among incumbent systems, radars are the largest consumers of spectrum in the sub 6 GHz regime, and are being used increasingly for civilian applications such as traffic control, adaptive cruise control, and collision avoidance, the need for radars which can adaptively tune specific transmission parameters in an intelligent manner to promote coexistence with other systems has arisen. Thus, fully-aware, dynamic, cognitive radar has been proposed as target for radars to evolve towards. In this thesis, we extend current research thrusts towards cognitive radar to utilize Reinforcement Learning (RL) techniques which allow a radar system to learn desired behavior using information obtained from past transmissions. Since radar systems inherently interact with their electromagnetic environment, it is natural to view the use of reinforcement learning techniques as a straightforward extension to previous adaptive techniques. However, in designing learning algorithms for radar systems, we must carefully define goal-driven rewards, formalize the learning process, and consider an appropriate amount of environmental information. In this thesis, we apply well-established and emerging reinforcement learning approaches to meet the demands of modern radar coexistence problems. In particular, function estimation using deep neural networks is examined, as Deep RL presents a scalable learning framework which allows many environmental states to be considered in the decision-making process. We then show how these techniques can be used to improve traditional radar performance metrics, such as interference avoidance, spectral efficiency, and target detectibility with simulated and experimental results. We also compare the learning techniques to each other and naive approaches, such as fixed bandwidth radar and avoiding interference reactively. Finally, online learning strategies are considered which aim to balance the fundamental learning trade-off between exploration and exploitation. We show that online learning techniques can be used to select individual waveforms or applied as a high-level controller in a hierarchical learning scheme based on the biologically inspired concept of metacognition. The general use of RL techniques provides a robust framework for decision making under uncertainty that is more flexible than previously proposed cognitive radar strategies. Further, the wide array of RL models and algorithms allow the underlying structure to be applied to both small and large-scale radar scenarios. / Master of Science / Society's newfound dependence on wireless transmission systems has driven demand for control of the electromagnetic (EM) spectrum to an all-time high. In particular, federal spectrum auctions and the fifth generation of wireless technologies have contributed to the scarcity of frequency bands below 6GHz. These frequencies are widely used by both radar and communications systems due to favorable propagation characteristics. However, current radar systems typically occupy a fixed bandwidth and are tend to be poorly equipped to share their allocated spectrum with other users, which has become a necessity given the growth of wireless traffic. In this thesis, we study learning algorithms which enable a radar to optimize its electromagnetic pulses based on feedback from received signals. In particular, we are interested in reinforcement learning algorithms which allow a radar to learn optimal behavior based on rewards defined by a human. Using these algorithms, radar system designers can choose which metrics may be most important for a given radar application which can then be optimized for the given setting. However, scaling reinforcement learning to real-world problems such as radar optimization is often difficult due to the massive scope of the problem. Here we attempt to identify potential issues with implementation of each algorithm and narrow in on algorithms that are well-suited for real-time radar operation.

Melanoma is the most deadly form of skin cancer worldwide, which causes the 75% of deaths related to skin cancer. National Cancer Institute estimated that 91,270 new case and 9,320 deaths are expected in 2018 caused by melanoma. Early detection of melanoma is the key for the treatment. The image technique to diagnose skin cancer is dermoscopy, which leads to improved diagnose accuracy compared to traditional ABCD criteria. But reading and examining dermoscopic images is a time-consuming and complex process. Therefore, computerized analysis methods of dermoscopic images have been developed to assist the visual interpretation of dermoscopic images. The automatic segmentation of skin lesion attributes is a key step in computerized analysis of dermoscopic images. The International Skin Imaging Collaboration (ISIC) hosted the 2018 Challenges to help the diagnosis of melanoma based on dermoscopic images. In this thesis, I develop a deep learning based approach to automatically segment the attributes from dermoscopic skin lesion images. The approach described in the thesis achieved the Jaccard index of 0.477 on the official test dataset, which ranked 5th place in the challenge. / Master of Science / Melanoma is the most deadly form of skin cancer worldwide, which causes the 75% of deaths related to skin cancer. Early detection of melanoma is the key for the treatment. The image technique to diagnose skin cancer is called dermoscopy. It has become increasingly conveniently to use dermoscopic device to image the skin in recent years. Dermoscopic lens are available in the market for individual customer. When coupling the dermoscopic lens with smartphones, people are be able to take dermoscopic images of their skin even at home. However, reading and examining dermoscopic images is a time-consuming and complex process. It requires specialists to examine the image, extract the features, and compare with criteria to make clinical diagnosis. The time-consuming image examination process becomes the bottleneck of fast diagnosis of melanoma. Therefore, computerized analysis methods of dermoscopic images have been developed to promote the melanoma diagnosis and to increase the survival rate and save lives eventually. The automatic segmentation of skin lesion attributes is a key step in computerized analysis of dermoscopic images. In this thesis, I developed a deep learning based approach to automatically segment the attributes from dermoscopic skin lesion images. The segmentation result from this approach won 5th place in a public competition. It has the potential to be utilized in clinic application in the future.

The ability to understand occlusion and disocclusion is critical in analyzing motion and forecasting changes. For example, when we see a car gradually blocks our view of a human figure, we know that either the car or the human is moving. We also know that the human behind the car will be visible again if we move to other positions. As many vision-based intelligent systems need to handle and react to visual data with potentially intensive motions, it is therefore beneficial to incorporate the occlusion reasoning into such systems. In this thesis, we study how we can improve the performance of vision-based deep learning models by harnessing the power of occlusion handling. We first visit the problem of optical flow estimation for motion analysis. We present a deep learning module that builds upon occlusion handling methods in classic Computer Vision literature. Our results show performance improvement in occluded regions on standard benchmarks, as well as real-world applications. We then examine the problem of view synthesis for 3D photography. We propose an inpainting method that leverages local color and depth context for novel view synthesis. We validate the proposed inpainting approach with a series of quantitative and qualitative experiments, and demonstrate promising results in predicting plausible content in occluded regions. / Master of Science / Human has the ability to understand occlusion, and make use of such knowledge to make predictions about motions and occluded contents. For example, when we see a car gradually blocks our view of a human figure, we know that either the car or the human is moving. We also know that the human behind the car will be visible again if we move to other positions. In this thesis, we study how we can replicate such an ability to artificial intelligence systems. We first investigate the effect of occlusion reasoning in the task of predicting motion. Our experimental results show that a system equipped with our occlusion reasoning module can better capture the motions happening in image sequences. Next, we examine the problem of hallucinating visual contents that are blocked in an image. We develop a model that can produce plausible content in occluded regions. In our experiments, we show that given one single RGB image with an estimated depth map, our model can produce a corresponding 3D photo by hallucinating the structures that are not visible in the image.

The ability to extract figures and tables from scientific documents can solve key use-cases such as their semantic parsing, summarization, or indexing. Although a few methods have been developed to extract figures and tables from scientific documents, their performance on scanned counterparts is considerably lower than on born-digital ones. To facilitate this, we propose methods to effectively extract figures and tables from Electronic Theses and Dissertations (ETDs), that out-perform existing methods by a considerable margin. Our contribution towards this goal is three-fold. (a) We propose a system/model for improving the performance of existing methods on scanned scientific documents for figure and table extraction. (b) We release a new dataset containing 10,182 labelled page-images spanning across 70 scanned ETDs with 3.3k manually annotated bounding boxes for figures and tables. (c) Lastly, we release our entire code and the trained model weights to enable further research (https://github.com/SampannaKahu/deepfigures-open). / Master of Science / Portable Document Format (PDF) is one of the most popular document formats. However, parsing PDF files is not a trivial task. One use-case of parsing PDF files is the search functionality on websites hosting scholarly documents (i.e., IEEE Xplore, etc.). Having the ability to extract figures and tables from a scholarly document helps this use-case, among others. Methods using deep learning exist which extract figures from scholarly documents. However, a large number of scholarly documents, especially the ones published before the advent of computers, have been scanned from hard paper copies into PDF. In particular, we focus on scanned PDF versions of long documents, such as Electronic Theses and Dissertations (ETDs). No experiments have been done yet that evaluate the efficacy of the above-mentioned methods on this scanned corpus. This work explores and attempts to improve the performance of these existing methods on scanned ETDs. A new gold standard dataset is created and released as a part of this work for figure extraction from scanned ETDs. Finally, the entire source code and trained model weights are made open-source to aid further research in this field.

This thesis presents both a control scheme for naturalistic control of an exoskeleton glove and a glove design. Exoskeleton development has been focused primarily on design, improving soft actuator and cable-driven systems, with only limited focus on intelligent control. There is a need for control that is not limited to position or force reference signals and is user-driven. By implementing a motion amplification controller to increase weak movements of an impaired individual, a finger joint trajectory can be observed and used to predict their grasping intention. The motion amplification functions off of a virtual dynamical system that safely enforces the range of motion of the finger joints and ensures stability. Three grasp prediction algorithms are developed with improved levels of accuracy: regression, trajectory, and deep learning based. These algorithms were tested on published finger joint trajectories. The fusion of the amplification and prediction could be used to achieve naturalistic, user-guided control of an exoskeleton glove. The key to accomplishing this is series elastic actuators to move the finger joints, thereby allowing the wearer to deflect against the glove and inform the controller of their intention. These actuators are used to move the fingers in a nine degree of freedom exoskeleton that is capable of achieving all the grasps used most frequently in daily life. The controllers and exoskeleton presented here are the basis for improved exoskeleton glove control that can be used to assist or rehabilitate impaired individuals. / Master of Science / Millions of Americans report difficulty holding small or even lightweight objects. In many of these cases, their difficulty stems from a condition such as a stroke or arthritis, requiring either rehabilitation or assistance. For both treatments, exoskeleton gloves are a potential solution; however, widespread deployment of exoskeletons in the treatment of hand conditions requires significant advancement. Towards that end, the research community has devoted itself to improving the design of exoskeletons. Systems that use soft actuation or are driven by artificial tendons have merit in that they are comfortable to the wearer, but lack the rigidity required for monitoring the state of the hand and controlling it. Electromyography sensors are also a commonly explored technology for determining motion intention; however, only primitive conclusions can be drawn when using these sensors on the muscles that control the human hand. This thesis proposes a system that does not rely on soft actuation but rather a deflectable exoskeleton that can be used in rehabilitation or assistance. By using series elastic actuators to move the exoskeleton, the wearer of the glove can exert their influence over the machine. Additionally, more intelligent control is needed in the exoskeleton. The approach taken here is twofold. First, a motion amplification controller increases the finger movements of the wearer. Second, the amplified motion is processed using machine learning algorithms to predict what type of grasp the user is attempting. The controller would then be able to fuse the two, the amplification and prediction, to control the glove naturalistically.

In Indian Classical Music (ICM), the Raga is a musical piece's melodic framework. It encompasses the characteristics of a scale, a mode, and a tune, with none of them fully describing it, rendering the Raga a unique concept in ICM. The Raga provides musicians with a melodic fabric, within which all compositions and improvisations must take place. Identifying and categorizing the Raga is challenging due to its dynamism and complex structure as well as the polyphonic nature of ICM. Hence, Raga recognition—identify the constituent Raga in an audio file—has become an important problem in music informatics with several known prior approaches. Advancing the state of the art in Raga recognition paves the way to improving other Music Information Retrieval tasks in ICM, including transcribing notes automatically, recommending music, and organizing large databases. This thesis presents a novel melodic pattern-based approach to recognizing Ragas by representing this task as a document classification problem, solved by applying a deep learning technique. A digital audio excerpt is hierarchically processed and split into subsequences and gamaka sequences to mimic a textual document structure, so our model can learn the resulting tonal and temporal sequence patterns using a Recurrent Neural Network. Although training and testing on these smaller sequences, we predict the Raga for the entire audio excerpt, with the accuracy of 90.3% for the Carnatic Music Dataset and 95.6% for the Hindustani Music Dataset, thus outperforming prior approaches in Raga recognition. / Master of Science / In Indian Classical Music (ICM), the Raga is a musical piece's melodic framework. The Raga is a unique concept in ICM, not fully described by any of the fundamental concepts of Western classical music. The Raga provides musicians with a melodic fabric, within which all compositions and improvisations must take place. Raga recognition refers to identifying the constituent Raga in an audio file, a challenging and important problem with several known prior approaches and applications in Music Information Retrieval. This thesis presents a novel approach to recognizing Ragas by representing this task as a document classification problem, solved by applying a deep learning technique. A digital audio excerpt is processed into a textual document structure, from which the constituent Raga is learned. Based on the evaluation with third-party datasets, our recognition approach achieves high accuracy, thus outperforming prior approaches.

Text detection in the wild is the problem of locating text in images of everyday scenes. It is a challenging problem due to the complexity of everyday scenes. This problem possesses a great importance for many trending applications, such as self-driving cars. Previous research in text detection has been dominated by multi-stage sequential approaches which suffer from many limitations including error propagation from one stage to the next. Another line of work is the use of deep learning techniques. Some of the deep methods used for text detection are box detection models and fully convolutional models. Box detection models suffer from the nature of the annotations, which may be too coarse to provide detailed supervision. Fully convolutional models learn to generate pixel-wise maps that represent the location of text instances in the input image. These models suffer from the inability to create accurate word level annotations without heavy post processing. To overcome these aforementioned problems we propose a novel end-to-end system based on a mix of novel deep learning techniques. The proposed system consists of an attention model, based on a new deep architecture proposed in this dissertation, followed by a deep network based on Faster-RCNN. The attention model produces a high-resolution map that indicates likely locations of text instances. A novel aspect of the system is an early fusion step that merges the attention map directly with the input image prior to word-box prediction. This approach suppresses but does not eliminate contextual information from consideration. Progressively larger models were trained in 3 separate phases. The resulting system has demonstrated an ability to detect text under difficult conditions related to illumination, resolution, and legibility. The system has exceeded the state of the art on the ICDAR 2013 and COCO-Text benchmarks with F-measure values of 0.875 and 0.533, respectively. / Ph. D. / Text detection and recognition in the wild is the problem of locating and reading text in images of everyday scenes. Text detection refers to finding the bounding boxes that describe the location of text areas in an input image, while text recognition describes the problem of generating a transcript out of the detected text areas. Recognition can be viewed as simply Optical Character Recognition (OCR). OCR is an old problem where the developed models are considered mature. Text detection and recognition are challenging problems due to the complexity of everyday scenes, compared to the simpler problem of recognizing text in scanned documents. This problem possesses a great importance to many trending applications that need to locate and read text in the wild, such as self-driving cars. Researchers tend to focus on the text detection problem only due to the maturity of research related to text recognition. Previous research in text detection has been dominated by multi-stage sequential approaches. Those methods suffer from many limitations including, but not limited to, error propagation from the earlier stages to the later stages of the pipeline. Another line of work is the use of deep learning techniques. Deep learning is the state of the art in machine learning. It has demonstrated great success in many domains, including computer vision. Some of the deep methods used for text detection are box detection models and fully convolutional models. Box detection models learn to generate bounding box coordinates for text instances that exist in the input image. Box detection models suffer from the nature of the annotations, which may be too coarse to provide detailed supervision. Fully convolutional models learn to generate pixel-wise maps that represent the location of text instances in the input image. These models suffer from the inability to create accurate word level annotations without heavy post processing. To overcome these aforementioned problems we propose a novel end-to-end system based on a mix of novel deep learning techniques. The proposed system consists of an attention model followed by a network based on Faster-RCNN that has been conditioned to generate word-box predictions. The attention model produces a high-resolution map that indicates likely locations of text instances. A novel aspect of the system is an early fusion step that merges the attention map directly with the input image prior to word-box prediction. This approach suppresses but does not eliminate contextual information from consideration, and avoids the common problem of discarding small text regions. To facilitate training of the end-to-end system, progressively larger models were trained in 3 separate phases. The resulting system has demonstrated an ability to detect text under difficult conditions related to illumination, resolution, and legibility. The system has exceeded the state of the art on the well-known ICDAR 2013 and COCO-Text benchmarks. For the former case, the system has produced results with an F-measure value of 0.875. For the more challenging COCO-Text dataset, the system has shown a dramatic increase in performance with an F-measure value to 0.533, as compared to previously reported values in the range of 0.33 to 0.37. In order to build a powerful system, we introduced a novel deep learning architecture that achieved impressive performance on standard benchmarks. This architecture has been used as a backbone for the proposed attention model. A description of the proposed end-to-end system, as well as the implementation steps, will be detailed in the following sections.