Contemporary machine learning models exhibit unprecedented performance in the text, vision, and time-series domains, but at the cost of significant computational and human resources. Applying these technologies for science requires balancing accuracy and resource allocation, which I investigate here via three unique case studies. In Chapter 1, I present a deep learning system for animal pose estimation from video. Existing approaches rely on frame-by-frame supervised deep learning, which requires extensive manual labeling, fails to generalize to data far outside of its training set, and occasionally produces scientifically-critical errors that are hard to detect. The solution proposed here includes semi-supervised learning on unlabeled videos, video-centric network architectures, and a post-processing step that combines network ensembling and state-space modeling. These methods improve performance both with scarce and abundant labels, and are implemented in an easy-to-use software package and cloud application. In Chapter 2, I turn to the Gaussian process, a canonical nonparametric model, known for its poor scaling with dataset size. Existing methods accelerate Gaussian processes at the cost of modeling biases. I analyze two common techniques -- early truncated conjugate gradients and random Fourier features -- showing that they find hyperparameters that underfit and overfit the data, respectively. I then propose to eliminate these biases in exchange of increased variance, via randomized truncation estimators. In In Chapter 3, I investigate continual learning, or "finetuning", in large language models (LLMs) with billions of weights. Training these models requires more memory than typically available in academic clusters. Low-Rank Adaptation (LoRA) is a widely-used technique that saves memory by training only low rank perturbations to selected weight matrices in a so-called "base model'". I compare the performance of LoRA and full finetuning on two target domains, programming and mathematics, across different data regimes. I find that in most common settings, LoRA underperforms full finetuning, but it nevertheless exhibits a desirable form of regularization: it better maintains the base model's performance on tasks outside the target domain. I then propose best practices for finetuning with LoRA. In summary, applying state-of-the-art models to large scientific datasets necessitates taking computational shortcuts. This thesis highlights the implications of these shortcuts and emphasizes the need for careful empirical and theoretical investigation to find favorable trade-offs between accuracy and resource allocation.

No description available.

This thesis presents new methods in two closely related areas of computer vision: human pose estimation, and gesture recognition in videos. In human pose estimation, we show that random forests can be used to estimate human pose in monocular videos. To this end, we propose a co-segmentation algorithm for segmenting humans out of videos, and an evaluator that predicts whether the estimated poses are correct or not. We further extend this pose estimator to new domains (with a transfer learning approach), and enhance its predictions by predicting the joint positions sequentially (rather than independently) in an image, and using temporal information in the videos (rather than predicting the poses from a single frame). Finally, we go beyond random forests, and show that convolutional neural networks can be used to estimate human pose even more accurately and efficiently. We propose two new convolutional neural network architectures, and show how optical flow can be employed in convolutional nets to further improve the predictions. In gesture recognition, we explore the idea of using weak supervision to learn gestures. We show that we can learn sign language automatically from signed TV broadcasts with subtitles by letting algorithms 'watch' the TV broadcasts and 'match' the signs with the subtitles. We further show that if even a small amount of strong supervision is available (as there is for sign language, in the form of sign language video dictionaries), this strong supervision can be combined with weak supervision to learn even better models.

Desenvolveu-se, pela Universidade de São Paulo, o protótipo de um equipamento submersível de captura para estudo de plâncton. Baseado na técnica shadowgraph, é formado por um feixe de LED infravermelho colimado e uma câmera de alta resolução, executados por um sistema de controle automatizado. Foram utilizados softwares de visão computacional desenvolvidos pelo Laboratório de Sistemas Planctônicos (LAPS) que executam várias tarefas, incluindo a captura e segmentação de imagens e a extração de informações com o intuito de classificar automaticamente novos conjuntos de regiões de interesse (ROIs). O teste de aprendizado de máquina contou com 57 mil quadros e 230 mil ROIs e teve, como base, dois algoritmos de classificação: o Support Vector Machine (SVM) e o Random Forest (RF). O conjunto escolhido para o treinamento inicial continha 15 classes de fito e zooplâncton, às quais foi atribuído um subconjunto de 5 mil ROIs. Os ROIs foram separados em grandes classes de, pelo menos, 100 ROIs cada. O resultado, calculado por meio do algoritmo de aprendizagem RF e SVM e fundamentado no método de validação cruzada, teve uma precisão de 0,78 e 0,79, respectivamente. O conjunto de imagens é proveniente de Ubatuba, no estado de São Paulo. Os perfis verticais elaborados apresentaram diferentes padrões de distribuição de partículas. O instrumento tem sido útil para a geração de dados espacialmente refinados em ecossistemas costeiros e oceânicos. / The University of São Paulo developed an underwater image capture system prototype to study plankton. Based on the shadowgraphic image technique, the system consists of a collimated infrared LED beam and a high-resolution camera, both executed by an automated control system. Computer vision software developed by the research laboratory was used to perform various tasks, including image capturing; image segmentation; and extract information to automatic classify news regions of interest (ROIs). The machine learning test had 57,000 frames and 230,000 ROIs, based on two classification algorithms: Support Vector Machine (SVM) and Random Forest (RF). The chosen set of the initial training had 15 classes of phytoplankton and zooplankton, which was assigned a subset of 5,000 ROIs. Big classes of, at least, 100 ROIs each were organized. The result, calculated by the RF and SVM learning algorithm and based on the cross-validation method, had a 0.78 and 0.79 precision score, respectively. The image package comes from Ubatuba, in the state of São Paulo. The vertical profiles elaborated presented different particles distribution patterns. The instrument has been useful for spatially refined data generation in coastal and oceanic ecosystems.

Desenvolveu-se, pela Universidade de São Paulo, o protótipo de um equipamento submersível de captura para estudo de plâncton. Baseado na técnica shadowgraph, é formado por um feixe de LED infravermelho colimado e uma câmera de alta resolução, executados por um sistema de controle automatizado. Foram utilizados softwares de visão computacional desenvolvidos pelo Laboratório de Sistemas Planctônicos (LAPS) que executam várias tarefas, incluindo a captura e segmentação de imagens e a extração de informações com o intuito de classificar automaticamente novos conjuntos de regiões de interesse (ROIs). O teste de aprendizado de máquina contou com 57 mil quadros e 230 mil ROIs e teve, como base, dois algoritmos de classificação: o Support Vector Machine (SVM) e o Random Forest (RF). O conjunto escolhido para o treinamento inicial continha 15 classes de fito e zooplâncton, às quais foi atribuído um subconjunto de 5 mil ROIs. Os ROIs foram separados em grandes classes de, pelo menos, 100 ROIs cada. O resultado, calculado por meio do algoritmo de aprendizagem RF e SVM e fundamentado no método de validação cruzada, teve uma precisão de 0,78 e 0,79, respectivamente. O conjunto de imagens é proveniente de Ubatuba, no estado de São Paulo. Os perfis verticais elaborados apresentaram diferentes padrões de distribuição de partículas. O instrumento tem sido útil para a geração de dados espacialmente refinados em ecossistemas costeiros e oceânicos. / The University of São Paulo developed an underwater image capture system prototype to study plankton. Based on the shadowgraphic image technique, the system consists of a collimated infrared LED beam and a high-resolution camera, both executed by an automated control system. Computer vision software developed by the research laboratory was used to perform various tasks, including image capturing; image segmentation; and extract information to automatic classify news regions of interest (ROIs). The machine learning test had 57,000 frames and 230,000 ROIs, based on two classification algorithms: Support Vector Machine (SVM) and Random Forest (RF). The chosen set of the initial training had 15 classes of phytoplankton and zooplankton, which was assigned a subset of 5,000 ROIs. Big classes of, at least, 100 ROIs each were organized. The result, calculated by the RF and SVM learning algorithm and based on the cross-validation method, had a 0.78 and 0.79 precision score, respectively. The image package comes from Ubatuba, in the state of São Paulo. The vertical profiles elaborated presented different particles distribution patterns. The instrument has been useful for spatially refined data generation in coastal and oceanic ecosystems.

Since its discovery in 1995 resting state functional connectivity derived from functional MRI data has become a popular neuroimaging method for study psychiatric disorders. Current methods for analyzing resting state functional connectivity in disease involve thousands of univariate tests, and the specification of regions of interests to employ in the analysis. There are several drawbacks to these methods. First the mass univariate tests employed are insensitive to the information present in distributed networks of functional connectivity. Second, the null hypothesis testing employed to select functional connectivity dierences between groups does not evaluate the predictive power of identified functional connectivities. Third, the specification of regions of interests is confounded by experimentor bias in terms of which regions should be modeled and experimental error in terms of the size and location of these regions of interests. The objective of this dissertation is to improve the methods for functional connectivity analysis using multivariate predictive modeling, feature selection, and whole brain parcellation. A method of applying Support vector classification (SVC) to resting state functional connectivity data was developed in the context of a neuroimaging study of depression. The interpretability of the obtained classifier was optimized using feature selection techniques that incorporate reliability information. The problem of selecting regions of interests for whole brain functional connectivity analysis was addressed by clustering whole brain functional connectivity data to parcellate the brain into contiguous functionally homogenous regions. This newly developed famework was applied to derive a classifier capable of correctly seperating the functional connectivity patterns of patients with depression from those of healthy controls 90% of the time. The features most relevant to the obtain classifier match those previously identified in previous studies, but also include several regions not previously implicated in the functional networks underlying depression.

Speech communication encompasses diverse types of information, including phonetics, affective state, voice quality, and speaker identity. From a speech production standpoint, the acoustic speech signal can be mainly divided into glottal source and vocal tract components, which play distinct roles in rendering the various types of information it contains. Most deployed speech analysis systems, however, do not explicitly represent these two components as distinct entities, as their joint estimation from the acoustic speech signal becomes an ill-defined blind deconvolution problem. Nevertheless, because of the desire to understand glottal behavior and how it relates to perceived voice quality, there has been continued interest in explicitly estimating the glottal component of the speech signal. To this end, several inverse filtering (IF) algorithms have been proposed, but they are unreliable in practice because of the blind formulation of the separation problem. In an effort to develop a method that can bypass the challenging IF process, this thesis proposes a new glottal source information extraction method that relies on supervised machine learning to transform smoothed spectral representations of speech, which are already used in some of the most widely deployed and successful speech analysis applications, into a set of glottal source features. A transformation method based on Gaussian mixture regression (GMR) is presented and compared to current IF methods in terms of feature similarity, reliability, and speaker discrimination capability on a large speech corpus, and potential representations of the spectral envelope of speech are investigated for their ability represent glottal source variation in a predictable manner. The proposed system was found to produce glottal source features that reasonably matched their IF counterparts in many cases, while being less susceptible to spurious errors. The development of the proposed method entailed a study into the aspects of glottal source information that are already contained within the spectral features commonly used in speech analysis, yielding an objective assessment regarding the expected advantages of explicitly using glottal information extracted from the speech signal via currently available IF methods, versus the alternative of relying on the glottal source information that is implicitly contained in spectral envelope representations.

This dissertation presents the development of a semi-supervised incremental learning framework with a multi-view perspective for image concept modeling. For reliable image concept characterization, having a large number of labeled images is crucial. However, the size of the training set is often limited due to the cost required for generating concept labels associated with objects in a large quantity of images. To address this issue, in this research, we propose to incrementally incorporate unlabeled samples into a learning process to enhance concept models originally learned with a small number of labeled samples. To tackle the sub-optimality problem of conventional techniques, the proposed incremental learning framework selects unlabeled samples based on an expected error reduction function that measures contributions of the unlabeled samples based on their ability to increase the modeling accuracy. To improve the convergence property of the proposed incremental learning framework, we further propose a multi-view learning approach that makes use of multiple features such as color, texture, etc., of images when including unlabeled samples. For robustness to mismatches between training and testing conditions, a discriminative learning algorithm, namely a kernelized maximal- figure-of-merit (kMFoM) learning approach is also developed. Combining individual techniques, we conduct a set of experiments on various image concept modeling problems, such as handwritten digit recognition, object recognition, and image spam detection to highlight the effectiveness of the proposed framework.

The term “Google” has become a verb for most of us. Search engines, however, have certain limitations. For example ask it for the impact of the current global financial crisis in different parts of the world, and you can expect to sift through thousands of results for the answer. This motivates the research in complex question answering where the purpose is to create summaries of large volumes of information as answers to complex questions, rather than simply offering a listing of sources. Unlike simple questions, complex questions cannot be answered easily as they often require inferencing and synthesizing information from multiple documents. Hence, this task is accomplished by the query-focused multidocument summarization systems. In this thesis we apply different supervised learning techniques to confront the complex question answering problem. To run our experiments, we consider the DUC-2007 main task. A huge amount of labeled data is a prerequisite for supervised training. It is expensive and time consuming when humans perform the labeling task manually. Automatic labeling can be a good remedy to this problem. We employ five different automatic annotation techniques to build extracts from human abstracts using ROUGE, Basic Element (BE) overlap, syntactic similarity measure, semantic similarity measure and Extended String Subsequence Kernel (ESSK). The representative supervised methods we use are Support Vector Machines (SVM), Conditional Random Fields (CRF), Hidden Markov Models (HMM) and Maximum Entropy (MaxEnt). We annotate DUC-2006 data and use them to train our systems, whereas 25 topics of DUC-2007 data set are used as test data. The evaluation results reveal the impact of automatic labeling methods on the performance of the supervised approaches to complex question answering. We also experiment with two ensemble-based approaches that show promising results for this problem domain. / x, 108 leaves : ill. ; 29 cm

Le Pronostic et l'étude de l'état de santé (en anglais Prognostics and Health Management (PHM)) vise à étendre le cycle de vie d'un actif physique, tout en réduisant les coûts d'exploitation et de maintenance. Pour cette raison, le pronostic est considéré comme un processus clé avec des capacités de prédictions. En effet, des estimations précises de la durée de vie avant défaillance d'un équipement, Remaining Useful Life (RUL), permettent de mieux définir un plan d'actions visant à accroître la sécurité, réduire les temps d'arrêt, assurer l'achèvement de la mission et l'efficacité de la production. Des études récentes montrent que les approches guidées par les données sont de plus en plus appliquées pour le pronostic de défaillance. Elles peuvent être considérées comme des modèles de type " boite noire " pour l'étude du comportement du système directement à partir des données de surveillance d'état, pour définir l'état actuel du system et prédire la progression future de défauts. Cependant, l'approximation du comportement des machines critiques est une tâche difficile qui peut entraîner des mauvais pronostics. Pour la compréhension de la modélisation de pronostic guidé par les données, on considère les points suivants. 1) Comment traiter les données brutes de surveillance pour obtenir des caractéristiques appropriées reflétant l'évolution de la dégradation ? 2) Comment distinguer les états de dégradation et définir des critères de défaillance (qui peuvent varier d'un cas à un autre)? 3) Comment être sûr que les modèles définis seront assez robustes pour montrer une performance stable avec des entrées incertaines s'écartant des expériences acquises, et seront suffisamment fiables pour intégrer des données inconnues (c'est à dire les conditions de fonctionnement, les variations de l'ingénierie, etc.)? 4) Comment réaliser facilement une intégration sous des contraintes et des exigences industrielles? Ces questions sont des problèmes abordés dans cette thèse. Elles ont conduit à développer une nouvelle approche allant au-delà des limites des méthodes classiques de pronostic guidé par les données. Les principales contributions sont les suivantes. <br>- L'étape de traitement des données est améliorée par l'introduction d'une nouvelle approche d'extraction des caractéristiques à l'aide de fonctions trigonométriques et cumulatives qui sont basées sur trois caractéristiques : la monotonie, la "trendability" et la prévisibilité. L'idée principale de ce développement est de transformer les données brutes en indicateur qui améliorent la précision des prévisions à long terme. <br>- Pour tenir compte de la robustesse, la fiabilité et l'applicabilité, un nouvel algorithme de prédiction est proposé: Summation Wavelet-Extreme Learning Machine (SWELM). Le SW-ELM assure de bonnes performances de prédiction, tout en réduisant le temps d'apprentissage. Un ensemble de SW-ELM est également proposé pour quantifier l'incertitude et améliorer la précision des estimations. <br>- Les performances du pronostic sont également renforcées grâce à la proposition d'un nouvel algorithme d'évaluation de la santé: Subtractive-Maximum Entropy Fuzzy Clustering (S-MEFC). S-MEFC est une approche de classification non supervisée qui utilise l'inférence de l'entropie maximale pour représenter l'incertitude de données multidimensionnelles. Elle peut automatiquement déterminer le nombre d'états, sans intervention humaine. <br>- Le modèle de pronostic final est obtenu en intégrant le SW-ELM et le S-MEFC pour montrer l'évolution de la dégradation de la machine avec des prédictions simultanées et l'estimation d'états discrets. Ce programme permet également de définir dynamiquement les seuils de défaillance et d'estimer le RUL des machines surveillées. Les développements sont validés sur des données réelles à partir de trois plates-formes expérimentales: PRONOSTIA FEMTO-ST (banc d'essai des roulements), CNC SIMTech (Les fraises d'usinage), C-MAPSS NASA (turboréacteurs) et d'autres données de référence. En raison de la nature réaliste de la stratégie d'estimation du RUL proposée, des résultats très prometteurs sont atteints. Toutefois, la perspective principale de ce travail est d'améliorer la fiabilité du modèle de pronostic.