Le développement du domaine de l’apprentissage profond doit une grande part de son avancée aux idées inspirées par la neuroscience et aux études sur l’apprentissage humain. De la découverte de l’algorithme de rétropropagation à la conception d’architectures neuronales comme les Convolutional Neural Networks, ces idées ont été couplées à l’ingénierie et aux améliorations technologiques pour engendrer des algorithmes performants en utilisation aujourd’hui. Cette thèse se compose de trois articles, chacun éclairant des aspects distincts du thème central de ce domaine interdisciplinaire. Le premier article explore la modélisation prédictive avec des données d’imagerie du cerveau de haute dimension en utilisant une nouvelle approche de régularisation hybride. Dans de nombreuses applications pratiques (comme l’imagerie médicale), l’attention se porte non seulement sur la précision, mais également sur l’interprétabilité d’un modèle prédictif formé sur des données haute dimension. Cette étude s’attache à combiner la régularisation l1 et l2, qui régularisent la norme des gradients, avec l’approche récemment proposée pour la modélisation prédictive robuste, l’Invariant Learning Consistency, qui impose l’alignement entre les gradients de la même classe lors de l’entraînement. Nous examinons ici la capacité de cette approche combinée à identifier des prédicteurs robustes et épars, et nous présentons des résultats prometteurs sur plusieurs ensembles de données. Cette approche tend à améliorer la robustesse des modèles épars dans presque tous les cas, bien que les résultats varient en fonction des conditions. Le deuxième article se penche sur les algorithmes d’apprentissage inspirés de la biologie, en se concentrant particulièrement sur la méthode Difference Target Propagation (DTP) tout en l’intégrant à l’optimisation Gauss-Newton. Le développement de tels algorithmes biologiquement plausibles possède une grande importance pour comprendre les processus d’apprentissage neuronale, cependant leur extensibilité pratique à des tâches réelles est souvent limitée, ce qui entrave leur potentiel explicatif pour l’apprentissage cérébral réel. Ainsi, l’exploration d’algorithmes d’apprentissage qui offrent des fondements théoriques solides et peuvent rivaliser avec la rétropropagation dans des tâches complexes gagne en importance. La méthode Difference Target Propagation (DTP) se présente comme une candidate prometteuse, caractérisée par son étroite relation avec les principes de l’optimisation Gauss-Newton. Néanmoins, la rigueur de cette relation impose des limites, notamment en ce qui concerne la formation couche par couche des poids synaptiques du chemin de rétroaction, une configuration considérée comme plus biologiquement plausible. De plus, l’alignement entre les mises à jour des poids DTP et les gradients de perte est conditionnel et dépend des scénarios d’architecture spécifiques. Cet article relève ces défis en introduisant un schéma innovant d’entraînement des poids de rétroaction. Ce schéma harmonise la DTP avec la BP, rétablissant la viabilité de la formation des poids de rétroaction couche par couche sans compromettre l’intégrité théorique. La validation empirique souligne l’efficacité de ce schéma, aboutissant à des performances exceptionnelles de la DTP sur CIFAR-10 et ImageNet 32×32. Enfin, le troisième article explore la planification efficace dans la prise de décision séquentielle en intégrant le calcul adaptatif à des architectures d’apprentissage profond existantes, dans le but de résoudre des casse-tête complexes. L’étude introduit des principes de calcul adaptatif inspirés des processus cognitifs humains, ainsi que des avancées récentes dans le domaine du calcul adaptatif. En explorant en profondeur les comportements émergents du modèle de mémoire adaptatif entraîné, nous identifions plusieurs comportements reconnaissables similaires aux processus cognitifs humains. Ce travail élargit la discussion sur le calcul adaptatif au-delà des gains évidents en efficacité, en explorant les comportements émergents en raison des contraintes variables généralement attribuées aux processus de la prise de décision chez les humains. / The development of the field of deep learning has benefited greatly from biologically inspired insights from neuroscience and the study of human learning more generally, from the discovery of backpropagation to neural architectures such as the Convolutional Neural Network. Coupled with engineering and technological improvements, the distillation of good strategies and algorithms for learning inspired from biological observation is at the heart of these advances. Although it would be difficult to enumerate all useful biases that can be learned by observing humans, they can serve as a blueprint for intelligent systems. The following thesis is composed of three research articles, each shedding light on distinct facets of the overarching theme. The first article delves into the realm of predictive modeling on high-dimensional fMRI data, a landscape where not only accuracy but also interpretability are crucial. Employing a hybrid approach blending l1 and l2 regularization with Invariant Learning Consistency, this study unveils the potential of identifying robust, sparse predictors capable of transmuting noise laden datasets into coherent observations useful for pushing the field forward. Conversely, the second article delves into the domain of biologically-plausible learning algorithms, a pivotal endeavor in the comprehension of neural learning processes. In this context, the investigation centers upon Difference Target Propagation (DTP), a prospective framework closely related to Gauss-Newton optimization principles. This exploration delves into the intricate interplay between DTP and the tenets of biologically-inspired learning mechanisms, revealing an innovative schema for training feedback weights. This schema reinstates the feasibility of layer-wise feedback weight training within the DTP framework, while concurrently upholding its theoretical integrity. Lastly, the third article explores the role of memory in sequential decision-making, and proposes a model with adaptive memory. This domain entails navigating complex decision sequences within discrete state spaces, where the pursuit of efficiency encounters difficult scenarios such as the risk of critical irreversibility. The study introduces adaptive computation principles inspired by human cognitive processes, as well as recent advances in adaptive computing. By studying in-depth the emergent behaviours exhibited by the trained adaptive memory model, we identify several recognizable behaviours akin to human cognitive processes. This work expands the discussion of adaptive computing beyond the obvious gains in efficiency, but to behaviours emerging due to varying constraints usually attributable to dynamic response times in humans.

An artificial neural network (ANN) is used to model nonlinear, large deformation plastic behavior of a material. This ANN model establishes a relationship between flow stress and dislocation structure content. The density of geometrically necessary dislocations (GNDs) was calculated based on analysis of local lattice curvature evolution. The model includes essential statistical measures extracted from the distributions of dislocation microstructures, including substructure cell size, wall thickness, and GND density as the input variables to the ANN model. The model was able to successfully predict the flow stress of aluminum alloy 6022 as a function of its dislocation structure content. Furthermore, a sensitivity analysis was performed to identify the significance of individual dislocation parameters on the flow stress. The results show that an ANN model can be used to calibrate and predict inelastic material properties that are often cumbersome to model with rigorous dislocation-based plasticity models.

No description available.

No description available.

[pt] A presente pesquisa tem como foco o desenvolvimento de abordagens híbridas que combinam algoritmos de aprendizado de máquina baseados em conjuntos (ensembles) e técnicas de modelagem e previsão de séries temporais. A pesquisa também inclui o desenvolvimento de heurísticas inteligentes de seleção, isto é, procedimentos capazes de selecionar, dentre o pool de preditores originados por meio dos métodos de conjunto, aqueles com os maiores potenciais de originar previsões agregadas mais acuradas. A agregação de funcionalidades de diferentes métodos visa à obtenção de previsões mais acuradas sobre o comportamento de uma vasta gama de eventos/séries temporais. A tese está dividida em uma sequência de ensaios. Como primeiro esforço, propôsse um método alternativo de geração de conjunto de previsões, o que resultou em previsões satisfatórias para certos tipos de séries temporais de consumo de energia elétrica. A segunda iniciativa consistiu na proposição de uma nova abordagem de previsão combinando algoritmos de Bootstrap Aggregation (Bagging) e técnicas de regularização para se obter previsões acuradas de consumo de gás natural e de abastecimento de energia em diferentes países. Uma nova variante de Bagging, na qual a construção do conjunto de classificadores é feita por meio de uma reamostragem de máxima entropia, também foi proposta. A terceira contribuição trouxe uma série de inovações na maneira pela qual são conduzidas as rotinas de seleção e combinação de modelos de previsão. Os ganhos em acurácia oriundos dos procedimentos propostos são demonstrados por meio de um experimento extensivo utilizando séries das Competições M1, M3 e M4. / [en] This research focuses on the development of hybrid approaches that combine ensemble-based supervised machine learning techniques and time series methods to obtain accurate forecasts for a wide range of variables and processes. It also includes the development of smart selection heuristics, i.e., procedures that can select, among the pool of forecasts originated via ensemble methods, those with the greatest potential of delivering accurate forecasts after aggregation. Such combinatorial approaches allow the forecasting practitioner to deal with different stylized facts that may be present in time series, such as nonlinearities, stochastic components, heteroscedasticity, structural breaks, among others, and deliver satisfactory forecasting results, outperforming benchmarks on many occasions. The thesis is divided into a series of essays. The first endeavor proposed an alternative method to generate ensemble forecasts which delivered satisfactory forecasting results for certain types of electricity consumption time series. In a second effort, a novel forecasting approach combining Bootstrap aggregating (Bagging) algorithms, time series methods and regularization techniques was introduced to obtain accurate forecasts of natural gas consumption and energy supplied series across different countries. A new variant of Bagging, in which the set of classifiers is built by means of a Maximum Entropy Bootstrap routine, was also put forth. The third contribution brought a series of innovations to model selection and model combination in forecasting routines. Gains in accuracy for both point forecasts and prediction intervals were demonstrated by means of an extensive empirical experiment conducted on a wide range of series from the M- Competitions.

<p>This thesis gives a partial answer to the question: “Can type systems detect modeling errors in scientific computing, particularly for inverse problems derived from physical models?” by considering, in detail, the major aspects of inverse problems in Magnetic Resonance Imaging (MRI). We define a type-system that can capture all correctness properties for MRI inverse problems, including many properties that are not captured with current type-systems, e.g., frames of reference. We implemented a type-system in the Haskell language that can capture the errors arising in translating a mathe- matical model into a linear or nonlinear system, or alternatively into an objective function. Most models are (or can be approximated by) linear transformations, and we demonstrate the feasibility of capturing their correctness at the type level using what is arguably the most difficult case, the (discrete) Fourier transformation (DFT). By this, we mean that we are able to catch, at compile time, all known errors in ap- plying the DFT. The first part of this thesis describes the Haskell implementation of vector size, physical units, frame of reference, and so on required in the mathemat- ical modelling of inverse problems without regularization. To practically solve most inverse problems, especially those including noisy data or ill-conditioned systems, one must use regularization. The second part of this thesis addresses the question of defining new regularizers and identifying existing regularizers the correctness of which (in our estimation) can be formally verified at the type level. We describe such Bayesian regularization schemes based on probability theory, and describe a novel simple regularizer of this type. We leave as future work the formalization of such regularizers.</p> / Master of Science (MSc)

Modern deep neural networks are proficient at solving various visual recognition and understanding tasks, as long as a sufficiently large labeled dataset is available during the training time. However, the progress of these visual tasks is limited by the number of manual annotations. On the other hand, it is usually time-consuming and error-prone to annotate visual data, rendering the challenge of scaling up human labeling for many visual tasks. Fortunately, it is easy to collect large-scale, diverse unlabeled visual data from the Internet. And we can acquire a large amount of synthetic visual data with annotations from game engines effortlessly. In this dissertation, we explore how to utilize the unlabeled data and synthetic labeled data for various visual tasks, aiming to replace or reduce the direct supervision from the manual annotations. The key idea is to encourage deep neural networks to produce consistent predictions across different transformations (\eg geometry, temporal, photometric, etc.). We organize the dissertation as follows. In Part I, we propose to use the consistency over different geometric formulations and a cycle consistency over time to tackle the low-level scene geometry perception tasks in a self-supervised learning setting. In Part II, we tackle the high-level semantic understanding tasks in a semi-supervised learning setting, with the constraint that different augmented views of the same visual input maintain consistent semantic information. In Part III, we tackle the cross-domain image segmentation problem. By encouraging an adaptive segmentation model to output consistent results for a diverse set of strongly-augmented synthetic data, the model learns to perform test-time adaptation on unseen target domains with one single forward pass, without model training or optimization at the inference time. / Doctor of Philosophy / Recently, deep learning has emerged as one of the most powerful tools to solve various visual understanding tasks. However, the development of deep learning methods is significantly limited by the amount of manually labeled data. On the other hand, it is usually time-consuming and error-prone to annotate visual data, making the human labeling process not easily scalable. Fortunately, it is easy to collect large-scale, diverse raw visual data from the Internet (\eg search engines, YouTube, Instagram, etc.). And we can acquire a large amount of synthetic visual data with annotations from game engines effortlessly. In this dissertation, we explore how we can utilize the raw visual data and synthetic data for various visual tasks, aiming to replace or reduce the direct supervision from the manual annotations. The key idea behind this is to encourage deep neural networks to produce consistent predictions of the same visual input across different transformations (\eg geometry, temporal, photometric, etc.). We organize the dissertation as follows. In Part I, we propose using the consistency over different geometric formulations and a forward-backward cycle consistency over time to tackle the low-level scene geometry perception tasks, using unlabeled visual data only. In Part II, we tackle the high-level semantic understanding tasks using both a small amount of labeled data and a large amount of unlabeled data jointly, with the constraint that different augmented views of the same visual input maintain consistent semantic information. In Part III, we tackle the cross-domain image segmentation problem. By encouraging an adaptive segmentation model to output consistent results for a diverse set of strongly-augmented synthetic data, the model learns to perform test-time adaptation on unseen target domains.

For many scientific applications, inverse problems have played a key role in solving important problems by enabling researchers to estimate desired parameters of a system from observed measurements. For example, large-scale inverse problems arise in many global problems and medical imaging problems such as greenhouse gas tracking and computational tomography reconstruction. This dissertation describes advancements in computational tools for solving large-scale inverse problems and for uncertainty quantification. Oftentimes, inverse problems are ill-posed and large-scale. Iterative projection methods have dramatically reduced the computational costs of solving large-scale inverse problems, and regularization methods have been critical in obtaining stable estimations by applying prior information of unknowns via Bayesian inference. However, by combining iterative projection methods and variational regularization methods, hybrid projection approaches, in particular generalized hybrid methods, create a powerful framework that can maximize the benefits of each method. In this dissertation, we describe various advancements and extensions of hybrid projection methods that we developed to address three recent open problems. First, we develop hybrid projection methods that incorporate mixed Gaussian priors, where we seek more sophisticated estimations where the unknowns can be treated as random variables from a mixture of distributions. Second, we describe hybrid projection methods for mean estimation in a hierarchical Bayesian approach. By including more than one prior covariance matrix (e.g., mixed Gaussian priors) or estimating unknowns and hyper-parameters simultaneously (e.g., hierarchical Gaussian priors), we show that better estimations can be obtained. Third, we develop computational tools for a respirometry system that incorporate various regularization methods for both linear and nonlinear respirometry inversions. For the nonlinear systems, blind deconvolution methods are developed and prior knowledge of nonlinear parameters are used to reduce the dimension of the nonlinear systems. Simulated and real-data experiments of the respirometry problems are provided. This dissertation provides advanced tools for computational inversion and uncertainty quantification. / Doctor of Philosophy / For many scientific applications, inverse problems have played a key role in solving important problems by enabling researchers to estimate desired parameters of a system from observed measurements. For example, large-scale inverse problems arise in many global problems such as greenhouse gas tracking where the problem of estimating the amount of added or removed greenhouse gas at the atmosphere gets more difficult. The number of observations has been increased with improvements in measurement technologies (e.g., satellite). Therefore, the inverse problems become large-scale and they are computationally hard to solve. Another example of an inverse problem arises in tomography, where the goal is to examine materials deep underground (e.g., to look for gas or oil) or reconstruct an image of the interior of the human body from exterior measurements (e.g., to look for tumors). For tomography applications, there are typically fewer measurements than unknowns, which results in non-unique solutions. In this dissertation, we treat unknowns as random variables with prior probability distributions in order to compensate for a deficiency in measurements. We consider various additional assumptions on the prior distribution and develop efficient and robust numerical methods for solving inverse problems and for performing uncertainty quantification. We apply our developed methods to many numerical applications such as greenhouse gas tracking, seismic tomography, spherical tomography problems, and the estimation of CO2 of living organisms.

Develop new strategies to design and operate the multilateration systems, used for air traffic control operations, in a more efficient way. The design strategies are based on the utilization of metaheuristic optimization techniques and they are intended to found the optimal spatial distribution of the system ground stations, taking into account the most relevant system operation parameters. The strategies to operate the systems are based on the development of new positioning methods which allow solving the problems of uncertainty position and poor accuracy that the current systems can present. The new strategies can be applied to design, deploy and operate the multilateration systems for airport surface surveillance as well as takeoff-landing, approach and enroute control. An important advance in the current knowledge of air traffic control is expected from the development of these strategies, because they solve several deficiencies that have been made clear, by the international scientific community, in the last years. / Mantilla Gaviria, IA. (2013). New Strategies to Improve Multilateration Systems in the Air Traffic Control [Tesis doctoral]. Editorial Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/29688

[pt] A análise de agrupamento não supervisionado, o processo que consistem em agrupar conjuntos de pontos de acordo com um ou mais critérios de similaridade, tem desempenhado um papel essencial em vários campos. O dois algoritmos mais populares para esse processão são o k-means e o Gaussian Mixture Models (GMM). O primeiro atribui cada ponto a um único cluster e usa a distância Euclidiana como similaridade. O último determina uma matriz de probabilidade de pontos pertencentes a clusters, e usa distância de Mahalanobis como similaridade. Além da diferença no método de atribuição - a chamada atribuição hard para o primeiro e a atribuição soft para o último - os algoritmos também diferem em relação à estrutura do cluster, ou forma: o k-means considera estruturas esféricas no dados; enquanto o GMM considera elipsoidais através da estimação de matrizes de covariância. Neste trabalho, um problema de otimização matemática que combina a atribuição hard com a estrutura do cluster elipsoidal é detalhado e formulado. Uma vez que a estimativa da covariância desempenha um papel importante no comportamento de estruturas agrupamentos elipsoidais, técnicas de regularizações são exploradas. Neste contexto, dois métodos de meta-heurística, uma perturbação Random Swap e um algoritmo híbrido genético, são adaptados, e seu impacto na melhoria do desempenho dos métodos é estudado. O objetivo central dividido em três: compreender as condições em que as estruturas de agrupamento elipsoidais são mais benéficas que as esféricas; determinar o impacto da estimativa de covariância com os métodos de regularização; e analisar o efeito das meta-heurísticas de otimização global na análise de agrupamento não supervisionado. Finalmente, a fim de fornecer bases para a comparação das presentes descobertas com futuros trabalhos relacionados, foi gerada uma base de dados com um extenso benchmark contendo análise das variações de diferentes tamanhos, formas, número de grupos e separabilidade, e seu impacto nos resultados de diferentes algoritmos de agrupamento. Além disso, pacotes escritos na linguagem Julia foram disponibilizados com os algoritmos estudados ao longo deste trabalho. / [en] Unsupervised cluster analysis, the process of grouping sets of points according to one or more similarity criteria, plays an essential role in various fields. The two most popular algorithms for this process are the k-means and the Gaussian Mixture Models (GMM). The former assigns each point to a single cluster and uses Euclidean distance as similarity. The latter determines a probability matrix of points to belong to clusters, and the Mahalanobis distance is the underlying similarity. Apart from the difference in the assignment method - the so-called hard assignment for the former and soft assignment for the latter - the algorithms also differ concerning the cluster structure, or shape: the k-means considers spherical structures in the data; while the GMM considers ellipsoidal ones through the estimation of covariance matrices. In this work, a mathematical optimization problem that combines the hard assignment with the ellipsoidal cluster structure is detailed and formulated. Since the estimation of the covariance plays a major role in the behavior of ellipsoidal cluster structures, regularization techniques are explored. In this context, two meta-heuristic methods, a Random Swap perturbation and a hybrid genetic algorithm, are adapted, and their impact on the improvement of the performance of the methods is studied. The central objective is three-fold: to gain an understanding of the conditions in which ellipsoidal clustering structures are more beneficial than spherical ones; to determine the impact of covariance estimation with regularization methods; and to analyze the effect of global optimization meta-heuristics on unsupervised cluster analysis. Finally, in order to provide grounds for comparison of the present findings to future related works, a database was generated together with an extensive benchmark containing an analysis of the variations of different sizes, shapes, number of clusters, and separability and their impact on the results of different clustering algorithms. Furthermore, packages written in the Julia language have been made available with the algorithms studied throughout this work.