In the era of Internet of Things (IoT) and big data, collecting, processing and analyzing enormous data faces unprecedented challenges even when being stored in preprocessed form. Anomaly detection, statistically viewed as identifying outliers having low probabilities from the modelling of data distribution p(x), becomes more crucial. In this Master thesis, two (supervised and unsupervised) novel deep anomaly detection frameworks are presented which can achieve state-of-art performance on a range of datasets. Capsule net is an advanced artificial neural network, being able to encode intrinsic spatial relationship between parts and a whole. This property allows it to work as both a classifier and a deep autoencoder. Taking this advantage of CapsNet, a new anomaly detection technique named AnoCapsNet is proposed and three normality score functions are designed: prediction-probability-based (PP-based) normality score function, reconstruction-error-based (RE-based) normality score function, and a normality score function that combines prediction-probability-based and reconstruction-error-based together (named as PP+RE-based normality score function) for evaluating the "outlierness" of unseen images. The results on four datasets demonstrate that the PP-based method performs consistently well, while the RE-based approach is relatively sensitive to the similarity between labeled and unlabeled images. The PP+RE-based approach effectively takes advantages of both methods and achieves state-of-the-art results. In many situations, neither the domain of anomalous samples can be fully understood, nor the domain of the normal samples is straightforward. Thus deep generative models are more suitable than supervised methods in such cases. As a variant of variational autoencoder (VAE), beta-VAE is designed for automated discovery of interpretable factorised latent representations from raw image data in a completely unsupervised manner. The t-Distributed Stochastic Neighbor Embedding (t-SNE), an unsupervised non-linear technique primarily used for data exploration and visualizing high-dimensional data, has advantages at creating a single map that reveals local and important global structure at many different scales. Taking advantages of both disentangled representation learning (using beta-VAE as an implementation) and low-dimensional neighbor embedding (using t-SNE as an implementation), another novel anomaly detection approach named AnoDM (stands for Anomaly detection based on unsupervised Disentangled representation learning and Manifold learning) is presented. A new anomaly score function is defined by combining (1) beta-VAE's reconstruction error, and (2) latent representations' distances in the t-SNE space. This is a general framework, thus any disentangled representation learning and low-dimensional embedding techniques can be applied. AnoDM is evaluated on both image and time-series data and achieves better results than models that use just one of the two measures and other existing advanced deep learning methods.

Two different catalysts (bis[N-(3-tert-butylsalicylidene)-pentafluoroanilinato] titanium (IV) dichloride and [1-(8-quinolyl)indenyl] chromium (III) dichloride catalysts) activated with aluminoxane based co-catalysts (MAO, PMAO, MMAO12 and MMAO3A) have been evaluated in the polymerisation of ethylene leading to UHMWPE having a reduced number of entanglements between the chains. The effect of a co-catalyst modifier (BHT) on the catalytic systems and the resulting polymers is also addressed. Both catalysts are capable to promote the synthesis of UHMWPE having a reduced amount of entanglements in the conditions used. Uniaxial solid-state deformation of UHMWPE samples of different molar masses have been evaluated and related with the entanglement state of the polymers synthesised. A clear relationship between some mechanical properties and the molar mass/entanglement density of the polymers synthesised has been found.

Ultra High Molecular Weight Polyethylene (UHMw-PE) is an engineering polymer that is widely used in demanding applications because of its un-paralleled properties such as high abrasion resistance, high-modulus and high-strength tapes and fibres, biaxial films etc. In common practice, to achieve the uniaxial and the biaxial products, the solution processing route is adopted to reduce the number of entanglements per chain, such as found in Dyneema(R) from DSM(R). Another elegant route to reduce the number of entanglements to ease solid-state processing is through controlled polymerisation using a single-site catalytic system. In this theses, how different polymerisation condition, such as temperature and time control molecular weight and the resultant entangled state in synthesised disentangled UHMw-PE is addressed. Linear dynamic melt rheology is used to follow entanglement formation in an initially disentangled melt. With the help of rheological studies, heterogeneity in the distribution of entanglements along the chain length and the crystal morphology produced during polymerisation is considered. For the understanding of influence of large shear flow on melt dynamics large amplitude oscillatory shear (LAOS) is used and the non-linear viscoelastic regime is explored. A remarkable feature of overshoot in loss (viscous) modulus with increasing deformation (strain) in UHMw-PE melt in the LAOS is observed. This observation is characteristic of colloidal systems. The role of entanglement density in the amorphous region of the synthesised disentangled UHMw-PE (semi-crystalline polymers) on the melting and crystallisation is presented. To understand the effect of topological differences on melting behaviour, nascent entangled, nascent disentangled and melt-crystallised samples have been used. The role of superheating on the melting process is also addressed. Preliminary results on characteristic melting time of a crystal using TM-DSC are also presented.

We introduce Domain Aware Continual Zero-Shot Learning (DACZSL), the task of visually recognizing images of unseen categories in unseen domains sequentially. We created DACZSL on top of the DomainNet dataset by dividing it into a sequence of tasks, where classes are incrementally provided on seen domains during training and evaluation is conducted on unseen domains for both seen and unseen classes. We also proposed a novel Domain-Invariant CZSL Network (DIN), which outperforms state-of-the-art baseline models that we adapted to DACZSL setting. We adopt a structure-based approach to alleviate forgetting knowledge from previous tasks with a small per-task private network in addition to a global shared network. To encourage the private network to capture the domain and task-specific representation, we train our model with a novel adversarial knowledge disentanglement setting to make our global network task-invariant and domain-invariant over all the tasks. Our method also learns a class-wise learnable prompt to obtain better class-level text representation, which is used to represent side information to enable zero-shot prediction of future unseen classes. Our code and benchmarks are made available at https://zero-shot-learning.github.io/daczsl.

Severe acute respiratory syndrome (SARS)-CoV-2 emerged in late 2019, then became an unprecedented public health crisis. Hundreds of millions of people have been affected. What is worse, many researchers have revealed that COVID-19 may have long-term effects on varieties of organs even after recovery. Consequently, there is a need for the study of its sequelae. The purpose of this project is to use machine learning algorithms to study the relationship between patients’ EMR data and long-term sequelae, especially kidney diseases. Inspired by a recent learning disentangled representation for recommendation work, this project proposes a method that (i) predicts the development trend of the kidney disease; (ii) learn representations that uncover and disentangle factors related to kidney diseases. The major contribution is that this model has high interpretability which enables medical works to infer the development of patients' condition.

Deep Neural Networks have long been considered black box systems, where their interpretability is a concern when applied in safety critical systems. In this work, a novel approach of interpreting the decisions of DNNs is proposed. The approach depends on exploiting generative models and the interpretability of their latent space. Three methods for ranking features are explored, two of which depend on sensitivity analysis, and the third one depends on Random Forest model. The Random Forest model was the most successful to rank the features, given its accuracy and inherent interpretability.

Regardless of the simplicity in molecular structure, polyethylene is used in high-performance applications such as medical prostheses and ballistics. Recent advancements in homogeneous catalysis produced UHMWPE in the low-entangled or dis-entangled state that allowed solvent-free-solid-state processing to achieve ultimate mechanical properties ever achieved for a synthetic polymer. Although several homogeneous complexes are known to produce dis-UHMWPE, existing major challenges are uncontrolled nascent polymer morphology, as a consequence reactor fouling/wall sheeting. In such a scenario, a heterogeneous catalyst that can produce dis-UHWMPE to an extent that the characteristics and properties equivalent to that obtained in homogeneous condition, remains an open challenge. The thesis will discuss the know-how for the synthesis of dis-UHMWPE via heterogeneous route to facilitate industrial production by following fundamental understanding of polymerization catalysis, physics, processing, and testing. In this thesis, in-situ formed nano activator/support MgClx/EtnAly(2-ethyl-1-hexoxide)z is employed with a highly active bis[N-(3-tert-butylsalicylidene)pentafluoroanilinato] titanium (IV) dichloride (Cat. 1) for synthesis of dis-UHMWPE. In addition, the relatively easy formation of the MgClx/RnClmAly(OR’) activators/supports allows tailoring by the selection of different aluminum-alkyls and alcohols, giving access to a variety of co-catalysts. This investigation resulted in UHMWPE having Mw from 3 to an unprecedented 43 M g/mol and Ð from 3 to 38 with very high activities up to 2750 kgPE molcat.-1 bar-1 h-1. The adopted route resulted in nano-support that allows tailoring of the entangled state and control over the nascent morphology without reactor fouling, thus providing feasibility of pursuing the polymerization via a continuous process. The nascent polymer shows formation of single crystals of linear UHMWPE and is suggestive of the low-entangled state. The topological differences, with the commercial entangled sample, are identified solid-state NMR, DSC, and rheology. The disentangled crystals allowed desired chain orientation for securing unprecedented tensile modulus (>200 N/tex) and tensile strength (>4.0 N/tex) via solid-state processing. Additionally, the investigation of creep response in the uniaxial tapes has revealed strong influence of molecular weight and entanglement density. These unique characteristics and unprecedented mechanical properties are equivalent to that perceived using a homogeneous catalysis and are the first of their kind achieved for a polymer synthesized using a heterogeneous catalysis.

Numerical methods, the collective name for numerical analysis and optimization techniques, have been widely used in the field of computer vision and deep learning. In this thesis, we investigate the algorithms of some numerical methods and their relevant applications in deep learning. These studied numerical techniques mainly include differentiable matrix power functions, differentiable eigendecomposition (ED), feasible orthogonal matrix constraints in optimization and latent semantics discovery, and physics-informed techniques for solving partial differential equations in disentangled and equivariant representation learning. We first propose two numerical solvers for the faster computation of matrix square root and its inverse. The proposed algorithms are demonstrated to have considerable speedup in practical computer vision tasks. Then we turn to resolve the main issues when integrating differentiable ED into deep learning -- backpropagation instability, slow decomposition for batched matrices, and ill-conditioned input throughout the training. Some approximation techniques are first leveraged to closely approximate the backward gradients while avoiding gradient explosion, which resolves the issue of backpropagation instability. To improve the computational efficiency of ED, we propose an efficient ED solver dedicated to small and medium batched matrices that are frequently encountered as input in deep learning. Some orthogonality techniques are also proposed to improve input conditioning. All of these techniques combine to mitigate the difficulty of applying differentiable ED in deep learning. In the last part of the thesis, we rethink some key concepts in disentangled representation learning. We first investigate the relation between disentanglement and orthogonality -- the generative models are enforced with different proposed orthogonality to show that the disentanglement performance is indeed improved. We also challenge the linear assumption of the latent traversal paths and propose to model the traversal process as dynamic spatiotemporal flows on the potential landscapes. Finally, we build probabilistic generative models of sequences that allow for novel understandings of equivariance and disentanglement. We expect our investigation could pave the way for more in-depth and impactful research at the intersection of numerical methods and deep learning.

Dans les annales de l'Intelligence Artificielle (IA), la quête incessante pour émuler la cognition humaine dans les machines a sous-tendu l'évolution technologique, repoussant les limites du potentiel humain et des capacités de résolution de problèmes. L'intégration de l'IA a catalysé des progrès remarquables, pénétrant divers domaines et redéfinissant des industries. Cependant, un défi demeure imperturbable : l'obstacle de la généralisation hors de la distribution (OOD). Alors que l'IA triomphe avec des données familières, elle échoue avec des données en dehors de son domaine d'entraînement. En santé, en finance et au-delà, les limitations de l'IA entravent l'adaptation à des scénarios nouveaux. Cette lacune découle de l'écart entre les schémas appris et les caractéristiques causales et invariantes sous-jacentes, entravant l'adaptabilité à des scénarios inexplorés. Cette thèse franchit des étapes significatives pour aborder cette question en innovant et en exploitant des méthodes issues de l'apprentissage de structure causale et de représentation. Le parcours commence par un algorithme novateur d'apprentissage de structure, les ``Reusable Factor Graphs'', qui tire parti des biais inductifs issus de la causalité et de la cognition humaine pour une meilleure généralisation. Ensuite, en explorant l'apprentissage de représentation causale, nous découvrons des représentations désenchevêtrées centrées sur les objets en utilisant une supervision faible basée sur une connaissance partielle de la structure causale des données. Ces connaissances se conjuguent pour préconiser l'apprentissage conjoint de la structure causale et de la représentation. L'architecture proposée, les ``Reusable Slotwise Mechanisms'' (RSM), relie théorie et pratique, démontrant une promesse réelle à travers ses représentations centrées sur les objets et ses mécanismes causaux réutilisables. Cette fusion offre une solution potentielle pour surmonter les limitations de la généralisation OOD en IA. / In the annals of Artificial Intelligence (AI), an enduring quest to emulate human cognition in machines has underpinned technological evolution, driving the boundaries of human potential and problem-solving capabilities. The integration of AI has catalyzed remarkable progress, infiltrating various domains and redefining industries. Yet, a challenge remains unshaken: the hurdle of out-of-distribution (OOD) generalization. While AI triumphs with familiar data, it falters with data outside its training realm. In healthcare, finance, and beyond, AI's limitations hinder adaptation to novel scenarios. This deficiency arises from the gap between learned patterns and underlying causal and invariant features, hindering adaptability to uncharted scenarios. This thesis takes significant steps toward tackling this issue by innovating and leveraging methods from causal structure and representation learning. The journey begins with an innovative structure learning algorithm, Reusable Factor Graphs, leveraging inductive biases from causality and human cognition for improved generalization. Next, delving into causal representation learning, we uncover object-centric disentangled representations using weak supervision from partial knowledge of the causal structure of data. These insights synergize in advocating joint learning of causal structure and representation. The proposed Reusable Slotwise Mechanisms (RSM) architecture bridges theory and practice, demonstrating real-world promise through its object-centric representations and reusable causal mechanisms. This fusion offers a potential solution for tackling OOD generalization limitations in AI.