Machine learning has recently made significant progress due to modern neural network architectures and training procedures. When neural networks learn a task, they create internal representations of the input data. The specific neural network architecture, training process, and task being addressed will influence the way in which the neural network interprets and explains the patterns in the data. The goal of representation learning is to train the neural network to create representations that effectively capture the overall structure of the data. However, the process by which these representations are generated is not fully understood because of the complexity of neural network data manipulations. This makes it difficult to choose the correct training procedure in real world applications. In this dissertation, we apply representation learning to improve the performance of neural networks in three different areas: NBA movement data, material property prediction, and generative protein modeling. First, we propose a novel deep learning approach for predicting human trajectories in sporting events using advanced object tracking data. Our method leverages recent advances in deep learning techniques, including the use of recurrent neural networks and long short-term memory cells, to accurately predict the future movements of players and the ball in a basketball game. We evaluate our approach using data from the NBA's advanced object tracking system and demonstrate improved performance compared to existing methods. Our results have the potential to inform real-time decision making in sports analytics and improve the understanding of player behavior and strategy. Next, we focused on group activity recognition (GAR) in basketball. In basketball, players engage in various activities, both collaborative and adversarial, in order to win the game. Identifying and analyzing these activities is important for sports analytics as it can inform better strategies and decisions by players and coaches. We introduce a novel deep learning approach for GAR in team sports called NETS. NETS utilizes a Transformer-based architecture combined with LSTM embedding and a team-wise pooling layer to recognize group activity. We test NETS using tracking data from 632 NBA games and found that it was able to learn group activities with high accuracy. Additionally, self- and weak-supervised training in NETS improved the accuracy of GAR. Then, study an application of neural networks on protein modeling. Recent work on autoregressive direct coupling analysis (arDCA) has shown promising potential to efficiently train a generative protein sequence model (GPSM) to adequately model protein sequence data. We propose an extension to this work by adding a higher order coupling estimator to build a model called autoregressive higher order coupling analysis (arHCA). We show that our model can correctly identify higher order couplings in a synthetic dataset and that our model improves the performance of arDCA when trained on real-world sequence data. Finally, we study material property prediction. Incorporation of physical principles in a machine learning (ML) architecture is a fundamental step toward the continued development of AI for inorganic materials. As inspired by the Pauling’s rule, we propose that structure motifs in inorganic crystals can serve as a central input to a machine learning framework. To demonstrate the use of structure motif information, a motif-centric learning framework is created by combining motif information with the atom-based graph neural networks to form an atom-motif dual graph network (AMDNet), which is more accurate in predicting the electronic structures of metal oxides such as bandgaps. The work illustrates the route toward fundamental design of graph neural network learning architecture for complex materials by incorporating beyond-atom physical principles. / Computer and Information Science

Convolutional Neural Networks (CNNs) have dominated the task of imageclassification since 2012. Some key components of their success are that the underlying architecture integrates a set inductive biases such as translational invariance and the training computation can be significantly reduced by employing weight sharing. CNNs are powerful tools for generating new representations of images tailored to a particular task such as classification. However, because each image is passed through the network independent of other images, CNNs are not able to effectively aggregate information between examples. In this thesis, we explore the idea of using Graph Neural Networks (GNNs) in conjunction with CNNs to produce an architecture that has both the representational capacity of a CNN and the ability to aggregate information between examples. Graph Neural Networks apply the concept of convolutions directly on graphs. A result of this is that GNNs are able to learn from the connections between nodes. However, when working with image datasets, there is no obvious choice on how to construct a graph. There are certain heuristics such as ensuring homophily that have empirically been shown to increase the performance of GNNs. In this thesis, we apply different schemes of constructing a graph from image data for the downstream task of image classification and experiment with settings such as using multiple feature spaces and enforcing a bipartite graph structure. We also propose a model that allows for end to end training using CNNs and GNNs with proxies and attention that improves classification accuracy in comparison to a regular CNN. / Computer and Information Science

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第24725号 / 情博第813号 / 新制||情||137(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 鹿島 久嗣, 教授 山本 章博, 教授 吉川 正俊 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

MRI is favorable for brain imaging due to its excellent soft tissue contrast and absence of harmful ionizing radiation. Many have proposed supervised multimodal neural networks for automatic brain tumor segmentation and showed promising results. However, they rely on large amounts of labeled data to generalize well. The trained network is also highly specific to the task and input. Missing inputs will most likely have a detrimental effect on the network’s predictions, if it works at all. The aim of this thesis work is to implement a deep neural network that learns the general representation of multimodal MRI images in an unsupervised manner and is insensitive to missing modalities. With the latent representation, labeled data are then used for brain tumor segmentation. A variational autoencoder and an unified representation network are used for repre- sentation learning. Fine-tuning or joint training was used for segmentation task. The performances of the algorithms at the reconstruction task was evaluated using the mean- squared error and at the segmentation task using the Dice coefficient. Both networks demonstrated the possibility in learning brain MR representations, but the unified representation network was more successful at the segmentation task.

This work is devoted to self-supervised representation learning (SSL). We consider both contrastive and non-contrastive methods and present a new loss function for SSL based on feature whitening. Our solution is conceptually simple and competitive with other methods. Self-supervised representations are beneficial for most areas of deep learning, and reinforcement learning is of particular interest because SSL can compensate for the sparsity of the training signal. We present two methods from this area. The first tackles the partial observability providing the agent with a history, represented with temporal alignment, and improves performance in most Atari environments. The second addresses the exploration problem. The method employs a world model of the SSL latent space, and the prediction error of this model indicates novel states required to explore. It shows strong performance on exploration-hard benchmarks, especially on the notorious Montezuma's Revenge. Finally, we consider the metric learning problem, which has much in common with SSL approaches. We present a new method based on hyperbolic embeddings, vision transformers and contrastive loss. We demonstrate the advantage of hyperbolic space over the widely used Euclidean space for metric learning. The method outperforms the current state-of-the-art by a significant margin.

Many machine learning problems involve dealing with a large amount of high-dimensional data across diverse domains. In addition, annotating or labeling the data is expensive as it involves significant human effort. This dissertation explores a joint solution to both these problems by exploiting the property that high-dimensional data in real-world application domains often lies on a lower-dimensional structure, whose geometry can be modeled as a graph or manifold. In particular, we propose a set of novel manifold-alignment based approaches for transfer learning. The proposed approaches transfer knowledge across different domains by finding low-dimensional embeddings of the datasets to a common latent space, which simultaneously match corresponding instances while preserving local or global geometry of each input dataset. We develop a novel two-step transfer learning method called Procrustes alignment. Procrustes alignment first maps the datasets to low-dimensional latent spaces reflecting their intrinsic geometries and then removes the translational, rotational and scaling components from one set so that the optimal alignment between the two sets can be achieved. This approach can preserve either global geometry or local geometry depending on the dimensionality reduction approach used in the first step. We propose a general one-step manifold alignment framework called manifold projections that can find alignments, both across instances as well as across features, while preserving local domain geometry. We develop and mathematically analyze several extensions of this framework to more challenging situations, including (1) when no correspondences across domains are given; (2) when the global geometry of each input domain needs to be respected; (3) when label information rather than correspondence information is available. A final contribution of this thesis is the study of multiscale methods for manifold alignment. Multiscale alignment automatically generates alignment results at different levels by discovering the shared intrinsic multilevel structures of the given datasets, providing a common representation across all input datasets.

With increasing data and technology, multiple types of data from the same set of nodes have been generated. Since each data modality contains a unique aspect of the underlying mechanisms, multiple datatypes are integrated. In addition to multiple datatypes, networks are important to store information representing associations between entities such as genes of a protein-protein interaction network and authors of a citation network. Recently, some advanced approaches to graph-structured data leverage node associations and features simultaneously, called Graph Neural Network (GNN), but they have limitations for integrative approaches. The overall aim of this dissertation is to integrate multiple data modalities on graph-structured data to infer some context-specific gene regulation and predict outcomes of interest. To this end, first, we introduce a computational tool named CRINET to infer genome-wide competing endogenous RNA (ceRNA) networks. By integrating multiple data properly, we had a better understanding of gene regulatory circuitry addressing important drawbacks pertaining to ceRNA regulation. We tested CRINET on breast cancer data and found that ceRNA interactions and groups were significantly enriched in the cancer-related genes and processes. CRINET-inferred ceRNA groups supported the studies claiming the relation between immunotherapy and cancer. Second, we present SUPREME, a node classification framework, by comprehensively analyzing multiple data and associations between nodes with graph convolutions on multiple networks. Our results on survival analysis suggested that SUPREME could demystify the characteristics of classes with proper utilization of multiple data and networks. Finally, we introduce an attention-aware fusion approach, called GRAF, which fuses multiple networks and utilizes attention mechanisms on graph-structured data. Utilization of learned node- and association-level attention with network fusion allowed us to prioritize the edges properly, leading to improvement in the prediction results. Given the findings of all three tools and their outperformance over state-of-the-art methods, the proposed dissertation shows the importance of integrating multiple types of data and the exploitation of multiple graph structured data.

Internet services, such as online shopping and chat apps, have been spreading significantly in recent years, generating substantial amounts of data. These data are precious for machine learning and consist of connections between different entities, such as users and items. These connections contain important information essential for ML models to exploit, and the need to extract this information from graphs gives rise to Graph Representation Learning. By training on these data using Graph Representation Learning methods, hidden information can be obtained, and services can be improved. Initially, the models used for Graph Representation Learning were unsupervised, such as the Deepwalk and Node2vec. These models originated from the field of Natural Language Processing. These models are easy to apply, but their performance is not satisfactory. On the other hand, while supervised models like GNN and GCN have better performance than unsupervised models, they require a huge effort to label the data and finetune the model. Nowadays, the datasets have become larger and more complex, which makes the burden heavier for applying these supervised models. A recent breakthrough in the field of Natural Language Processing may solve the problem. In the paper ‘Attention is all you need’, the authors introduce the Transformer model, which shows excellent performance in NLP. Considering that the field of NLP has many things in common with the GRL and the first supervised models all originated from NLP, it is reasonable to guess whether we can take advantage of the Transformer in improving the performance of the unsupervised model in GRL. Generating embedding for nodes in the graph is one of the significant tasks of GRL. In this thesis, the performance of the Transformer model on generating embedding is tested. Three popular datasets (Cora, Citeseer, Pubmed) are used in training, and the embedding quality is measured through node classification with a linear classification algorithm. Another part of the thesis is to finetune the model to determine the effect of model parameters on embedding accuracy. In this part, comparison experiments are conducted on the dimensions, the number of layers, the sample size, and other parameters. The experiments show that the Transformer model performs better in generating embedding than the original methods, such as the Deepwalk. Compared to supervised methods, it requires less finetuning and less training time. The characteristic of the Transformer model revealed from the experiments shows that it is a good alternative to the baseline model for embedding generation. Improvement may be made on the prepossessing and loss function of the model to get higher performance. / Internettjänster, som onlineshopping och chattappar, har spridits avsevärt de senaste åren och genererat betydande mängder data. Dessa data är värdefulla för maskininlärning och består av kopplingar mellan olika enheter, såsom användare och objekt. Dessa kopplingar innehåller viktig information som är väsentlig för ML-modeller att utnyttja, och behovet av att extrahera denna information från grafer ger upphov till Graph Representation Learning. Genom att träna på dessa data med hjälp av Graph Representation Learning-metoder kan dold information erhållas och tjänster kan förbättras. Till en början var modellerna som användes för Graph Representation Learning oövervakade, såsom Deepwalk och Node2vec. Dessa modeller härstammar från området Natural Language Processing. Dessa modeller är lätta att applicera, men deras prestanda är inte tillfredsställande. Å andra sidan, medan övervakade modeller som GNN och GCN har bättre prestanda än oövervakade modeller, kräver de en enorm ansträngning för att märka data och finjustera modellen. Numera har datamängderna blivit större och mer komplexa, vilket gör bördan tyngre för att tillämpa dessa övervakade modeller. Ett nyligen genomfört genombrott inom området Natural Language Processing kan lösa problemet. I tidningen ‘Attention is all you need’ introducerar författarna Transformer-modellen, som visar utmärkta prestanda i NLP. Med tanke på att området NLP har många saker gemensamt med GRL och att de första övervakade modellerna alla härstammar från NLP, är det rimligt att gissa om vi kan dra fördel av Transformatorn för att förbättra prestandan för den oövervakade modellen i GRL. Att generera inbäddning för noder i grafen är en av GRL:s viktiga uppgifter. I detta examensarbete testas transformatormodellens prestanda för att generera inbäddning. Tre populära datamängder (Cora, Citeseer, Pubmed) används i utbildningen, och inbäddningskvaliteten mäts genom nodklassificering med en linjär klassificeringsalgoritm. En annan del av avhandlingen är att finjustera modellen för att bestämma effekten av modellparametrar på inbäddningsnoggrannheten. I den här delen utförs jämförelseexperiment på dimensionerna, antalet lager, provstorleken och andra parametrar. Experimenten visar att Transformer-modellen presterar bättre när det gäller att generera inbäddning än de ursprungliga metoderna, såsom Deep-walk. Jämfört med övervakade metoder kräver det mindre finjustering och mindre träningstid. Den egenskap hos transformatormodellen som avslöjades från experimenten visar att den är ett bra alternativ till baslinjemodellen för inbäddningsgenerering. Förbättringar kan göras av modellens preposseing- och förlustfunktion för att få högre prestanda.

This thesis investigates the possibility of efficiently adapting self-supervised representation learning on visual domains beyond natural scenes, e.g., medical imagining and non-RGB sensory images. The thesis contributes to i) formalizing the self-supervised representation learning paradigm in a unified conceptual framework and ii) proposing the hypothesis based on supervision signal from data, called data-prior. Method adaptations following the hypothesis demonstrate significant progress in downstream tasks performance on microscopic histopathology and 3-dimensional particle management (3DPM) mining material non-RGB image domains. Supervised learning has proven to be obtaining higher performance than unsupervised learning on computer vision downstream tasks, e.g., image classification, object detection, etc. However, it imposes limitations due to human supervision. To reduce human supervision, end-to-end learning, i.e., transfer learning, remains proven for fine-tuning tasks but does not leverage unlabeled data. Representation learning in a self-supervised manner has successfully reduced the need for labelled data in the natural language processing and vision domain. Advances in learning effective visual representations without human supervision through a self-supervised learning approach are thought-provoking. This thesis performs a detailed conceptual analysis, method formalization, and literature study on the recent paradigm of self-supervised representation learning. The study’s primary goal is to identify the common methodological limitations across the various approaches for adaptation to the visual domain beyond natural scenes. The study finds a common component in transformations that generate distorted views for invariant representation learning. A significant outcome of the study suggests this component is closely dependent on human knowledge of the real world around the natural scene, which fits well the visual domain of the natural scenes but remains sub-optimal for other visual domains that are conceptually different. A hypothesis is proposed to use the supervision signal from data (data-prior) to replace the human-knowledge-driven transformations in self-supervised pretraining to overcome the stated challenge. Two separate visual domains beyond the natural scene are considered to explore the mentioned hypothesis, which is breast cancer microscopic histopathology and 3-dimensional particle management (3DPM) mining material non-RGB image. The first research paper explores the breast cancer microscopic histopathology images by actualizing the data-prior hypothesis in terms of multiple magnification factors as supervision signal from data, which is available in the microscopic histopathology images public dataset BreakHis. It proposes a self-supervised representation learning method, Magnification Prior Contrastive Similarity, which adapts the contrastive learning approach by replacing the standard image view transformations (augmentations) by utilizing magnification factors. The contributions to the work are multi-folded. It achieves significant performance improvement in the downstream task of malignancy classification during label efficiency and fully supervised settings. Pretrained models show efficient knowledge transfer on two additional public datasets supported by qualitative analysis on representation learning. The second research paper investigates the 3DPM mining material non-RGB image domain where the material’s pixel-mapped reflectance image and height (depth map) are captured. It actualizes the data-prior hypothesis by using depth maps of mining material on the conveyor belt. The proposed method, Depth Contrast, also adapts the contrastive learning method while replacing standard augmentations with depth maps for mining materials. It outperforms material classification over ImageNet transfer learning performance in fully supervised learning settings in fine-tuning and linear evaluation. It also shows consistent improvement in performance during label efficiency. In summary, the data-prior hypothesis shows one promising direction for optimal adaptations of contrastive learning methods in self-supervision for the visual domain beyond the natural scene. Although, a detailed study on the data-prior hypothesis is required to explore other non-contrastive approaches of recent self-supervised representation learning, including knowledge distillation and information maximization.

Intracranial Aneurysms are a prevalent vascular pathology present in 3-4% of the population with an inherent risk of rupture. The growing accessibility of angiography has led to a rising incidence of detected aneurysms. An accurate assessment of the rupture risk is of utmost importance for the very high disability and mortality rates in case of rupture and the non-negligible risk inherent to surgical treatment. However, human evaluation is rather subjective, and current treatment guidelines, such as the PHASES score, remain inefficient. Therefore we aimed to develop an automatic machine learning-based rupture prediction model. Our study utilized 686 CTA scans, comprising 844 intracranial aneurysms. Among these aneurysms, 579 were classified as ruptured, while 265 were categorized as non-ruptured. Notably, the CTAs of ruptured aneurysms were obtained within a week after rupture, during which negligible morphological changes were observed compared to the aneurysm’s pre-rupture shape, as established by previous research. Based on this observation, our rupture risk assessment focused on the models’ ability to classify between ruptured and unruptured IAs. In our investigation, we implemented an automated vessel and aneurysm segmentation, vessel labeling, and feature extraction framework. The rupture risk prediction involved the use of deep learning-based vessel and aneurysm shape features, along with a combination of demographic features (patient sex and age) and morphological features (aneurysm location, size, surface area, volume, sphericity, etc.). An ablation-type study was conducted to evaluate these features. Eight different machine learning models were trained with the objective of identifying ruptured aneurysms. The best performing model achieved an area under the receiver operating characteristic curve (AUC) of 0.833, utilizing a random forest algorithm with feature selection based on Spearman’s rank correlation thresholding, which effectively eliminated highly correlated and anti-correlated features...:1 Introduction 1.1 Intracranial aneurysms 1.1.1 Treatment strategy 1.1.2 Rupture risk assesment 1.2 Artificial Intelligence 1.3 Thesis structure 1.4 Contribution of the author 2 Theory 2.1 Rupture risk assessment guidelines 2.1.1 PHASES score 2.1.2 ELAPSS score 2.2 Literature review: Aneurysm rupture prediction 2.3 Machine learning classifiers 2.3.1 Decision Tree 2.3.2 Random Forests 2.3.3 XGBoost 2.3.4 K-Nearest-Neighbor 2.3.5 Multilayer Perceptron 2.3.6 Logistic Regression 2.3.7 Support Vector Machine 2.3.8 Naive Bayes 2.4 Latent feature vectors in deep learning 2.5 PointNet++ 3 Methodology 3.1 Data 3.2 Vessel segmentation 3.3 Feature extraction 3.3.1 Deep vessel features 3.3.2 Deep aneurysm features 3.3.3 Conventional features 3.4 Rupture classification 3.4.1 Univariate approach 3.4.2 Multivariate approach 3.4.3 Deep learning approach 3.4.4 Deep learning amplified multivariate approach 3.5 Feature selection 3.5.1 Correlation-based feature selection 3.5.2 Permutation feature importance 3.6 Implementation 3.7 Evaluation 4 Results 4.1 Univariate approach 4.2 Multivariate approach 4.3 Deep learning approach 4.3.1 Deep vessel features 4.3.2 Deep aneurysm features 4.3.3 Deep vessel and deep aneurysm features 4.4 Deep learning amplified multivariate approach 4.4.1 Conventional and deep vessel features 4.4.2 Conventional and deep aneurysm features 4.4.3 Conventional, deep vessel, and deep aneurysm features 5 Discussion and Conclusions 5.1 Overview of results 5.2 Feature selection 5.3 Feature analysis 5.3.1 Deep vessel features 5.3.2 Deep aneurysm features 5.3.3 Conventional features 5.3.4 Summary 5.4 Comparison to other methods 5.5 Outlook Bibliography / Intrakranielle Aneurysmen sind eine weit verbreitete vaskuläre Pathologie, die bei 3 bis 4% der Bevölkerung auftritt und ein inhärentes Rupturrisiko birgt. Mit der zunehmenden Verfügbarkeit von Angiographie wird eine steigende Anzahl von Aneurysmen entdeckt. Angesichts der sehr hohen permanenten Beeinträchtigungs- und Sterblichkeitsraten im Falle einer Ruptur und des nicht zu vernachlässigenden Risikos einer chirurgischen Behandlung ist eine genaue Bewertung des Rupturrisikos von größter Bedeutung. Die Beurteilung durch den Menschen ist jedoch sehr subjektiv, und die derzeitigen Behandlungsrichtlinien, wie der PHASES-Score, sind nach wie vor ineffizient. Daher wollten wir ein automatisches, auf maschinellem Lernen basierendes Modell zur Rupturvorhersage entwickeln. Für unsere Studie wurden 686 CTA-Scans von 844 intrakraniellen Aneurysmen verwendet, von denen 579 rupturiert waren und 265 nicht rupturiert waren. Dabei ist zu beachten, dass die CTAs der rupturierten Aneurysmen innerhalb einer Woche nach der Ruptur gewonnen wurden, in der im Vergleich zur Form des Aneurysmas vor der Ruptur nur geringfügige morphologische Veränderungen zu beobachten waren, wie in vorhergegangenen Studient festgestellt wurde. Im Rahmen unserer Untersuchung haben wir eine automatische Segmentierung von Adern und Aneurysmen, ein Aderlabeling und eine Merkmalsextraktion implementiert. Für die Vorhersage des Rupturrisikos wurden auf Deep Learning basierende Ader- und Aneurysmaformmerkmale zusammen mit einer Kombination aus demografischen Merkmalen (Geschlecht und Alter des Patienten) und morphologischen Merkmalen (u. A. Lage, Größe, Oberfläche, Volumen, Sphärizität des Aneurysmas) verwendet. Zur Bewertung dieser Merkmale wurde eine Ablationsstudie durchgeführt. Acht verschiedene maschinelle Lernmodelle wurden mit dem Ziel trainiert, rupturierte Aneurysmen zu erkennen...:1 Introduction 1.1 Intracranial aneurysms 1.1.1 Treatment strategy 1.1.2 Rupture risk assesment 1.2 Artificial Intelligence 1.3 Thesis structure 1.4 Contribution of the author 2 Theory 2.1 Rupture risk assessment guidelines 2.1.1 PHASES score 2.1.2 ELAPSS score 2.2 Literature review: Aneurysm rupture prediction 2.3 Machine learning classifiers 2.3.1 Decision Tree 2.3.2 Random Forests 2.3.3 XGBoost 2.3.4 K-Nearest-Neighbor 2.3.5 Multilayer Perceptron 2.3.6 Logistic Regression 2.3.7 Support Vector Machine 2.3.8 Naive Bayes 2.4 Latent feature vectors in deep learning 2.5 PointNet++ 3 Methodology 3.1 Data 3.2 Vessel segmentation 3.3 Feature extraction 3.3.1 Deep vessel features 3.3.2 Deep aneurysm features 3.3.3 Conventional features 3.4 Rupture classification 3.4.1 Univariate approach 3.4.2 Multivariate approach 3.4.3 Deep learning approach 3.4.4 Deep learning amplified multivariate approach 3.5 Feature selection 3.5.1 Correlation-based feature selection 3.5.2 Permutation feature importance 3.6 Implementation 3.7 Evaluation 4 Results 4.1 Univariate approach 4.2 Multivariate approach 4.3 Deep learning approach 4.3.1 Deep vessel features 4.3.2 Deep aneurysm features 4.3.3 Deep vessel and deep aneurysm features 4.4 Deep learning amplified multivariate approach 4.4.1 Conventional and deep vessel features 4.4.2 Conventional and deep aneurysm features 4.4.3 Conventional, deep vessel, and deep aneurysm features 5 Discussion and Conclusions 5.1 Overview of results 5.2 Feature selection 5.3 Feature analysis 5.3.1 Deep vessel features 5.3.2 Deep aneurysm features 5.3.3 Conventional features 5.3.4 Summary 5.4 Comparison to other methods 5.5 Outlook Bibliography