The dependence on Natural Language Processing (NLP) systems has grown significantly in the last decade. Recent advances in deep learning have enabled language models to generate high-quality text at the same level as human-written text. If this growth continues, it can potentially lead to increased misinformation, which is a significant challenge. Although claim verification techniques exist, they lack proper explainability. Numerical scores such as Attention and Lime and visualization techniques such as saliency heat maps are insufficient because they require specialized knowledge. It is inaccessible and challenging for the nonexpert to understand black-box NLP systems. We propose a novel approach called, ExClaim for explainable claim verification using NLP rationalization. We demonstrate that our approach can predict a verdict for the claim but also justify and rationalize its output as a natural language explanation (NLE). We extensively evaluate the system using statistical and Explainable AI (XAI) metrics to ensure the outcomes are valid, verified, and trustworthy to help reinforce the human-AI trust. We propose a new subfield in XAI called Rational AI (RAI) to improve research progress on rationalization and NLE-based explainability techniques. Ensuring that claim verification systems are assured and explainable is a step towards trustworthy AI systems and ultimately helps mitigate misinformation. / Master of Science / The dependence on Natural Language Processing (NLP) systems has grown significantly in the last decade. Recent advances in deep learning have enabled text generation models to generate high-quality text that is at the same level as human-written text. If this growth continues, it can potentially lead to increased misinformation, which is a major societal challenge. Although claim verification techniques exist, they lack proper explainability. It is difficult for the average user to understand the model's decision-making process. Numerical scores and visualization techniques exist to provide explainability, but they are insufficient because they require specialized domain knowledge. This makes it inaccessible and challenging for the nonexpert to understand black-box NLP systems. We propose a novel approach called, ExClaim for explainable claim verification using NLP rationalization. We demonstrate that our approach can predict a verdict for the claim but also justify and rationalize its output as a natural language explanation (NLE). We extensively evaluate the system using statistical and Explainable AI (XAI) metrics to ensure the outcomes are valid, verified, and trustworthy to help reinforce the human-AI trust. We propose a new subfield in XAI called Rational AI (RAI) to improve research progress on rationalization and NLE-based explainability techniques. Ensuring that claim verification systems are assured and explainable is a step towards trustworthy AI systems and ultimately helps mitigate misinformation.

Autonomous robots have proven to be useful for search and rescue (SAR) by being deployed in emergency situations and removing the need for direct human presence. However, there remains a need for effective communication between the robot and human operator to cooperate as a team. This thesis investigates the question of when to explain an autonomous agent’s behaviour in the setting of human-robot teaming. A game environment is developed to conduct a virtual SAR experiment to test the effect of explanation timings between two conditions: (1) always explaining - the robot provides explanations whenever possible, and (2) anticipatory explaining - the robot determines when to explain based on the context. When to explain is determined through the construction of argumentation frameworks modelling the context. The effect of anticipatory information pushing is tested on two metrics: team performance and explanation experience. Results indicate anticipatory explaining does not have a significant effect on team performance and participants’ explanation satisfaction. Additionally, participant feedback shows they prefer to be in control instead of cooperating as a team. These findings underline the importance of studying explanation presentation in high-demanding environments and indicate a need for interdisciplinary discussion on the design of human-robot teaming.

With the combination of Reinforcement Learning (RL) and Artificial Neural Networks (ANNs), Deep Reinforcement Learning (DRL) agents are shifted towards being non-interpretable black-box models. Developers of DRL agents, however, could benefit from enhanced interpretability of the agents’ behavior, especially during the training process. Improved interpretability could enable developers to make informed adaptations, leading to better overall performance. The explainability methods Partial Dependence Plot (PDP), Accumulated Local Effects (ALE) and SHapley Additive exPlanations (SHAP) were considered to provide insights into how an agent’s behavior evolves during training. Additionally, a decision tree as a surrogate model was considered to enhance the interpretability of a trained agent. In a case study, the methods were tested on a Deep Deterministic Policy Gradient (DDPG) agent that was trained in an Obstacle Avoidance (OA) scenario. PDP, ALE and SHAP were evaluated towards their ability to provide explanations as well as the feasibility of their application in terms of computational overhead. The decision tree was evaluated towards its ability to approximate the agent’s policy as a post-hoc method. Results demonstrated that PDP, ALE and SHAP were able to provide valuable explanations during the training. Each method contributed additional information with their individual advantages. However, the decision tree failed to approximate the agent’s actions effectively to be used as a surrogate model.:List of Figures List of Tables List of Abbreviations 1 Introduction 2 Foundations 2.1 Machine Learning 2.1.1 Deep Learning 2.2 Reinforcement Learning 2.2.1 Markov Decision Process 2.2.2 Limitations of Optimal Solutions 2.2.3 Deep Reinforcement Learning 2.3 Explainability 2.3.1 Obstacles for Explainability Methods 3 Applied Explainability Methods 3.1 Real-Time Methods 3.1.1 Partial Dependence Plot 3.1.1.1 Incremental Partial Dependence Plots for Dynamic Modeling Scenarios 3.1.1.2 PDP-based Feature Importance 3.1.2 Accumulated Local Effects 3.1.3 SHapley Additive exPlanations 3.2 Post-Hoc Method: Global Surrogate Model 4 Case Study: Obstacle Avoidance 4.1 Environment Representation 4.2 Agent 4.3 Application Settings 5 Results 5.1 Problems of the Incremental Partial Dependence Plot 5.2 Real-Time Methods 5.2.1 Feature Importance 5.2.2 Computational Overhead 5.3 Global Surrogate Model 6 Discussion 7 Conclusion Bibliography Appendix A Incremental Partial Dependence Results

[pt] Investigamos dois métodos de clusterização com restrições nas partições geradas: a clusterização explicável, em que a partição deve ser induzida por uma árvore de decisão binária (ou seja, por cortes paralelos aos eixos); e a clusterização de tamanho mínimo, na qual todos os clusters devem ter pelo menos um número predeterminado de elementos. Para a clusterização explicável, apresentamos algoritmos e garantias teóricas para as funções de custo k-centers, k-medians, k-means e espaçamento mínimo. Introduzimos também três algoritmos práticos para a popular função de custo k-means: ExGreedy, com resultados geralmente melhores do que os de algoritmos comparáveis na literatura; ExShallow, com um termo de penalidade relacionado à profundidade da árvore que induz a partição, permitindo um equilíbrio entre desempenho (redução da função de custo) e explicabilidade (geração de árvores mais rasas); e ExBisection, que, até onde sabemos, é o primeiro algoritmo de clusterização explicável baseado em árvores de decisão para a função de custo k-means que constrói uma partição explicável do zero (ou seja, sem usar uma partição irrestrita como ponto de partida). Para a clusterização de tamanho mínimo, focamos em medidas interclusterização. Mostramos que Single-Linkage, o algoritmo que maximiza o espaçamento mínimo, também maximiza o custo da árvore de geração mínima de um grafo induzido pela partição gerada por ele; no entanto, este algoritmo tende a gerar muitos clusters pequenos, o que motiva a busca por algoritmos com bons resultados para essas funções de custo que garantam um número mínimo de elementos por cluster. Introduzimos um algoritmo de aproximação para cada função de custo e apresentamos os resultados de experimentos que mostram que eles produzem partições com melhores resultados do que o popular algoritmo k-means para essas instâncias do problema de clusterização. / [en] We investigate two methods of clustering with constraints on the partitions being generated: explainable clustering, in which the partition must be induced by a binary decision tree (i.e., by cuts that are parallel to the axes); and minimum-size clustering, in which all clusters must have at least a predetermined number of elements. For explainable clustering, we present theoretical algorithms and bounds for the k-centers, k-medians, k-means, and minimum-spacing cost functions. We also introduce three practical algorithms for the popular k-means cost function: ExGreedy, which presents results generally better than comparable algorithms in the literature; ExShallow, with a penalty term related to the depth of the tree that induces the partition, allowing for a trade-off between performance (reducing the cost function) and explainability (generating shallower trees); and ExBisection, to our knowledge the first explainable clustering algorithm based on decision trees for the k-means cost function that builds an explainable partition from scratch (i.e., without starting from an unrestricted partition). For minimum-size clustering, our focus is on inter-clustering measures. We show that Single-Linkage, the algorithm that maximizes the minimum spacing, also maximizes the minimum-spanning-tree cost of a graph induced by the partition it generates; however, it is also prone to generating small clusters, which motivates the search for algorithms that perform well for these cost functions without suffering from this tendency. We introduce one approximation algorithm for each cost function, and present the results of experiments showing that they produce partitions that perform better than the popular k-means algorithm for these instances of the clustering task.

<div>Many real-world domains are relational, consisting of objects (e.g., users and pa- pers) linked to each other in various ways. Because class labels in graphs are often only available for a subset of the nodes, semi-supervised learning for graphs has been studied extensively to predict the unobserved class labels. For example, we can pre- dict political views in a partially labeled social graph dataset and get expected gross incomes of movies in an actor/movie graph with a few labels. Recently, advances in representation learning for graph data have made great strides for the semi-supervised node classification. However, most of the methods have mainly focused on learning node representations by considering simple relational properties (e.g., random walk) or aggregating nearby attributes, and it is still challenging to learn complex inter- action patterns in partially labeled graphs and provide explanations on the learned representations. </div><div><br></div><div>In this dissertation, multiple methods are proposed to alleviate both challenges for semi-supervised node classification. First, we propose a graph neural network architecture, REGNN, that leverages local inferences for unlabeled nodes. REGNN performs graph convolution to enable label propagation via high-order paths and predicts class labels for unlabeled nodes. In particular, our proposed attention layer of REGNN measures the role equivalence among nodes and effectively reduces the noise, which is generated during the aggregation of observed labels from distant neighbors at various distances. Second, we also propose a neural network archi- tecture that jointly captures both temporal and static interaction patterns, which we call Temporal-Static-Graph-Net (TSGNet). The architecture learns a latent rep- resentation of each node in order to encode complex interaction patterns. Our key insight is that leveraging both a static neighbor encoder, that learns aggregate neigh- bor patterns, and a graph neural network-based recurrent unit, that captures complex interaction patterns, improves the performance of node classification. Lastly, in spite of better performance of representation learning on node classification tasks, neural network-based representation learning models are still less interpretable than the pre- vious relational learning models due to the lack of explanation methods. To address the problem, we show that nodes with high bridgeness scores have larger impacts on node embeddings such as DeepWalk, LINE, Struc2Vec, and PTE under perturbation. However, it is computationally heavy to get bridgeness scores, and we propose a novel gradient-based explanation method, GRAPH-wGD, to find nodes with high bridgeness efficiently. In our evaluations, our proposed architectures (REGNN and TSGNet) for semi-supervised node classification consistently improve predictive performance on real-world datasets. Our GRAPH-wGD also identifies important nodes as global explanations, which significantly change both predicted probabilities on node classification tasks and k-nearest neighbors in the embedding space after perturbing the highly ranked nodes and re-learning low-dimensional node representations for DeepWalk and LINE embedding methods.</div>

Brain tumors present the second highest cause of death among pediatric cancers. About 60% are located in the posterior fossa region of the brain; among the most frequent types the ones considered for this project were astrocytomas, medulloblastomas, and ependymomas. Diagnosis can be done either through invasive histopathology exams or by non-invasive magnetic resonance (MR) scans. The tumors listed can be difficult to diagnose, even for trained radiologists, so machine learning methods, in particular deep learning, can be useful in helping to assess a diagnosis. Deep learning has been investigated only in a few other studies.The dataset used included 115 different subjects, some with multiple scan sessions, for which there were 142 T2-w, 119 T1Gd-w, and 89 volumes that presented both MR modalities. 2D slices have been manually extracted from the registered and skull-stripped volumes in the transversal, sagittal, and frontal anatomical plane and have been preprocessed by normalizing them and selecting the slices containing the tumor. The scans employed are T2-w, T1Gd-w, and a combination of the two referred to as multimodal images. The images were divided session-wise into training, validation, and testing, using stratified cross-validation and have also been augmented. The convolutional neural networks (CNN) investigated were ResNet50, VGG16, and MobileNetV2. The model performances were evaluated for two-class and three-class classification tasks by computing the confusion matrix, accuracy, receiver operating characteristic curve (ROC), the area under the curve (AUROC), and F1-score. Moreover,  explanations for the behavior of networks were investigated using GradCAMs and occlusion maps. Preliminary investigations showed that the best plane and modality were the transversal one and T2-w images. Overall the best model was VGG16, for the two-class tasks the best classification was between astrocytomas and medulloblastomas which reached an F1-score of 0.86 for both classes on multimodal images, followed by astrocytomas and ependymomas with an F1-score of 0.76 for astrocytomas and 0.74 for ependymomas on T2-w, and last F1-score of 0.30 for ependymomas and 0.65 for medulloblastomas on multimodal images. The three-class classification reached F1-score values of 0.59 for astrocytomas, 0.46 for ependymomas, and 0.64 for medulloblastomas on T2-w images. GradCAMs and occlusion maps showed that VGG16 was able to focus mostly on the tumor region but that there also seemed to be other information in the background of the images that contributed to the final classification.To conclude, the classification of infratentorial pediatric brain tumors can be achieved with acceptable results by means of deep learning and using a single MR modality, though one might have to account for the dataset size, number of classes and class imbalance. GradCAMs and occlusion maps offer important insights into the decision process of the networks

I en framtid där förutsägelser och beslut som tas av maskininlärningssystem överträffar människors förmåga behöver systemen att vara tolkbara för att vi skall kunna lita på och förstå dem. Vår studie utforskar världen av tolkbar maskininlärning genom att designa och undersöka artefakter. Vi genomför experiment för att utforska förklarbarhet, tolkbarhet samt tekniska utmaningar att skapa maskininlärningsmodeller för att identifiera liknande men unika objekt. Slutligen genomför vi ett användartest för att utvärdera toppmoderna förklaringsverktyg i ett direkt mänskligt sammanhang. Med insikter från dessa experiment diskuterar vi den potentiella framtiden för detta fält / In a future where predictions and decisions made by machine learning systems outperform humans we need the systems to be interpretable in order for us to trust and understand them. Our study explore the realm of interpretable machine learning through designing artifacts. We conduct experiments to explore explainability, interpretability as well as technical challenges of creating machine learning models to identify objects that appear similar to humans. Lastly, we conduct a user test to evaluate current state-of-the-art visual explanatory tools in a human setting. From these insights, we discuss the potential future of this field.

This thesis presents a novel approach to make Artificial Intelligence (AI) more explainable by using a Belief Rule Based Expert System (BRBES). A BRBES is a type of expert system that can handle both qualitative and quantitative information under uncertainty and incompleteness by using if-then rules with belief degrees. The BRBES can model the human inference process and provide transparent and interpretable reasoning for its decisions. However, designing a BRBES requires tuning several parameters, such as the rule weights, the belief degrees, and the inference parameters. To address this challenge, this thesis report proposes to use a Differential Evolution (DE) algorithm to optimize these parameters automatically. A DE algorithm such as BRB adaptive DE (BRBaDE) and Joint Optimization of BRB is a metaheuristic that optimizes a problem by iteratively creating new candidate solutions by combining existing ones according to some simple formulae. The DE algorithm does not require any prior knowledge of the problem or its gradient, and can handle complex optimization problems with multiple objectives and constraints. This model can provide explainability by using different model agnostic method including Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP). The proposed approach is applied to calculate Air Quality Index (AQI) using particle data. The results show that the proposed approach can improve the performance and explainability of AI systems compared to other existing methods. Moreover, the proposed model can ensure the balance between accuracy and explainablity in comparison to other models.

Explainability is a key component in building trust for computer calculated predictions when they are applied to areas with influence over individual people. This bachelor thesis project report focuses on the explanation regarding the decision making process of the machine learning method Logistic Regression when predicting mortality. The aim is to present theoretical information about the predictive model as well as an explainable interpretation when applied on the clinical MIMIC-III database. The project found that there was a significant difference between particular features considering the impact of each individual feature on the classification. The feature that showed the greatest impact was the Glasgow Coma Scale value, which could be proven through the fact that a good classifier could be constructed with only that and one other feature. An important conclusion from this study is that a great focus should be enforced early in the implementation process when the features are selected. In this specific case, when medical artificial intelligence is implemented, medical expertise is desired in order to make a good feature selection. / Förklarbarhet är en viktig komponent för att skapa förtroende för datorframtagna prognoser när de appliceras på områden som påverkar individuella personer. Denna kandidatexamensarbetesrapport fokuserar på förklarandet av beslutsprocessen hos maskininlärningsmetoden Logistic Regression när dödlighet ska förutsägas. Målet är att presentera information om den förutsägande modellen samt en förklarbar tolkning av resultaten när modellen appliceras på den kliniska databasen MIMIC-III. Projektet fann att det fanns signifikanta skillnader mellan särskilda egenskaper med hänsyn till den påverkan varje enskild egenskap har på klassificeringen. Den egenskapen som visade ha störst inverkan var Glascow Coma Scale värdet, vilket kunde visas via det faktum att en god klassificerare kunde konstrueras med endast den och en annan egenskap. En viktig slutsats av denna studie är att stort fokus bör läggas tidigt i implementationsprocessen då egenskaperna väljs. I detta specifika fall, då medicinsk artificiell intelligens implementeras, krävs medicinsk expertis för att göra ett gott egenskapsurval. / Kandidatexjobb i elektroteknik 2022, KTH, Stockholm

Artificial Intelligence (AI) has proven to be highly suitable for a wide range of problems in manufacturing environments, including the prediction of lead times. Most of these solutions are based on ”black-box” algorithms, which hinder practitioners to understand the prediction process. Explainable Artificial Intelligence (XAI) provides numerous tools and methods to counteract this problem. There is however a need to qualify the methods with human-centered studies in manufacturing environments, since explainabilityis context-specific. The purpose of this mixed-method case study is to examine the explainability of regression models for lead time prediction in quality control laboratories at a biopharmaceutical production site in Sweden. This entails the research questions of which methods can increase the explainability of lead time prediction, what type of explanation is required to enable explainability and what are the benefits of explaining regression models in this context. This is why relevant literature in the field of XAI and AI-based lead time prediction is reviewed. An explainable lead time prediction modelis developed and a Delphi study is carried out to gauge the importance of different explanation types and to identify explainability-related benefits. The results show a transparency-performance trade-off and highlight eight benefits that are mapped to the model’s life cycle. These findings provide new insights into the explainability requirements and benefits in quality control processes and support practitioners in steering their implementation efforts. / Artificiell Intelligens (AI) har visat sig vara mycket lämplig för ett stort antal problem i tillverkningsmiljöer, bland annat när det gäller att förutsäga ledtider. De flesta av dessa lösningar är baserade på algoritmer som är ”svarta lådor”, vilket gör det svårt för tillämparna att förstå förutsägelseprocessen. Explainable Artificial Intelligence (XAI) erbjuder många verktyg och metoder för att motverka detta problem. Det finns dock ett behov av att kvalificera metoderna med människocentrerade studier i tillverkningsmiljöer, eftersom förklarbarhet är kontextspecifikt. Syftet med denna fallstudie med blandad metod är att undersöka förklaringsbarheten hos regressionsmodeller för prediktion av ledtider i kvalitets kontrolllaboratorier vid en biopharmaceutisk produktionsanläggning i Sverige. Vilket syftar till forskningsfrågorna samt vilka metoder som kan öka förklaringsbarheten och av prognoser för ledtider, vilken typ av förklaring som krävs för att möjliggöra en förklarbarhet och vilka fördelar som finns med att förklara regressionsmodeller i detta sammanhang. Det är därför som relevant litteratur på området XAI och AI baserade prognostisering av ledtider granskas. En förklaringsbar modell för prognostisering av ledtider utvecklas och en Delphi-studie genomförs för att bedöma betydelsen av olika typer av förklaringar och för att identifiera förklaringsrelaterade fördelar.