Deep Neural Networks have long been considered black box systems, where their interpretability is a concern when applied in safety critical systems. In this work, a novel approach of interpreting the decisions of DNNs is proposed. The approach depends on exploiting generative models and the interpretability of their latent space. Three methods for ranking features are explored, two of which depend on sensitivity analysis, and the third one depends on Random Forest model. The Random Forest model was the most successful to rank the features, given its accuracy and inherent interpretability.

In this study, the prediction model based on the Stacking principle is called the Stacking fusion model. Little evidence demonstrates that the Stacking fusion model possesses better prediction performance in the field of heart disease diagnosis than other classification models. Since this model belongs to the family of ensemble learning models, which has a bad interpretability, it should be used with caution in medical diagnoses. The purpose of this study is to verify whether the Stacking fusion model has better prediction performance than stand-alone machine learning models and other ensemble classifiers in the field of heart disease diagnosis, and to find ways to explain this model. This study uses experiment and quantitative analysis to evaluate the prediction performance of eight models in terms of prediction ability, algorithmic stability, false negative rate and run-time. It is proved that the Stacking fusion model with Naive Bayes classifier, XGBoost and Random forest as the first-level learners is superior to other classifiers in prediction ability. The false negative rate of this model is also outstanding. Furthermore, the Stacking fusion model is explained from the working principle of the model and the SHAP framework. The SHAP framework explains this model’s judgement of the important factors that influence heart disease and the relationship between the value of these factors and the probability of disease. Overall, two research problems in this study help reveal the prediction performance and reliability of the cardiac disease prediction model based on the Stacking principle. This study provides practical and theoretical support for hospitals to use the Stacking principle in the diagnosis of heart disease.

Abstract  Context. In machine learning, ensemble methods continue to receive increased attention. Since machine learning approaches that generate a single classifier or predictor have shown limited capabilities in some contexts, ensemble methods are used to yield better predictive performance. One of the most interesting and effective ensemble algorithms that have been introduced in recent years is Random Forests. A common approach to ensure that Random Forests can achieve a high predictive accuracy is to use a large number of trees. If the predictive accuracy is to be increased with a higher number of trees, this will result in a more complex model, which may be more difficult to interpret or analyse. In addition, the generation of an increased number of trees results in higher computational power and memory requirements.  Objectives. This thesis explores automatic simplification of Random Forest models via post-pruning as a means to reduce the size of the model and increase interpretability while retaining or increasing predictive accuracy. The aim of the thesis is twofold. First, it compares and empirically evaluates a set of state-of-the-art post-pruning techniques on the simplification task. Second, it investigates the trade-off between predictive accuracy and model interpretability.  Methods. The primary research method used to conduct this study and to address the research questions is experimentation. All post-pruning techniques are implemented in Python. The Random Forest models are trained, evaluated, and validated on five selected datasets with varying characteristics.  Results. There is no significant difference in predictive performance between the compared techniques and none of the studied post-pruning techniques outperforms the other on all included datasets. The experimental results also show that model interpretability is proportional to model accuracy, at least for the studied settings. That is, a positive change in model interpretability is accompanied by a negative change in model accuracy.  Conclusions. It is possible to reduce the size of a complex Random Forest model while retaining or improving the predictive accuracy. Moreover, the suitability of a particular post-pruning technique depends on the application area and the amount of training data available. Significantly simplified models may be less accurate than the original model but tend to be perceived as more comprehensible. / Sammanfattning  Kontext. Ensemble metoder fortsätter att få mer uppmärksamhet inom maskininlärning. Då maskininlärningstekniker som genererar en enskild klassificerare eller prediktor har visat tecken på begränsad kapacitet i vissa sammanhang, har ensemble metoder vuxit fram som alternativa metoder för att åstadkomma bättre prediktiva prestanda. En av de mest intressanta och effektiva ensemble algoritmerna som har introducerats under de senaste åren är Random Forests. För att säkerställa att Random Forests uppnår en hög prediktiv noggrannhet behöver oftast ett stort antal träd användas. Resultatet av att använda ett större antal träd för att öka den prediktiva noggrannheten är en komplex modell som kan vara svår att tolka eller analysera. Problemet med det stora antalet träd ställer dessutom högre krav på såväl lagringsutrymmet som datorkraften.  Syfte. Denna uppsats utforskar möjligheten att automatiskt förenkla modeller som är genererade av Random Forests i syfte att reducera storleken på modellen, öka dess tolkningsbarhet, samt bevara eller förbättra den prediktiva noggrannheten. Syftet med denna uppsats är tvåfaldigt. Vi kommer först att jämföra och empiriskt utvärdera olika beskärningstekniker. Den andra delen av uppsatsen undersöker sambandet mellan den prediktiva noggrannheten och modellens tolkningsbarhet.  Metod. Den primära forskningsmetoden som har använts för att genomföra den studien är experiment. Alla beskärningstekniker är implementerade i Python. För att träna, utvärdera, samt validera de olika modellerna, har fem olika datamängder använts.  Resultat. Det finns inte någon signifikant skillnad i det prediktiva prestanda mellan de jämförda teknikerna och ingen av de undersökta beskärningsteknikerna är överlägsen på alla plan. Resultat från experimenten har också visat att sambandet mellan tolkningsbarhet och noggrannhet är proportionellt, i alla fall för de studerade konfigurationerna. Det vill säga, en positiv förändring i modellens tolkningsbarhet åtföljs av en negativ förändring i modellens noggrannhet.  Slutsats. Det är möjligt att reducera storleken på en komplex Random Forests modell samt bibehålla eller förbättra den prediktiva noggrannheten. Dessutom beror valet av beskärningstekniken på användningsområdet och mängden träningsdata tillgänglig. Slutligen kan modeller som är signifikant förenklade vara mindre noggranna men å andra sidan tenderar de att uppfattas som mer förståeliga.

京都大学 / 0048 / 新制・課程博士 / 博士(農学) / 甲第21429号 / 農博第2307号 / 学位論文||H30||N5157(農学部図書室) / 京都大学大学院農学研究科地域環境科学専攻 / (主査)教授 近藤 直, 准教授 小川 雄一, 教授 飯田 訓久 / 学位規則第4条第1項該当 / Doctor of Agricultural Science / Kyoto University / DGAM

Background Machine learning techniques, such as convolutional networks, have shown promise in medical image analysis, including the detection of pediatric pneumonia. However, the interpretability of these models is often lacking, compromising their trustworthiness and acceptance in medical applications. The interpretability of machine learning models in medical applications is crucial for trust and bias identification. Aim The aim is to create a locally interpretable neural network that performs comparably to black-box models while being inherently interpretable, enhancing trust in medical machine learning models. Method An MLP ReLU network is trained with Guangzhou Women and Children's Medical Center pediatric chest x-ray image dataset and utilize Aletheia unwrapper for interpretability. A 5-fold cross-validation assesses the network's performance, measuring accuracy and F1 score. The average accuracy and F1 score are 0.90 and 0.91, respectively. To assessthe interpretability results are compared against a CNN network aided with LIME and SHAP to generate explanations. Results Despite lacking convolutional layers, the MLP network satisfactorily categorizes pneumonia images and explanations align with relevant areas of interest from previous studies. Moreover, by comparing it with a state of the art network aided with LIME and SHAP explanations, the local explanations demonstrate to be consistent within areas of the lungs while the post-hoc alternatives often highlighted areas not relevant for the specific task. Conclusion The developed locally interpretable neural network demonstrates promising performance and interpretability. However, additional research and implementation are required for it to outperform the so-called black box models. In a medical setting, a more accurate model despite the score could be crucial, as it could potentially save more lives, which is the ultimate goal of healthcare.

Anomaly detection plays a crucial role in various domains, such as transportation, cybersecurity, and industrial monitoring, where the timely identification of unusual patterns or outliers is of utmost importance. Traditional statistical techniques have limitations in handling complex and highdimensional data, which motivates the use of deep learning approaches. The project proposes designing and implementing deep neural networks, tailored explicitly for time series multivariate data from sensors incorporated in vehicles, to effectively capture intricate temporal dependencies and interactions among variables. As this project is conducted in collaboration with Scania, Sweden, the models are trained on datasets encompassing various vehicle sensor data. Different deep learning architectures, including Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs), are explored and compared to identify the most suitable model for anomaly detection tasks for the specified time series data and CNN found to perform well for the data used in the study. Furthermore, interpretability techniques are incorporated into the developed models to enhance their transparency and provide insights into the reasons behind detected anomalies. Interpretability is crucial in real-world applications to facilitate trust, understanding, and decision-making. Both model-agnostic and model-specific interpretability methods were employed to highlight the relevant features and contribute to the interpretability of the anomaly detection models. The performance of the proposed models is evaluated using test datasets with anomaly data, and comparisons are made against existing anomaly detection methods to demonstrate their effectiveness. Evaluation metrics such as precision, recall, false positive rate, F1 score, and composite F1 score are employed to assess the anomaly detection models' detection accuracy and robustness. For evaluating the interpretability method, Kolmogorov-Smirnov Test is used on counterfactual examples. The outcomes of this research project will contribute to developing advanced anomaly detection techniques that can effectively analyse time series multivariate data collected from sensors incorporated in vehicles. Incorporating interpretability techniques will provide valuable insights into the detected anomalies, enabling better decision-making and improved trust in the deployed models. These advancements can potentially enhance anomaly detection systems across various domains, leading to more reliable and secure operations.

Churn prediction methods are widely used in Customer Relationship Management and have proven to be valuable for retaining customers. To obtain a high predictive performance, recent studies rely on increasingly complex machine learning methods, such as ensemble or hybrid models. However, the more complex a model is, the more difficult it becomes to understand how decisions are actually made. Previous studies on machine learning interpretability have used a global perspective for understanding black-box models. This study explores the use of local explanation models for explaining the individual predictions of a Random Forest ensemble model. The churn prediction was studied on the users of Tink – a finance app. This thesis aims to take local explanations one step further by making comparisons between churn indicators of different user groups. Three sets of groups were created based on differences in three user features. The importance scores of all globally found churn indicators were then computed for each group with the help of local explanation models. The results showed that the groups did not have any significant differences regarding the globally most important churn indicators. Instead, differences were found for globally less important churn indicators, concerning the type of information that users stored in the app. In addition to comparing churn indicators between user groups, the result of this study was a well-performing Random Forest ensemble model with the ability of explaining the reason behind churn predictions for individual users. The model proved to be significantly better than a number of simpler models, with an average AUC of 0.93. / Metoder för att prediktera utträde är vanliga inom Customer Relationship Management och har visat sig vara värdefulla när det kommer till att behålla kunder. För att kunna prediktera utträde med så hög säkerhet som möjligt har den senasteforskningen fokuserat på alltmer komplexa maskininlärningsmodeller, såsom ensembler och hybridmodeller. En konsekvens av att ha alltmer komplexa modellerär dock att det blir svårare och svårare att förstå hur en viss modell har kommitfram till ett visst beslut. Tidigare studier inom maskininlärningsinterpretering har haft ett globalt perspektiv för att förklara svårförståeliga modeller. Denna studieutforskar lokala förklaringsmodeller för att förklara individuella beslut av en ensemblemodell känd som 'Random Forest'. Prediktionen av utträde studeras påanvändarna av Tink – en finansapp. Syftet med denna studie är att ta lokala förklaringsmodeller ett steg längre genomatt göra jämförelser av indikatorer för utträde mellan olika användargrupper. Totalt undersöktes tre par av grupper som påvisade skillnader i tre olika variabler. Sedan användes lokala förklaringsmodeller till att beräkna hur viktiga alla globaltfunna indikatorer för utträde var för respektive grupp. Resultaten visade att detinte fanns några signifikanta skillnader mellan grupperna gällande huvudindikatorerna för utträde. Istället visade resultaten skillnader i mindre viktiga indikatorer som hade att göra med den typ av information som lagras av användarna i appen. Förutom att undersöka skillnader i indikatorer för utträde resulterade dennastudie i en välfungerande modell för att prediktera utträde med förmågan attförklara individuella beslut. Random Forest-modellen visade sig vara signifikantbättre än ett antal enklare modeller, med ett AUC-värde på 0.93.

Atrial fibrillation (AF) is our society's most common cardiac arrhythmic disease, leading to increased morbidity and mortality. Predicting AF episodes during sinus rhythm based on electrocardiograms (ECGs) allows timely interventions. It is known, that changes in selected ECG morphology features are a predictor for the onset of AF, but no systematic investigation of different ECG features' temporal changes has been performed so far. We split sinus rhythm episodes of 60 minutes preceding AF from the MIT-BIH AF database into segments of 5 minutes with 50% overlap (n=644) and calculated 155 features of different domains per segment. Logistic regression analyses between the segments preceding AF and others revealed the most significant effects for segments ending 5 minutes before AF onset, with PQ interval slope (p < 0.01), PQ interval correlation (p < 0.05), and median RR time (p < 0.05) being the most relevant features. A decision tree ensemble, trained with all features, achieved an accuracy of 0.87 when distinguishing 8 segment clusters. Our results confirm expected changes in ECG features (e.g., PQ interval) before AF episodes, indicating impaired atrial excitation, and show that the combination of interpretable features is sufficient to discriminate at different points in time before AF onset. For advanced analyses, more extensive databases should be included.

Recently, there has been a lot of interest in explainable predictions, with new explainability approaches being created for specific data modalities like images and text. However, there is a dearth of understanding and minimal exploration in terms of explainability in the multimodal machine learning domain, where diverse data modalities are fused together in the model. In this thesis project, we look into two multimodal model architectures namely single-stream and dual-stream for the Visual Entailment (VE) task, which compromises of image and text modalities. The models considered in this project are UNiversal Image-TExt Representation Learning (UNITER), Visual-Linguistic BERT (VLBERT), Vision-and-Language BERT (ViLBERT) and Learning Cross-Modality Encoder Representations from Transformers (LXMERT). Furthermore, we conduct three different experiments for multimodal explainability by applying the Local Interpretable Model-agnostic Explanations (LIME) technique. Our results show that UNITER has the best accuracy among these models for the problem of VE. However, the explainability of all these models is similar. / Under den senaste tiden har intresset för förklarbara prediktioner (eng. explainable predictions) varit stort, med nya metoder skapade för specifika datamodaliteter som bilder och text. Samtidigt finns en brist på förståelse och lite utforskning har gjorts när det gäller förklarbarhet för multimodal maskininlärning, där olika datamodaliteter kombineras i modellen. I detta examensarbete undersöker vi två multimodala modellarkitekturer, så kallade en-ström och två-strömsarkitekturer (eng. single-steam och dual-stream) för en uppgift som kombinerar bilder och text, Visual Entailment (VE). Modellerna som studeras är UNiversal Image-TExt Representation Learning (UNITER), Visual-Linguistic BERT (VLBERT), Vision-and-Language BERT (ViLBERT) och Learning Cross-Modality Encoder Representations from Transformers (LXMERT). Dessutom genomför vi tre olika experiment för multimodal förklarbarhet genom att tillämpa en metod som heter Local Interpretable Model-agnostic Explanations (LIME). Våra resultat visar att UNITER har bäst prestanda av dessa modeller för VE-uppgiften. Å andra sidan är förklarbarheten för alla dessa modeller likvärdig.

Recent advances in technology have led to an explosion in data collection over time rather than in a single snapshot. For example, microarray technology allows us to measure gene expression levels in different conditions over time. Such temporal data grants the opportunity for data miners to develop algorithms to address domain-related problems, e.g. a time series of several different classes can be created, by observing various patient attributes over time and the task is to classify unseen patient based on his temporal observations. In time-sensitive applications such as medical applications, some certain aspects have to be considered besides providing accurate classification. The first aspect is providing early classification. Accurate and timely diagnosis is essential for allowing physicians to design appropriate therapeutic strategies at early stages of diseases, when therapies are usually the most effective and the least costly. We propose a probabilistic hybrid method that allows for early, accurate, and patient-specific classification of multivariate time series that, by training on a full time series, offer classification at a very early time point during the diagnosis phase, while staying competitive in terms of accuracy with other models that use full time series both in training and testing. The method has attained very promising results and outperformed the baseline models on a dataset of response to drug therapy in Multiple Sclerosis patients and on a sepsis therapy dataset. Although attaining accurate classification is the primary goal of data mining task, in medical applications it is important to attain decisions that are not only accurate and obtained early, but can also be easily interpreted which is the second aspect of medical applications. Physicians tend to prefer interpretable methods rather than black-box methods. For that purpose, we propose interpretable methods for early classification by extracting interpretable patterns from the raw time series to help physicians in providing early diagnosis and to gain insights into and be convinced about the classification results. The proposed methods have been shown to be more accurate and provided classifications earlier than three alternative state-of-the-art methods when evaluated on human viral infection datasets and a larger myocardial infarction dataset. The third aspect has to be considered for medical applications is the need for predictions to be accompanied by a measure which allows physicians to judge about the uncertainty or belief in the prediction. Knowing the uncertainty associated with a given prediction is especially important in clinical diagnosis where data mining methods assist clinical experts in making decisions and optimizing therapy. We propose an effective method to provide uncertainty estimate for the proposed interpretable early classification methods. The method was evaluated on four challenging medical applications by characterizing decrease in uncertainty of prediction. We showed that our proposed method meets the requirements of uncertainty estimates (the proposed uncertainty measure takes values in the range [0,1] and propagates over time). To the best of our knowledge, this PhD thesis will have a great impact on the link between data mining community and medical domain experts and would give physicians sufficient confidence to put the proposed methods into real practice. / Computer and Information Science