Trustworthiness is a roadblock in mass adoption of artificial intelligence (AI) in medicine. This thesis developed a framework to explore the trustworthiness as it applies to AI in medicine with respect to common stakeholders in medical device development. Within this framework the element of explainability of AI models was explored by evaluating explainable AI (XAI) methods. In current literature a litany of XAI methods are available that provide a variety of insights into the learning and function of AI models. XAI methods provide a human readable output for the AI’s learning process. These XAI methods tend to be bespoke and provide very subjective outputs with varying degrees of quality. Currently, there are no metrics or methods of objectively evaluating XAI outputs against outputs from different types of XAI methods. This thesis presents a set of constituent elements (similarity, stability and novelty) to explore the concept of explainability and then presents a series of metrics to evaluate those constituent elements. Thus providing a repeatable and testable framework to evaluate XAI methods and their generated explanations. This is accomplished using subject matter expert (SME) annotated ECG signals (time-series signals) represented as images to AI models and XAI methods. A small subset from all available XAI methods, Vanilla Saliency, SmoothGrad, GradCAM and GradCAM++ were used to generate XAI outputs for a VGG-16 based deep learning classification model. The framework provides insights about XAI method generated explanations for the AI and how closely that learning corresponds to SME decision making. It also objectively evaluates how closely explanations generated by any XAI method resemble outputs from other XAI methods. Lastly, the framework provides insights about possible novel learning done by the deep learning model beyond what was identified by the SMEs in their decision making. / Thesis / Master of Applied Science (MASc) / The goal of this thesis was to develop a framework of how trustworthiness can be improved for a variety of stakeholders in the use of AI in medical applications. Trust was broken down into basic elements (Explainability, Verifiability, Fairness & Ro- bustness) and ’Explainability’ was further explored. This was done by determining how explainability (offered by XAI methods) can address the needs (Accuracy, Safety, and Performance) of stakeholders and how those needs can be evaluated. Methods of comparison (similarity, stability, and novelty) were developed that allow an objective evaluation of the explanations from various XAI methods using repeatable metrics (Jaccard, Hamming, Pearson Correlation, and TF-IDF). Combining the results of these measurements into the framework of trust, work towards improving AI trust- worthiness and provides a way to evaluate and compare the utility of explanations.

A 2023 cybersecurity research study highlighted the risk of increased technology investment not being matched by a proportional investment in cybersecurity, exposing organizations to greater cyber identity compromise vulnerabilities and risk. The result is that a survey of security professionals found that 240\% expected growth in digital identities, 68\% were concerned about insider threats from employee layoffs and churn, 99\% expect identity compromise due to financial cutbacks, geopolitical factors, cloud adoption and hybrid work, while 74\% were concerned about confidential data loss through employees, ex-employees and third party vendors. In the light of continuing growth of this type of criminal activity, those responsible for keeping such risks under control have no alternative than to use continually more defensive measures to prevent them from happening and causing unnecessary businesses losses. This research project explores a real-life case study: an Artificial Intelligence (AI) information systems solution implemented in a mid-size organization facing significant cybersecurity threats. A holistic approach was taken, where AI was complemented with key non-technical elements such as organizational structures, business processes, standard operating documentation and training - oriented towards driving behaviours conducive to a strong cybersecurity posture for the organization. Using Design Science Research (DSR) guidelines, the process for conceptualizing, designing, planning and implementing the AI project was richly described from both a technical and information systems perspective. In alignment with DSR, key artifacts are documented in this research, such as a model for AI implementation that can create significant value for practitioners. The research results illustrate how an iterative, data-driven approach to development and operations is essential, with explainability and interpretability taking centre stage in driving adoption and trust. This case study highlighted how critical communication, training and cost-containment strategies can be to the success of an AI project in a mid-size organization. / Thesis / Doctor of Science (PhD) / Artificial Intelligence (AI) is now pervasive in our lives, intertwined with myriad other technology elements in the fabric of society and organizations. Instant translations, complex fraud detection and AI assistants are not the fodder of science fiction any longer. However, realizing its bene fits in an organization can be challenging. Current AI implementations are different from traditional information systems development. AI models need to be trained with large amounts of data, iteratively focusing on outcomes rather than business requirements. AI projects may require an atypical set of skills and significant financial resources, while creating risks such as bias, security, interpretability, and privacy. The research explores a real-life case study in a mid-size organization using Generative AI to improve its cybersecurity posture. A model for successful AI implementations is proposed, including the non-technical elements that practitioners should consider when pursuing AI in their organizations.

The integration of Artificial Intelligence (AI) in collaborative tasks has gained momentum, with particular implications for critical infrastructure maintenance. This study examines the assurance goals of AI—security, explainability, and trustworthiness—within Virtual Reality (VR) environments for bridge maintenance. Adopting a within-subjects design approach, this research leverages VR environments to simulate real-world bridge maintenance scenarios and gauge user interactions with AI tools. With the industry transitioning from paper-based to digital bridge maintenance, this investigation underscores the imperative roles of security and trust in adopting AI-assisted methodologies. Recent advancements in AI assurance within critical infrastructure highlight its monumental role in ensuring safe, explainable, and trustworthy AI-driven solutions. / Master of Science / In today's rapidly advancing world, the traditional methods of inspecting and maintaining our bridges are being revolutionized by digital technology and artificial intelligence (AI). This study delves into the emerging role of AI in bridge maintenance, a field historically reliant on manual inspection. With the implementation of AI, we aim to enhance the efficiency and accuracy of assessments, ensuring that our bridges remain safe and functional. Our research employs virtual reality (VR) to create a realistic setting for examining how users interact with AI during bridge inspections. This immersive approach allows us to observe the decision-making process in a controlled environment that closely mimics real-life scenarios. By doing so, we can understand the potential benefits and challenges of incorporating AI into maintenance routines. One of the critical challenges we face is the balance of trust in AI. Too little trust could undermine the effectiveness of AI assistance, while too much could lead to overreliance and potential biases. Furthermore, the use of digital systems introduces the risk of cyber threats, which could compromise the security and reliability of the inspection data. Our research also investigates the impact of AI-generated explanations on users' decisions. In essence, we explore whether providing rationale behind AI's recommendations helps users make better judgments during inspections. The ultimate objective is to develop AI tools that are not only advanced but also understandable and reliable for those who use them, even if they do not have a deep background in technology. As we integrate AI into bridge inspections, it's vital to ensure that such systems are protected against cyber threats and that they function as reliable companions to human inspectors. This study seeks to pave the way for AI to become a trusted ally in maintaining the safety and integrity of our infrastructure.

No description available.

Deep Neural Networks (DNNs) are known as black box algorithmsthat lack transparency and interpretability for humans. eXplainableArtificial Intelligence (XAI) is introduced to tackle this problem. MostXAI methods are utilized post-training, providing explanations of themodel to clarify its predictions and inner workings for human understanding. However, there is a shortage of methods that utilize XAIduring training to not only observe the model’s behavior but alsoexploit this information for the benefit of the model.In our approach, we propose a novel method that leverages XAIduring the training process itself. Incorporating feedback from XAIcan give us insights into important features of input data that impact model decisions. This work explores focusing more on specificfeatures during training, which could potentially improve model performance introspectively throughout the training phase. We analyzethe stability of feature explanations during training and find thatthe model’s attention to specific features is consistent in the MNISTdataset. However, unimportant features lack stability. The OCTMNIST dataset, on the other hand, has stable explanations for important features but less consistent explanations for less significant features. Based on this observation, two types of masks, namely fixedand dynamic, are applied to the model’s structure using XAI’s feedback with minimal human intervention. These masks identify themore important features from the less important ones and set the pixels associated with less significant features to zero. The fixed mask isgenerated based on XAI feedback after the model is fully trained, andthen it is applied to the output of the first convolutional layer of a newmodel (with the same architecture), which is trained from scratch. Onthe other hand, the dynamic mask is generated based on XAI feedback during training, and it is applied to the model while the modelis still training. As a result, these masks are changing during different epochs. Examining these two methods on both deep and shallowmodels, we find that both masking methods, particularly the fixedone, reduce the focus of all models on the least important parts of theinput data. This results in improved accuracy and loss in all models.As a result, this approach enhances the model’s interpretability andperformance by incorporating XAI into the training process.

To produce paperboard is a complex process which requires sophisticated monitoring to achieve a paperboard of high quality. Holmen Iggesund is a company in the paperboard manufacturing industry, aiming to produce paperboard of world leading quality. Therefore, they continuously develop their knowledge the production process. In this study, conducted at Holmen Iggesund, the focus is the property of delamination, which is tested with a method called Scott bond. Seven different input signals, measured over a two-year period, were used as input to six different models and used to predict the output (Scott bond). The result showed that a Random Forest model provided the best prediction performance among the tested models. EXplainable Artificial Intelligence (XAI) was then used to better understand the predictions of the Random forest model. It provided an understanding of which input signals were most significant for the model predictions and the values that the input signals should have to predict a high or low value of the output signal. The results from the work give an increased understanding of the process behavior which may help to improve the monitoring of the process and how to counter interact when a process disturbance occurs. It also shows the potential of using complex machine learning models combined with XAI algorithms.

The advancement in the field of artificial intelligence has been rapid in recent years and has revolutionized various industries. For example, convolutional neural networks (CNNs) perform image classification at a level equivalent to that of humans on many image datasets. These state-of-the-art networks reached unprecedented success using complex architectures with billions of parameters, numerous kernel configurations, weight initialization and regularization methods. This transitioned the models into black-box entities with little to no information on the decision-making process. This lack of transparency in decision making and started raising concerns amongst some sectors of user community such as the sectors, amongst others healthcare, finance and justice. This challenge motivated our research where we successfully produced human interpretable influential features from CNN for image classification and captured the interactions between these features by producing a concise decision tree making accurate classification decisions. The proposed methodology made use of pre-trained VGG16 with finetuning to extract feature maps produced by learnt filters. A decision tree was then induced on these extracted features that captured important interactions between the features. On the CelebA image dataset, we successfully produced human interpretable rules capturing the main facial landmarks responsible for segmenting males from females with the use of a decision tree which achieved 89.57% accuracy, while on the Cats vs Dogs dataset 87.55% accuracy was achieved.

Deep learning methods have the ability to accurately predict and interpret what data represents. However, the decision making of a deep learning model is not comprehensible for humans. This is a problem for sectors like market surveillance which needs clarity in the decision making of the used algorithms. This thesis aimed to investigate how a deep learning model can be constructed to make the decision making of the model humanly comprehensible, and to investigate the potential impact on classification performance. A literature study was performed and publicly available explanation methods were collected. The explanation methods LIME, SHAP, model distillation and SHAP TreeExplainer were implemented and evaluated on a ResNet trained on three different time-series datasets. A decision tree was used as the student model for model distillation, where it was trained with both soft and hard labels. A survey was conducted to evaluate if the explanation method could increase comprehensibility. The results were that all methods could improve comprehensibility for people with experience in machine learning. However, none of the methods could provide full comprehensibility and clarity of the decision making. The model distillation reduced the performance compared to the ResNet model and did not improve the performance of the student model. / Djupinlärningsmetoder har egenskapen att förutspå och tolka betydelsen av data. Däremot så är djupinlärningsmetoders beslut inte förståeliga för människor. Det är ett problem för sektorer som marknadsövervakning som behöver klarhet i beslutsprocessen för använda algoritmer. Målet för den här uppsatsen är att undersöka hur en djupinlärningsmodell kan bli konstruerad för att göra den begriplig för en människa, och att undersöka eventuella påverkan av klassificeringsprestandan. En litteraturstudie genomfördes och publikt tillgängliga förklaringsmetoder samlades. Förklaringsmetoderna LIME, SHAP, modelldestillering och SHAP TreeExplainer blev implementerade och utvärderade med en ResNet modell tränad med tre olika dataset. Ett beslutsträd användes som studentmodell för modelldestillering och den blev tränad på båda mjuka och hårda etiketter. En undersökning genomfördes för att utvärdera om förklaringsmodellerna kan förbättra förståelsen av modellens beslut. Resultatet var att alla metoder kan förbättra förståelsen för personer med förkunskaper inom maskininlärning. Däremot så kunde ingen av metoderna ge full förståelse och insyn på hur beslutsprocessen fungerade. Modelldestilleringen minskade prestandan jämfört med ResNet modellen och förbättrade inte prestandan för studentmodellen.

Användningen av AI ökar alltmer inom offentlig förvaltning, däribland komplexa AI-system vars inre funktionalitet är svår att överskåda och förstå. Myndighetsutövning inom offentlig förvaltning måste upprätthålla en nivå av transparens som möjliggör för enskilda att förstå motiveringen bakom ett beslut. I denna studie undersöks hur detta dilemma hanteras inom svensk statlig myndighet, Bolagsverket, genom att applicera modellen Transparens genom Design av Felzmann et al. (2020) på Bolagsverkets arbete med transparens gällande AI-system. Resultaten visar att i stort sett hela problematiken är väl omhändertagen i arbetet på Bolagsverket, där medvetenheten om frågeställningen är mycket goda och väl hanterade; vidare är mycket på förhand givet tack vare svensk offentlighetsprincip men också europeisk lagstiftning. Idag fattar inte AI beslut i egentlig mening inom svensk förvaltning. Nettoresultatet vad gäller transparens i offentlig förvaltning kan mycket väl ge ett tranparenstillskott av tidigare oanat slag med väl planerade, designade och implementerade system, vilket i sin tur är till gagn för både effektivitet och demokratiska värden i den offentliga förvaltningen. / The use of AI is increasingly prevalent in public administration, including complex AI systems whose inner functionality is difficult to oversee and understand. Government agencies must maintain a level of transparency that allows individuals to comprehend the rationale behind a decision. In this study, I examine how this dilemma is addressed within the Swedish government agency Bolagsverket by applying the model Transparency by Design by Felzmann et al. (2020) to Bolagsverket’s work on transparency regarding AI systems. The results indicate that the majority of the issues are well-handled in Bolagsverket’s work, with a high level of awareness and effective management of the relevant questions. Additionally, much is predetermined thanks to the Swedish principle of public access to official documents and European legislation. Currently, AI does not make any literal decisions in Swedish administration. The net result regarding transparency in public administration can indeed provide an unprecedented boost in transparency with well-planned, designed, and implemented systems. This, in turn, benefits both efficiency and democratic values within the public sector.

The field of wireless communications is arguably one of the most rapidly developing technological fields. Therefore, with each new advancement in this field, the complexity of wireless systems can grow significantly. This phenomenon is most visible in mobile communications, where the current 5G and 6G radio access networks (RANs) have reached unprecedented complexity levels to satisfy diverse increasing demands. In such increasingly complex environments, managing resources is becoming more and more challenging. Thus, experts employed performant artificial intelligence (AI) techniques to aid radio resource management (RRM) decisions. However, these AI techniques are often difficult to understand by humans, and may receive unimportant inputs which unnecessarily increase their complexity. In this work, we propose an explainability pipeline meant to be used for increasing humans’ understanding of AI models for RRM, as well as for reducing the complexity of these models, without loss of performance. To achieve this, the pipeline generates diverse feature importance explanations of the models with the help of three explainable artificial intelligence (XAI) methods: Kernel SHAP, CERTIFAI, and Anchors, and performs an importance-based feature selection using one of three different strategies. In the case of Anchors, we formulate and utilize a new way of computing feature importance scores, since no current publication in the XAI literature suggests a way to do this. Finally, we applied the proposed pipeline to a reinforcement learning (RL)- based RRM system. Our results show that we could reduce the complexity of the RL model between ∼ 27.5% and ∼ 62.5% according to different metrics, without loss of performance. Moreover, we showed that the explanations produced by our pipeline can be used to answer some of the most common XAI questions about our RL model, thus increasing its understandability. Lastly, we achieved an unprecedented result showing that our RL agent could be completely replaced with Anchors rules when taking RRM decisions, without a significant loss of performance, but with a considerable gain in understandability. / Området trådlös kommunikation är ett av de snabbast utvecklande tekniska områdena, och varje framsteg riskerar att medföra en signifikant ökning av komplexiteten för trådlösa nätverk. Det här fenomenet är som tydligast i mobil kommunikaiton, framför allt inom 5G och 6G radioaccessnätvärk (RANs) som har nåt nivåer av komplexitet som saknar motstycke. Detta för att uppfylla de ökande kraven som ställs på systemet. I dessa komplexa system blir resurshantering ett ökande problem, därför används nu artificiell intelligens (AI) allt mer för att ta beslut om hantering av radioresurser (RRM). Dessa AI tekniker är dock ofta svåra att förstå för människor, och kan således ges oviktig input vilket leder till att öka AI modellernas komplexitet. I detta arbete föreslås en förklarande pipeline vars mål är att användas för att öka människors förståelse av AI modeller för RRM. Målet är även att minska modellernas komplexitet, utan att förlora prestanda. För att åstadkomma detta genererar pipelinen förklaringar av betydande predikatorer för modellen med hjälp av tre metoder för förklarande artificiell intelligens (XAI). Dessa tre metoder är, Kernel SHAP, CERTIFAI och Anchors. Sedan görs ett predikatorurval baserat på predikatorbetydelse med en av dessa tre metoder. För metoden Anchors formuleras ett nytt sätt att beräkna betydelsen hos predikatorer, eftersom tidigare forskning inte föreslår någon metod för detta. Slutligen appliceras den föreslagna pipelinen på en förstärkt inlärnings- (RL) baserat RRM system. Resultaten visar att komplexiteten av RL modellen kunde reduceras med mellan ∼ 27, 5% och ∼ 62, 5% baserat på olika nyckeltal:er, utan att förlora någon prestanda. Utöver detta visades även att förklaringarna som producerats kan användas för att svara på de vanligaste XAI frågoran om RL modellen, och på det viset har även förståelsen för modellen ökat. Sistnämnt uppnåddes enastående resultat som visade att RL modellen helt kunde ersättas med regler producerade av Anchor-metoden för beslut inom RRM, utan någon störra förlust av prestanda, men med an stor vinst i förståelse.