Stacking, also known as stacked generalization, is a method of ensemble learning where multiple base models are trained on the same dataset, and their predictions are used as input for one or more metamodels in an extra layer. This technique can lead to improved performance compared to single layer ensembles, but often requires a time-consuming trial-and-error process. Therefore, the previously developed Visual Analytics system, StackGenVis, was designed to help users select the set of the most effective and diverse models and measure their predictive performance. However, StackGenVis was developed with only one metamodel: Logistic Regression. The focus of this Bachelor's thesis is to examine how alternative metamodels affect the performance of stacked ensembles through the use of a visualization tool called MetaStackVis. Our interactive tool facilitates visual examination of individual metamodels and metamodels' pairs based on their predictive probabilities (or confidence), various supported validation metrics, and their accuracy in predicting specific problematic data instances. The efficiency and effectiveness of MetaStackVis are demonstrated with an example based on a real healthcare dataset. The tool has also been evaluated through semi-structured interview sessions with Machine Learning and Visual Analytics experts. In addition to this thesis, we have written a short research paper explaining the design and implementation of MetaStackVis. However, this thesis  provides further insights into the topic explored in the paper by offering additional findings and in-depth analysis. Thus, it can be considered a supplementary source of information for readers who are interested in diving deeper into the subject.

During 2018, the Public Transport Administration (Trafikförvaltningen) in the Stockholm region spent approximately 2.2 billion SEK on new infrastructure investments related to the public transport system, many of which were based on their public transport models. The previously used method for validating these models has lacked scientific rigour, efficiency and a systematic approach, which has led to uncertainty in decision making. Furthermore, few scientific studies have been conducted to develop validation methodologies for large-scale models, such as public transport models. For these reasons, a scientific validation methodology for public transport models has been developed in this thesis. This validation methodology has been applied on the 2014 route assignment model used by Trafikförvaltningen, for the transport modes bus, commuter train and local tram. In the developed validation methodology, the selected validation metrics called MAPE, %RMSE and R^2 are used to compare link loads from a route assignment model with observed link loads from an Automatic Passenger Counting (APC) system. To obtain an overview of the performance of the route assignment model, eight different scenarios are set, based on whether the validation metrics meet acceptable thresholds or not. In the application of the developed validation methodology, the average link loads for the morning rush have been validated. To adjust the developed validation methodology to system-specific factors and to set acceptable metric thresholds, discussions with model practitioners have taken place. The validation has been performed on both lines and links, and for bus entire line number series have been validated as well. The validation results show that commuter train meets the set threshold values in a higher proportion than bus and local tram do. However, Trafikförvaltningen is recommended to further calibrate the route assignment model in order to achieve a better model performance. The developed validation methodology can be used for validation of public transport models, and can in combination with model calibration be used in an iterative process to fine-tune model parameters for optimising validation results. Finally, a number of recommendations are proposed for Trafikförvaltningen to increase the efficiency and quality of the validation process, such as synchronising model data with the observed data. / Under 2018 spenderade Trafikförvaltningen ungefär 2,2 miljarder kronor på nya infrastrukturinvesteringar för kollektivtrafiksystemet i Stockholm, varav många av dessa baserades på deras kollektivtrafikmodeller. Den tidigare metoden för att valideras dessa modeller har saknat gedigen vetenskaplig grund, effektivitet och ett systematiskt tillvägagångssätt, vilket lett till osäkerhet gällande investeringsbeslut. Dessutom har få vetenskapliga studier genomförts för att ta fram valideringsmetodologier för storskaliga modeller, såsom kollektivtrafikmodeller. Av dessa skäl har en vetenskaplig valideringsmetodologi för kollektivtrafikmodeller tagits fram i detta examensarbete. Denna valideringsmetodologi har tillämpats på Trafikförvaltningens 2014 års nätutläggningsmodell, för trafikslagen buss, pendeltåg och spårväg. I den framtagna valideringsmetodologin har de valda valideringsmåtten vid namn MAPE, %RMSE och R^2 använts för att jämföra länkbelastningar från en nätutläggningsmodell med observerade länkbelastningar från ett Automatisk Trafikanträkning-system (ATR). För att ge en översikt över modellens precision har åtta scenarios satts baserat på om valideringsmåtten godkänns eller inte enligt tröskelvärden. I tillämpningen av den framtagna valideringsmetodologin har de genomsnittliga länkbelastningarna för morgonens rusningstrafik validerats. För att justera den framtagna valideringsmetodologin efter systemspecifika faktorer och för att sätta godkända tröskelvärden för valideringsmåtten, har diskussioner med trafikanalytiker hållits. Valideringen har utförts både på linjer och länkar, och för buss har även hela linjeserier validerats. Valideringsresultaten för pendeltåg har en högre andel godkända mätningar än buss och spårväg. Trafikförvaltningen rekommenderas dock att kalibrera nätutläggningsmodellen ytterligare för att uppnå ett bättre resultat. Den framtagna valideringsmetodologin kan användas för valideringar av kollektivtrafikmodeller, och kan i kombination med modellkalibrering användas i en iterativ process för att finjustera modellparametrar och därmed optimera valideringsresultaten. Slutligen föreslås ett antal rekommendationer för Trafikförvaltningen för att öka effektiviteten och kvaliteten på valideringsprocessen, till exempel att synkronisera modelldata med observerad data.

L'assistance informatique est devenue une partie indispensable pour la réalisation de procédures chirurgicales modernes. Le désir de créer une nouvelle génération de blocs opératoires intelligents a incité les chercheurs à explorer les problèmes de perception et de compréhension automatique de la situation chirurgicale. Dans ce contexte de prise de conscience de la situation, un domaine de recherche en plein essor adresse la reconnaissance automatique du flux chirurgical. De grands progrès ont été réalisés pour la reconnaissance des phases et des gestes chirurgicaux. Pourtant, il existe encore un vide entre ces deux niveaux de granularité dans la hiérarchie du processus chirurgical. Très peu de recherche se concentre sur les activités chirurgicales portant des informations sémantiques vitales pour la compréhension de la situation. Deux facteurs importants entravent la progression. Tout d'abord, la reconnaissance et la prédiction automatique des activités chirurgicales sont des tâches très difficiles en raison de la courte durée d'une activité, de leur grand nombre et d'un flux de travail très complexe et une large variabilité. Deuxièmement, une quantité très limitée de données cliniques ne fournit pas suffisamment d'informations pour un apprentissage réussi et une reconnaissance précise. À notre avis, avant de reconnaître les activités chirurgicales, une analyse soigneuse des éléments qui composent l'activité est nécessaire pour choisir les bons signaux et les capteurs qui faciliteront la reconnaissance. Nous avons utilisé une approche d'apprentissage profond pour évaluer l'impact de différents éléments sémantiques de l'activité sur sa reconnaissance. Grâce à une étude approfondie, nous avons déterminé un ensemble minimum d'éléments suffisants pour une reconnaissance précise. Les informations sur la structure anatomique et l'instrument chirurgical sont de première importance. Nous avons également abordé le problème de la carence en matière de données en proposant des méthodes de transfert de connaissances à partir d'autres domaines ou chirurgies. Les méthodes de ''word embedding'' et d'apprentissage par transfert ont été proposées. Ils ont démontré leur efficacité sur la tâche de prédiction d'activité suivante offrant une augmentation de précision de 22%. De plus, des observations pertinentes / Computer assistance became indispensable part of modern surgical procedures. Desire of creating new generation of intelligent operating rooms incited researchers to explore problems of automatic perception and understanding of surgical situations. Situation awareness includes automatic recognition of surgical workflow. A great progress was achieved in recognition of surgical phases and gestures. Yet, there is still a blank between these two granularity levels in the hierarchy of surgical process. Very few research is focused on surgical activities carrying important semantic information vital for situation understanding. Two important factors impede the progress. First, automatic recognition and prediction of surgical activities is a highly challenging task due to short duration of activities, their great number and a very complex workflow with multitude of possible execution and sequencing ways. Secondly, very limited amount of clinical data provides not enough information for successful learning and accurate recognition. In our opinion, before recognizing surgical activities a careful analysis of elements that compose activity is necessary in order to chose right signals and sensors that will facilitate recognition. We used a deep learning approach to assess the impact of different semantic elements of activity on its recognition. Through an in-depth study we determined a minimal set of elements sufficient for an accurate recognition. Information about operated anatomical structure and surgical instrument was shown to be the most important. We also addressed the problem of data deficiency proposing methods for transfer of knowledge from other domains or surgeries. The methods of word embedding and transfer learning were proposed. They demonstrated their effectiveness on the task of next activity prediction offering 22% increase in accuracy. In addition, pertinent observations about the surgical practice were made during the study. In this work, we also addressed the problem of insufficient and improper validation of recognition methods. We proposed new validation metrics and approaches for assessing the performance that connect methods to targeted applications and better characterize capacities of the method. The work described in this these aims at clearing obstacles blocking the progress of the domain and proposes a new perspective on the problem of surgical workflow recognition.