Artificial intelligence (AI) is revolutionizing contemporary industries and being applied in application domains ranging from recommendation systems to self-driving cars. In scenarios in which humans are interacting with an AI, inaccurate algorithms could lead to human mistreatment or even harmful events. Human-in-the-loop computing is a machine learning approach desiring hybrid intelligence, the combination of human and machine intelligence, to achieve accurate and interpretable results. This thesis applies human-in-the-loop computing in a Design Science Research project with a Swedish manufacturing company to make operational processes more efficient. The thesis aims to investigate emerging design principles useful for designing machine learning algorithms of hybrid intelligence. Hereby, the thesis has two key contributions: First, a theoretical framework is built that comprises general design knowledge originating from Information Systems (IS) research. Second, the analysis of empirical findings leads to the review of general IS design principles and to the formulation of useful design principles for human-in-the-loop computing. Whereas the principle of AI-readiness improves the likelihood of strategical AI success, the principle of hybrid intelligence shows how useful it can be to trigger a demand for human-in-the-loop computing in involved stakeholders. The principle of use case-marketing might help designers to promote the customer benefits of applying human-in-the-loop computing in a research setting. By utilizing the principle of power relationship and the principle of human-AI trust, designers can demonstrate the humans’ power over AI and build a trusting human-machine relationship. Future research is encouraged to extend and specify the formulated design principles and employ human-in-the-loop computing in different research settings. With regard to technological advancements in brain-machine interfaces, human-in-the-loop computing might even become much more critical in the future.

<p>With the advents in hardware techniques and mobile computing powers, Augmented Reality (AR) has been promising in various areas of our everyday life and work. By superimposing virtual assets onto the real world, the boundary between the digital and physical spaces has been significantly blurred, which bridges a large amount of digital augmentation and intelligence with the surroundings of the physical reality. Meanwhile, thanks to the increasing developments of Artificial Intelligence (AI) perception algorithms such as object detection, scene reconstruction, and human tracking, the dynamic behaviors of digital AR content have extensively been associated with the physical contexts of both humans and environments. This context-awareness enabled by the emerging techniques enriches the potential interaction modalities of AR experiences and improves the intuitiveness and effectiveness of the digital augmentation delivered to the consumers. Therefore, researchers are gradually motivated to include more contextual information in the AR domain to create novel AR experiences used for augmenting their activities in the physical world.</p> <p>    </p> <p>On a broader level, our work in this thesis focuses on novel designs and modalities that combine contextual information with AR content behaviors in context-aware AR experiences. In particular, we design the AR experiences by inspecting different types of contexts from the real world, namely 1) human actions, 2) physical entities, and 3) interactions between humans and physical environments. To this end, we explore 1) software and hardware modules, and conceptual models that perceive and interpret the contexts required by the AR experiences, and 2) supportive authoring tools and interfaces that enable users and designers to define the associations of the AR contents and the interaction modalities leveraging the contextual information. In this thesis, we mainly study the following workflows: 1) designing adaptive AR tutoring systems for human-machine-interactions, 2) customizing human-involved context-aware AR applications, 3) authoring shareable semantic-aware AR experiences, and 4) enabling hand-object-interaction datasets collection for scalable context-aware AR application deployment. We further develop the enabling techniques and algorithms including 1) an adaptation model that adaptively vary the AR tutoring elements based on the real-time learner's interactions with the physical machines, 2) a customized video-see-through AR headset for pervasive human-activity detecting, 3) a semantic adaptation model that adjusts the spatial relationships of the AR contents according to the semantic understanding of different physical entities and environments, and 4) an AR-based interface that empowers novice users to collect high-quality datasets used for training user- and cite-specific networks in hand-object-interaction-aware AR applications.</p> <p><br></p> <p>Takeaways from the research series include 1) the usage of the modern AI modules effectively enlarges both the spatial and contextual scalability of AR experiences, and 2) the design of the authoring systems and interfaces lowers the barrier for end-users and domain experts to leverage AI outputs in the creation of AR experiences that are tailored for target users. We conclude that involving AI techniques in both the creation and implementation stages of AR applications is crucial to building an intelligent, adaptive, and scalable ecosystem of context-aware AR applications.</p>

A paradigm shift is emerging in system reliability and maintainability. The military and industrial sectors are moving away from the traditional breakdown and scheduled maintenance to adopt concepts referred to as Condition Based Maintenance (CBM) and Prognostic Health Management (PHM). In addition to signal processing and subsequent diagnostic and prognostic algorithms these new technologies involve storage of large volumes of both quantitative and qualitative information to carry out maintenance tasks effectively. This not only requires research and development in advanced technologies but also the means to store, organize and access this knowledge in a timely and efficient fashion. Knowledge-based expert systems have been shown to possess capabilities to manage vast amounts of knowledge, but an intelligent systems approach calls for attributes like learning and adaptation in building autonomous decision support systems. This research presents an integrated knowledge-based approach to diagnostic reasoning for CBM of engineering systems. A two level diagnosis scheme has been conceptualized in which first a fault is hypothesized using the observational symptoms from the system and then a more specific diagnostic test is carried out using only the relevant sensor measurements to confirm the hypothesis. Utilizing the qualitative (textual) information obtained from these systems in combination with quantitative (sensory) information reduces the computational burden by carrying out a more informed testing. An Industrial Language Processing (ILP) technique has been developed for processing textual information from industrial systems. Compared to other automated methods that are computationally expensive, this technique manipulates standardized language messages by taking advantage of their semi-structured nature and domain limited vocabulary in a tractable manner. A Dynamic Case-based reasoning (DCBR) framework provides a hybrid platform for diagnostic reasoning and an integration mechanism for the operational infrastructure of an autonomous Decision Support System (DSS) for CBM. This integration involves data gathering, information extraction procedures, and real-time reasoning frameworks to facilitate the strategies and maintenance of critical systems. As a step further towards autonomy, DCBR builds on a self-evolving knowledgebase that learns from its performance feedback and reorganizes itself to deal with non-stationary environments. A unique Human-in-the-Loop Learning (HITLL) approach has been adopted to incorporate human feedback in the traditional Reinforcement Learning (RL) algorithm.

A helicopter can be used to transport a load hanging from a suspension cable. This technique is frequently used in construction, firefighting, and disaster relief operations, among other applications. Unfortunately, the suspended load swings, which makes load positioning difficult and can degrade control of the helicopter. This dissertation investigates the use of input shaping (a command-filtering technique for reducing vibration) to mitigate the load swing problem. The investigation is conducted using two different, but complementary, approaches. One approach studies manual tracking tasks, where a human attempts to make a cursor follow an unpredictably moving target. The second approach studies horizontal repositioning maneuvers on small-scale helicopter systems, including a novel testbed that limits the helicopter and suspended load to move in a vertical plane. Both approaches are used to study how input shaping affects control of a flexible element (the suspended load) and a driven base (the helicopter). In manual tracking experiments, conventional input shapers somewhat degraded control of the driven base but greatly improved control of the flexible element. New input shapers were designed to improve load control without negatively affecting base control. A method for adjusting the vibration-limiting aggressiveness of any input shaper between unshaped and fully shaped was also developed. Next, horizontal repositioning maneuvers were performed on the helicopter testbed using a human-pilot-like feedback controller from the literature, with parameter values scaled to match the fast dynamics of the model helicopter. It was found that some input shapers reduced settling time and peak load swing when applied to Attitude Command or Translational Rate Command response types. When the load was used as a position reference instead of the helicopter, the system was unstable without input shaping, and adding input shaping to a Translational Rate Command was able to stabilize the load-positioning system. These results show the potential to improve the safety and efficiency of helicopter suspended load operations.

With cancer being one of the leading causes of death globally, and with oral cancers being among the most common types of cancer, it is of interest to conduct large-scale oral cancer screening among the general population. Deep Learning can be used to make this possible despite the medical expertise required for early detection of oral cancers. A bottleneck of Deep Learning is the large amount of data required to train a good model. This project investigates two topics: certainty calibration, which aims to make a machine learning model produce more reliable predictions, and Active Learning, which aims to reduce the amount of data that needs to be labeled for Deep Learning to be effective. In the investigation of certainty calibration, five different methods are compared, and the best method is found to be Dirichlet calibration. The Active Learning investigation studies a single method, Cost-Effective Active Learning, but it is found to produce poor results with the given experiment setting. These two topics inspire the further development of the cytological annotation tool CytoBrowser, which is designed with oral cancer data labeling in mind. The proposedevolution integrates into the existing tool a Deep Learning-assisted annotation workflow that supports multiple users.

Recent urbanization and industrialization have brought tremendous pressure and challenges to modern autonomous systems. When considering multiple complex tasks, cooperation and coordination between multiple agents can improve efficiency in a system. In real-world applications, multi-agent systems (MAS) are widely used in various fields, such as robotics, unmanned aerial systems, autonomous vehicles, distributed sensor networks, etc. Unlike traditional MAS systems based on pre-defined algorithms and rules, a special human-in-loop (HIL) based MAS involves human interactions to enhance the system’s adaptability for special scenarios, as well as apply human preferences for robot control. However, existing HIL strategies are primarily based on human involvement at a low level, such as mixed-initiative control and mixed-agent scenarios with both human-driven and intelligent robots. There are fewer investigations on applying HIL in high-level coordination. In particular, designing a coordination strategy for multi-task multi-agent scenarios, which can also deal with real-time human commands, will be one of the key topics of this Master’s thesis project. In this thesis work, different kinds of tasks described by signal temporal logic (STL) are created for agents, which can be enforced by control barrier function (CBF) constraints. Both centralized and distributed frameworks are designed for agent coordination. In detail, the centralized strategy is developed for machine-to-infrastructure (M2I) communication, by using the nonlinear model predictive control (NMPC) method to obtain collision-free trajectories. The distributed strategy utilizing graph theory is proposed for machine-to-machine (M2M), in order to reduce computation time by offloading. Most importantly, a HIL model is generated for both frameworks to apply online human commands to the coordination, with a novel task allocation protocol. Simulations and experiments are carried out on both Matlab and Python-based ROS simulators, to show that proposed frameworks can achieve obvious performance advantages in safety, smoothness, and stability for task completion. Numerical results are provided to validate the feasibility and applicability of our algorithms. / Den senaste urbaniseringen och industrialiseringen har medfört enormt tryck och utmaningar för moderna autonoma system. Vid beaktande av flera komplexa uppgifter kan samarbete och samordning mellan flera agenter förbättra effektiviteten i ett system. I verkliga tillämpningar används multiagent-system (MAS) i stor utsträckning inom olika områden, såsom robotik, obemannade luftfarkoster, autonoma fordon, distribuerade sensorsystem etc. Till skillnad från traditionella MAS-system baserade på fördefinierade algoritmer och regler, innebär ett särskilt människa-i-loop (HIL)-baserat MAS mänsklig interaktion för att förbättra systemets anpassningsförmåga till speciella scenarier samt anpassa mänskliga preferenser för robotstyrning. Emellertid är befintliga HIL-strategier främst baserade på mänsklig inblandning på en låg nivå, såsom mixad-initiativkontroll och mixade agentscenarier med både människa-drivna och intelligenta robotar. Det finns färre undersökningar om att tillämpa HIL på högnivåkoordination. Särskilt att utforma en koordineringsstrategi för fleruppgiftsfleragent-scenarier, som också kan hantera mänskliga kommandon i realtid, kommer att vara ett av huvudämnena för detta masterprojekt. I detta examensarbete skapas olika typer av uppgifter beskrivna av signaltemporallogik (STL) för agenter, som kan upprätthållas genom styrbarriärfunktions (CBF) -begränsningar. Både centraliserade och distribuerade ramverk utformas för agentkoordination. Mer specifikt utvecklas den centraliserade strategin för maskin-till-infrastruktur (M2I)-kommunikation genom att använda icke-linjär modellprediktiv reglering (NMPC) för att erhålla kollisionsfria trajektorier. Den distribuerade strategin med användning av grafteori föreslås för maskin-till-maskin (M2M) för att minska beräkningstiden genom avlastning. Viktigast av allt genereras en HIL-modell för båda ramverken för att tillämpa online-mänskliga kommandon på koordinationen med en ny protokoll för uppgiftstilldelning. Simuleringar och experiment utförs på både Matlab och Python-baserade ROS-simulatorer för att visa att de föreslagna ramverken kan uppnå tydliga prestandafördelar när det gäller säkerhet, smidighet och stabilitet för uppgiftsslutförande. Numeriska resultat presenteras för att validera genomförbarheten och tillämpligheten hos våra algoritmer.

Microprocessor-controlled knees (MPK’s) allow amputees to walk with increasing ease and safety as technology progresses. As an amputee is fitted with a new MPK, the knee’s internal parameters are tuned to the user’s preferred settings in a controlled environment. These parameters determine various gait control settings, such as flexion target angle or swing extension resistance. Though these parameters may work well during the initial fitting, the MPK experiences various internal &amp; external environmental changes throughout its life-cycle, such as product wear, changes in the amputee’s muscle strength, temperature changes, etc. This work investigates the feasibility of using a reinforcement learning (RL) control to adapt the MPK’s swing resistance to consistently induce the amputee’s preferred swing performance in realtime. Three gait features were identified as swing performance indicators for the RL algorithm. Results show that the RL control is able to learn and improve its tuning performance in terms of Mean Absolute Error over two 40-45 minute training sessions with a human-in-the-loop. Additionally, results show promise in using transfer learning to reduce strenuous RL training times. / Mikroprocessorkontrollerade knän (MPK) gör att amputerade kan utföra fysiska aktiviteter med ökad lätthet och säkerhet allt eftersom tekniken fortskrider. När en ny MPK monteras på en amputerad person, anpassas knäts interna parametrar till användarens i ett kontrollerad miljö. Dessa parametrar styr olika gångkontrollinställningar, såsom flexionsmålvinkel eller svängförlängningsmotstånd. Även om parametrarna kan fungera bra under den initiala anpassningen, upplever den MPK olika interna och yttre miljöförändringar under sin hela livscykel, till exempel produktslitage, förändringar i den amputerades muskelstyrka, temperaturförändringar, etc. Detta arbete undersöker möjligheten av, med hjälp av en förstärkningsinlärningskontroll (RL), att anpassa MPK svängmotstånd för att konsekvent inducera den amputerades föredragna svängprestanda i realtid. Tre gångegenskaper identifierades som svingprestandaindikatorer för RL-algoritmen. Resultaten visar att RL-kontrollen kan lära sig och förbättra sin inställningsprestanda i termer av Mean Absolute Error under två 40-45 minuters träningspass med en människa-i-loopen. Dessutom är resultaten lovande när det gäller att använda överföringsinlärning för att minska ansträngande RL-träningstider.

<p dir="ltr">This research explores the potential of AR for lunar missions via the xEMU spacesuit. A market analysis of commercial off-the-shelf AR devices identifies technological trends and constraints that inform the architectural decisions for AR integration with the xEMU. User evaluations in simulated work environments ensure lunar informatics align with crew needs. Drawing insights from human-in-the-loop testing of COTS AR devices, qualitative test results underscore the importance of display optimization, occlusion management, and environmental considerations for enhancing the AR experience during lunar EVAs. Grounded in a task analysis from JETT3 analog testing, crew workflows and communication dynamics are baselined, underscoring the vital role of communication and collaboration. Integrating AR into the EVA work environment holds the potential to streamline decision-making, improve navigation, and enhance overall efficiency, but may come with unintended operational consequences. The human-centered approach prioritizes crew involvement, ensuring that technology remains a facilitator rather than an encumbering element in lunar exploration. The study's significance lies in advancing AR technology for lunar EVAs, guiding hardware design, and enabling seamless integration into the EVA work environment. AR holds promise in reshaping the human-technology relationship, empowering crew members, maximizing science output, and contributing to the next chapter in lunar exploration.</p>

<p dir="ltr">Data is collected everywhere in our world; however, it often is noisy and incomplete. Different sources of data may have different characteristics, quality levels, or come from dynamic and diverse environments. This poses challenges for both humans who want to gain insights from data and machines which are learning patterns from data. How can we leverage the diversity of data across domains to enhance our understanding and decision-making? In this thesis, we address this question by proposing novel methods and applications that use multiple domains as more holistic sources of information for both human and machine learning tasks. For example, to help human operators understand environmental dynamics, we show the detection and localization of distribution shifts to problematic features, as well as how interpretable distributional mappings can be used to explain the differences between shifted distributions. For robustifying machine learning, we propose a causal-inspired method to find latent factors that are robust to environmental changes and can be used for counterfactual generation or domain-independent training; we propose a domain generalization framework that allows for fast and scalable models that are robust to distribution shift; and we introduce a new dataset based on human matches in StarCraft II that exhibits complex and shifting multi-agent behaviors. We showcase our methods across various domains such as healthcare, natural language processing (NLP), computer vision (CV), etc. to demonstrate that learning from data across domains can lead to more faithful representations of data and its generating environments for both humans and machines.</p>

Nokia launched a website service Customer Insights (CI) to managers and executives from operator companies to track their customers’ experience. An upgraded mobile service is developed for providing more valuable information. The data was retrieved from the same dataset but less amount of information would be displayed in the mobile application. Two questions need to be answered in this design work, what to show in the application and how to show them. A tough situation in user research and a large amount of data made the user-centered design hard to answer the ‘what’ question. Based on experts’ view, data points that have different patterns from other data could be valuable. Considering ML is good at quantitative analysis tool and anomaly detection method can help filter outliers, we combined it with User-centered Design (UCD) in the content preparation. The challenge was how to mind the gap between experts and real users’ expectations. The initial user research was missed and involving users during the modeling progress was not realistic. Our strategy was to select information by anomaly detection methods, got users’ feedbacks after launching the application and utilized those feedbacks to improve the algorithm. Based on the study in ML, PCA anomaly detection was chosen and it worked well in filtering outliers in this case. Two validations proved the possibility of improving the precision and recall of the results based on supervised learning and labeled data. On the other hand, UCD focused on answering the ‘how’ problem based on a questionnaire, personas, scenarios and design guidelines. The results from ML research were also considered in the design work, thus the interface and interaction design would help the algorithm to a larger extent. Four experts participated in the design evaluation. All three iterations of the design helped us to summarize some universal guidance on how to design for similar mobile applications. / Nokia lanserade en webbtjänst Customer Insights (CI) för att chefer och ledare från operativa företag ska kunna följa kundernas erfarenheter. En uppgraderad mobiltjänst utvecklas för att ge mer värdefull information. Uppgifterna hämtas från samma datamängd, men mindre mängd information visas i mobilapplikationen. Två frågor måste besvaras i detta designarbete, nämligen vad som ska visas i applikationen och hur de ska visas. Den svåra situationen i användarforskningen och den stora mängden data gjorde det svårt att besvara frågan om "vad" i den användarcentrerade designen. Enligt experternas uppfattning kan datapunkter som har olika mönster jämfört med andra data vara värdefulla. Med tanke på att ML är ett bra verktyg för kvantitativ analys och att metoden för anomalidetektion kan hjälpa till att filtrera avvikelser, kombinerade vi den med UCD i innehållsberedningen. Utmaningen var hur vi skulle kunna hantera klyftan mellan experternas och de verkliga användarnas förväntningar. Den inledande användarundersökningen missades och det var inte realistiskt att involvera användarna under modelleringsprocessen. Vår strategi var att välja ut information med hjälp av metoder för anomalidetektion, få användarnas feedback efter lanseringen av applikationen och använda dessa feedback för att förbättra algoritmen. Baserat på studien om ML valdes PCA-anomalidetektion och den fungerade bra för att filtrera utfall i det här fallet. Två valideringar visade att det är möjligt att förbättra precisionen och återkallandet av resultaten baserat på övervakad inlärning och märkta data. Å andra sidan fokuserade UCD på att besvara "hur"-problemet med hjälp av ett frågeformulär, personas, scenarier och riktlinjer för utformning. Resultaten från ML-forskningen beaktades också i designarbetet, vilket innebär att gränssnitts- och interaktionsdesignen skulle hjälpa algoritmen i större utsträckning. Fyra experter deltog i designutvärderingen. Alla tre iterationer av designen hjälpte oss att sammanfatta några universella riktlinjer för hur man utformar liknande mobilapplikationer.