京都大学 / 新制・課程博士 / 博士(情報学) / 甲第24856号 / 情博第838号 / 新制||情||140(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 鹿島, 久嗣, 教授 河原, 達也, 教授 森本, 淳 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

No description available.

Human expert drivers have the unique ability to build complex perceptive models using correlated sensory inputs and outputs. In the case of longitudinal vehicle traction, this work will show a direct correlation in longitudinal acceleration to throttle input in a controlled laboratory environment. In fact, human experts have the ability to control a vehicle at or near the performance limits, with respect to vehicle traction, without direct knowledge of the vehicle states; speed, slip or tractive force. Traditional algorithms such as PID, full state feedback, and even sliding mode control have been very successful at handling low level tasks where the physics of the dynamic system are known and stationary. The ability to learn and adapt to changing environmental conditions, as well as develop perceptive models based on stimulus-response data, provides expert human drivers with significant advantages. When it comes to bandwidth, accuracy, and repeatability, automatic control systems have clear advantages over humans; however, most high performance control systems lack many of the unique abilities of a human expert. The underlying motivation for this work is that there are advantages to framing the traction control problem in a manner that more closely resembles how a human expert drives a vehicle. The fundamental idea is the belief that humans have a unique ability to adapt to uncertain environments that are both temporal and spatially varying. In this work, a novel approach to traction control is developed using an anthropomimetic control synthesis strategy. The proposed anthropomimetic traction control algorithm operates on the same correlated input signals that a human expert driver would in order to maximize traction. A gradient ascent approach is at the heart of the proposed anthropomimetic control algorithm, and a real-time implementation is described using linear operator techniques, even though the tire-ground interface is highly non-linear. Performance of the proposed anthropomimetic traction control algorithm is demonstrated using both a longitudinal traction case study and a combined mode traction case study, in which longitudinal and lateral accelerations are maximized simultaneously. The approach presented in this research should be considered as a first step in the development of a truly anthropomimetic solution, where an advanced control algorithm has been designed to be responsive to the same limited input signals that a human expert would rely on, with the objective of maximizing traction. This work establishes the foundation for a general framework for an anthropomimetic control algorithm that is capable of learning and adapting to an uncertain, time varying environment. The algorithms developed in this work are well suited for efficient real time control in ground vehicles in a variety of applications from a driver assist technology to fully autonomous applications. / Ph. D.

Human-AI interaction can improve overall performance, exceeding the performance that either humans or AI could achieve separately, thus producing a whole greater than the sum of the parts. Visual analytics enables collaboration between humans and AI through interactive visual interfaces. Semantic interaction is a design methodology to enhance visual analytics systems for sensemaking tasks. It is widely applied for sensemaking in high-stakes domains such as intelligence analysis and academic research. However, existing semantic interaction systems support collaboration between humans and traditional machine learning models only; they do not apply state-of-the-art deep learning techniques. The contribution of this work is the effective integration of deep neural networks into visual analytics systems with semantic interaction. More specifically, I explore how to redesign the semantic interaction pipeline to enable collaboration between human and deep learning models for sensemaking tasks. First, I validate that semantic interaction systems with pre-trained deep learning better support sensemaking than existing semantic interaction systems with traditional machine learning. Second, I integrate interactive deep learning into the semantic interaction pipeline to enhance inference ability in capturing analysts' precise intents, thereby promoting sensemaking. Third, I add semantic explanation into the pipeline to interpret the interactively steered deep learning model. With a clear understanding of DL, analysts can make better decisions. Finally, I present a neural design of the semantic interaction pipeline to further boost collaboration between humans and deep learning for sensemaking. / Doctor of Philosophy / Human AI interaction can harness the separate strengths of human and machine intelligence to accomplish tasks neither can solve alone. Analysts are good at making high-level hypotheses and reasoning from their domain knowledge. AI models are better at data computation based on low-level input features. Successful human-AI interactions can perform real-world, high-stakes tasks, such as issuing medical diagnoses, making credit assessments, and determining cases of discrimination. Semantic interaction is a visual methodology providing intuitive communications between analysts and traditional machine learning models. It is commonly utilized to enhance visual analytics systems for sensemaking tasks, such as intelligence analysis and scientific research. The contribution of this work is to explore how to use semantic interaction to achieve collaboration between humans and state-of-the-art deep learning models for complex sensemaking tasks. To do this, I first evaluate the straightforward solution of integrating the pretrained deep learning model into the traditional semantic interaction pipeline. Results show that the deep learning representation matches human cognition better than hand engineering features via semantic interaction. Next, I look at methods for supporting semantic interaction systems with interactive and interpretable deep learning. The new pipeline provides effective communication between human and deep learning models. Interactive deep learning enables the system to better capture users' intents. Interpretable deep learning lets users have a clear understanding of models. Finally, I improve the pipeline to better support collaboration using a neural design. I hope this work can contribute to future designs for the human-in-the-loop analysis with deep learning and visual analytics techniques.

Autonomous multi-robot systems have found many real-world applications in factory settings, rescue tasks and light shows. Albeit these successful applications, the multi-robot system is usually pre-programmed with limited flexibility for online adaptation. Having a human-in-the-loop feature would provide additional flexibility such as handling unexpected situations, detecting and correcting bad behaviours and supporting the automated decision making. In addition, it would also allow for an extra level of cooperation between the robots and the human that facilitates certain real-world tasks, for example in the agricultural sector. Control barrier functions (CBFs), as a convenient modular-design tool, will be mainly explored. CBFs have seen a lot of development in recent years and extending them to the field of multi-robot systems is still new. In particular, creating an original distributed approach, instead of a centralized one, will be one of the key topics of this Master’s thesis project. In this thesis work, several multi-robot coordination protocols and safety constraints will be identified and these constraints will be enforced using a control barrier function-enabled mixer module. This module will take in the commands from both the planner and the human operator, prioritizing the commands from the human operator as long as the safety constraints are not violated. Otherwise, the mixer module will filter the commands and send out a safe alternative. The underlying multi-robot tasks are expected to be achieved whenever feasible. Simulations in ROS, Python and MATLAB environments are developed to experimentally assess the safety and optimality of the system, and experiments with real robots in a lab are performed to show the applicability of this algorithm. Finally, a distributed approach to the mixer module has been developed, based on previous research and extended to allow for more versatility. This is of key importance since it would allow each robot to compute its own controller based on local information, making the multi-robot system both more robust and flexible to be deployed on real-world applications. / Autonoma multirobotsystem har fått många verkliga tillämpningar i fabriksmiljöer, räddningsuppdrag och ljusshower. Trots dessa framgångsrika tillämpningar är multirobotsystemet vanligtvis förprogrammerat med begränsad flexibilitet för anpassning online. En människa i loopen skulle ge ytterligare flexibilitet, t.ex. när det gäller att hantera oväntade situationer, upptäcka och korrigera dåliga beteenden och stödja det automatiska beslutsfattandet. Dessutom skulle det också möjliggöra en extra samarbetsnivå mellan robotarna och människan som underlättar vissa verkliga uppgifter, till exempel inom jordbrukssektorn. Kontrollbarriärfunktioner (CBF), som ett bekvämt verktyg för modulbaserad utformning, kommer huvudsakligen att undersökas. CBF:er har utvecklats mycket under de senaste åren och det är fortfarande nytt att utvidga dem till flerrobotsystem. Att skapa ett originellt distribuerat tillvägagångssätt i stället för ett centraliserat kommer att vara ett av de viktigaste ämnena i detta examensarbete. I detta examensarbete kommer flera samordningsprotokoll och säkerhetsbegränsningar för flera robotar att identifieras och dessa begränsningar kommer att upprätthållas med hjälp av en mixermodul med kontrollbarriärfunktion. Denna modul kommer att ta emot kommandon från både planeraren och den mänskliga operatören och prioritera kommandon från den mänskliga operatören så länge säkerhetsbegränsningarna inte överträds. I annat fall kommer mixermodulen att filtrera kommandona och skicka ut ett säkert alternativ. De underliggande multirobotuppgifterna förväntas uppnås närhelst det är möjligt. Simuleringar i ROS-, Python- och MATLAB-miljöerna utvecklas för att experimentellt bedöma systemets säkerhet och optimalitet, och experiment med riktiga robotar i ett labb utförs för att visa algoritmens tillämpbarhet. Slutligen har ett distribuerat tillvägagångssätt för mixermodulen utvecklats, baserat på tidigare forskning och utökat för att möjliggöra större mångsidighet. Detta är av central betydelse eftersom det skulle göra det möjligt för varje robot att beräkna sin egen styrning utifrån lokal information, vilket gör systemet med flera robotar både mer robust och flexibelt för att kunna användas i verkliga tillämpningar.

In this dissertation work, a methodology is proposed to enable a robot to identify an object to be grasped and its intended grasp configuration while a human is teleoperating a robot towards the desired object. Based on the detected object and grasp configuration, the human is assisted in the teleoperation task. The environment is unstructured and consists of a number of objects, each with various possible grasp configurations. The identification of the object and the grasp configuration is carried out in real time, by recognizing the intention of the human motion. Simultaneously, the human user is assisted to preshape over the desired grasp configuration. This is done by scaling the components of the remote arm end-effector motion that lead to the desired grasp configuration and simultaneously attenuating the components that are in perpendicular directions. The complete process occurs while manipulating the master device and without having to interact with another interface. Intention recognition from motion is carried out by using Hidden Markov Model (HMM) theory. First, the objects are classified based on their shapes. Then, the grasp configurations are preselected for each object class. The selection of grasp configurations is based on the human knowledge of robust grasps for the various shapes. Next, an HMM for each object class is trained by having a skilled teleoperator perform repeated preshape trials over each grasp configuration of the object class in consideration. The grasp configurations are modeled as the states of each HMM whereas the projections of translation and orientation vectors, over each reference vector, are modeled as observations. The reference vectors are the ideal translation and rotation trajectories that lead the remote arm end-effector towards a grasp configuration. During an actual grasping task performed by a novice or a skilled user, the trained model is used to detect their intention. The output probability of the HMM associated with each object in the environment is computed as the user is teleoperating towards the desired object. The object that is associated with the HMM which has the highest output probability, is taken as the desired object. The most likely Viterbi state sequence of the selected HMM gives the desired grasp configuration. Since an HMM is associated with every object, objects can be shuffled around, added or removed from the environment without the need to retrain the models. In other words, the HMM for each object class needs to be trained only once by a skilled teleoperator. The intention recognition algorithm was validated by having novice users, as well as the skilled teleoperator, grasp objects with different grasp configurations from a dishwasher rack. Each object had various possible grasp configurations. The proposed algorithm was able to successfully detect the operator's intention and identify the object and the grasp configuration of interest. This methodology of grasping was also compared with unassisted mode and maximum-projection mode. In the unassisted mode, the operator teleoperated the arm without any assistance or intention recognition. In the maximum-projection mode, the maximum projection of the motion vectors was used to determine the intended object and the grasp configuration of interest. Six healthy and one wheelchair-bound individuals, each executed twelve pick-and-place trials in intention-based assisted mode and unassisted mode. In these trials, they picked up utensils from the dishwasher and laid them on a table located next to it. The relative positions and orientations of the utensils were changed at the end of every third trial. It was observed that the subjects were able to pick-and-place the objects 51% faster and with less number of movements, using the proposed method compared to the unassisted method. They found it much easier to execute the task using the proposed method and experienced less mental and overall workloads. Two able-bodied subjects also executed three preshape trials over three objects in intention-based assisted and maximum projection mode. For one of the subjects, the objects were shuffled at the end of the six trials and she was asked to carry out three more preshape trials in the two modes. This time, however, the subject was made to change their intention when she was about to preshape to the grasp configurations. It was observed that intention recognition was consistently accurate through the trajectory in the intention-based assisted method except at a few points. However, in the maximum-projection method the intention recognition was consistently inaccurate and fluctuated. This often caused to subject to be assisted in the wring directions and led to extreme frustration. The intention-based assisted method was faster and had less hand movements. The accuracy of the intention based method did not change when the objects were shuffled. It was also shown that the model for intention recognition can be trained by a skilled teleoperator and be used by a novice user to efficiently execute a grasping task in teleoperation.

abstract: Robotic lower limb prostheses provide new opportunities to help transfemoral amputees regain mobility. However, their application is impeded by that the impedance control parameters need to be tuned and optimized manually by prosthetists for each individual user in different task environments. Reinforcement learning (RL) is capable of automatically learning from interacting with the environment. It becomes a natural candidate to replace human prosthetists to customize the control parameters. However, neither traditional RL approaches nor the popular deep RL approaches are readily suitable for learning with limited number of samples and samples with large variations. This dissertation aims to explore new RL based adaptive solutions that are data-efficient for controlling robotic prostheses. This dissertation begins by proposing a new flexible policy iteration (FPI) framework. To improve sample efficiency, FPI can utilize either on-policy or off-policy learning strategy, can learn from either online or offline data, and can even adopt exiting knowledge of an external critic. Approximate convergence to Bellman optimal solutions are guaranteed under mild conditions. Simulation studies validated that FPI was data efficient compared to several established RL methods. Furthermore, a simplified version of FPI was implemented to learn from offline data, and then the learned policy was successfully tested for tuning the control parameters online on a human subject. Next, the dissertation discusses RL control with information transfer (RL-IT), or knowledge-guided RL (KG-RL), which is motivated to benefit from transferring knowledge acquired from one subject to another. To explore its feasibility, knowledge was extracted from data measurements of able-bodied (AB) subjects, and transferred to guide Q-learning control for an amputee in OpenSim simulations. This result again demonstrated that data and time efficiency were improved using previous knowledge. While the present study is new and promising, there are still many open questions to be addressed in future research. To account for human adaption, the learning control objective function may be designed to incorporate human-prosthesis performance feedback such as symmetry, user comfort level and satisfaction, and user energy consumption. To make the RL based control parameter tuning practical in real life, it should be further developed and tested in different use environments, such as from level ground walking to stair ascending or descending, and from walking to running. / Dissertation/Thesis / Doctoral Dissertation Electrical Engineering 2020

Safety in human in the loop systems, systems that change behavior with human input, is difficult to achieve. This difficulty can cost lives. As desired system capability grows, so too does the requisite complexity of the system. This complexity can result in designers not accounting for every use case of the system and unintentionally designing in unsafe behavior. Furthermore, complexity of operation and control can result in operators becoming confused during use or receiving insufficient training in the first place. All these cases can result in unsafe operations. One method of improving safety is implementing the use of formal models during the design process. These formal models can be analyzed mathematically to detect dangerous conditions, but can be difficult to produce without time, money, and expertise.<br> This document details the study of potential methods for constructing formal models autonomously from recorded observations of system use, minimizing the need for system expertise, saving time, money, and personnel in this safety critical process. I first discuss how different system characteristics affect system modeling, isolating specific traits that most clearly affect the modeling process Then, I develop a technique for modeling a simple, digital, menu-based system based on a record of user inputs. This technique attempts to measure the availability of different inputs for the user, and then distinguishes states by comparing input availabilities. From there, I compare paths between states and check for shared behaviors. I then expand the general procedure to capture the behavior of a flight simulator. This system more closely resembles real-world safety critical systems and can therefore be used to approximate a real use case of the method outlined. I use machine learning tools for statistical analysis, comparing patterns in system behavior and user behaviors. Last, I discuss general conclusions on how the modeling approaches outlined in this document can be improved and expanded upon.<br> For simple systems, we find that inputs alone can produce state machines, but without corresponding system information, they are less helpful for determining relative safety of different use cases than is needed. Through machine learning, we find that records of complex system use can be decomposed into sets of nominal and anomalous states but determining the causal link between user inputs and transitions between these conditions is not simple and requires further research.

Machine Learning (ML) and Artificial Intelligence (AI) impact many aspects of human life, from recommending a significant other to assist the search for extraterrestrial life. The area develops rapidly and exiting unexplored design spaces are constantly laid bare. The focus in this work is one of these areas; ML systems where decisions concerning ML model training, usage and selection of target domain lay in the hands of domain experts.  This work is then on ML systems that function as a tool that augments and/or enhance human capabilities. The approach presented is denoted Human In Command ML (HIC-ML) systems. To enquire into this research domain design experiments of varying fidelity were used. Two of these experiments focus on augmenting human capabilities and targets the domains commuting and sorting batteries. One experiment focuses on enhancing human capabilities by identifying similar hand-painted plates. The experiments are used as illustrative examples to explore settings where domain experts potentially can: independently train an ML model and in an iterative fashion, interact with it and interpret and understand its decisions.  HIC-ML should be seen as a governance principle that focuses on adding value and meaning to users. In this work, concrete application areas are presented and discussed. To open up for designing ML-based products for the area an abstract model for HIC-ML is constructed and design guidelines are proposed. In addition, terminology and abstractions useful when designing for explicability are presented by imposing structure and rigidity derived from scientific explanations. Together, this opens up for a contextual shift in ML and makes new application areas probable, areas that naturally couples the usage of AI technology to human virtues and potentially, as a consequence, can result in a democratisation of the usage and knowledge concerning this powerful technology.

Human-in-the-loop simulators have long been used in the research community as well as in industry. The aviation field has been the pioneers in the use of simulators for design purposes. In contrast, corresponding activities in the automotive area have been less widespread. Published reports on experimental activities based on human-in-the-loop simulations have focused on methods used in the study, but nobody seems to have taken a step back and looked at the wider methodological picture of Simulator-Based Design. The purpose of this thesis is to fill this gap by drawing, in part, upon the author’s long experience in this field. In aircraft and lately also in ground vehicles there has been a technology shift from pure mechanics to computer-based systems. The physical interface has turned into screen-based solutions. This trend towards glass has just begun for ground vehicles. This development in vehicle technology has opened the door for new design approaches, not only for design itself, but also for the development process. Simulator-Based Design (SBD) is very compatible with this trend. The first part of this thesis proposes a structure for the process of SBD and links it to the corresponding methodology for software design. In the second part of the thesis the focus changes from methodology to application and specifically to the design of three-dimensional situation displays. Such displays are supposed to support the human operator with a view of a situation beyond the more or less limited visual range. In the aircraft application interest focuses on the surrounding air traffic in the light of the evolving free-flight concept, where responsibility for separation between aircraft will be (partly) transferred from ground-based flight controllers to air crews. This new responsibility must be supported by new technology and the situational view must be displayed from the perspective of the aircraft. Some basic design questions for such 3D displays were investigated resulting in an adaptive interface approach, where the current situation and task govern the details of information presentation. The thesis also discusses work on situation displays for ground vehicles. The most prominent example may be the Night Vision system, where the road situation ahead is depicted on a screen in the cab. The existing systems are based on continuous presentation, an approach that we have questioned, since there is strong evidence for negative behavioral adaptation. This means, for example, that the driver will drive faster, since vision has been enhanced, and thereby consume the safety margins that the system was supposed to deliver. Our investigation supports a situation-dependant approach and no continuous presentation. In conclusion, the results from our simulator-based studies showed advantages for adaptive interface solutions. Such design concepts are much more complicated than traditional static interfaces. This finding emphasizes the need for more dynamic design resources in order to have a complete understanding of the situation-related interface changes. The use of human-in-the-loop simulators and deployment of Simulator-Based Design will satisfy this need.