This thesis focuses on making improvements to time-delayed teleoperation systems, with both direct and semi-autonomous haptic control, by addressing the challenges associated with force-position (F-P) predictive architectures. As the time delay from the communication channel increases, system stability and performance degrade. Previously, solutions focused on communication channel stability and environment force estimation methods that primarily rely on linearization of the Hunt-Crossley (HC) contact model. These result in a loss of transparency in the system and limiting use cases from linearization assumptions. Moreover, semi-autonomous solutions aimed at decreasing user effort and automating subtasks, such as obstacle avoidance and user guidance, require training or singularly focus on joint space tasks. This work addresses the shortcomings of the aforementioned methods by refocusing on system components to achieve more favorable dynamics during environment contact with the use of a series elastic actuator (SEA), investigating alternative HC parameter estimation techniques, and synthesizing an assistive semi-autonomous control framework that predicts user intention recognition and automates gross motion tasks. Experimental results with a remote SEA demonstrate improved performance with stiff environments in delays of up to two seconds round trip time. The coupling of the force and position through the actuator along with simultaneous sensing capabilities also show robustness for contact with soft environments. Further improvements with soft environment contact are achieved through HC parameter estimation, with smooth parameter update switching using a Sigmoid function. A novel application of Chebyshev polynomial approximation for adaptive parameter estimation of the HC model was also proposed. This approach provides control via backstepping with adaptive parameter estimation using Lyapunov methods. Additionally, this method reduces excitation requirements by using nonlinear swapping and the data accumulation concept to guarantee parameter convergence. A simulated teleoperation system demonstrates the effectiveness of this approach and initial results from experiment show promise for this approach in practice. Finally, a user study involving a pick and place task produced favorable results for the proposed semi-autonomous framework which significantly reduced task completion times. / Master of Science / Teleoperated systems are powerful solutions for remotely executing tasks in situations where autonomous solutions are not robust enough and/or user knowledge is desired for a task. However, teleoperation performance and stability is degraded by delays in the communication channel. A common way to deal with time delay is to use a predictive controller on the local side to cancel out the delay by knowing the remote side dynamics. Previous approaches have focused on stabilizing the communication channel or the use of estimators and observers to better capture the remote side dynamics. The drawback of these approaches is that they achieve stability at the expense of system transparency, leading to divergence in the force and position matching between the master and remote side. Many of the methods for environment force estimation involves linearizing contact models, creating limitations in their application. Moreover, semi-autonomous solutions aimed at decreasing user effort and automating subtasks such as obstacle avoidance and user guidance require training data sets for the algorithm or only focus individually on joint space tasks. This thesis addresses the shortcomings of the aforementioned methods by refocusing on system components to achieve more favorable dynamics using a series elastic actuator (SEA) while interacting with the environment, investigating nonlinear and linear contact model estimation methods for identifying parameters of the Hunt-Crossley (HC) model, and synthesising an assistive semi-autonomous control framework that predicts user intention for task execution. Experimental results for the use of an SEA demonstrate improved performance with stiff environments in delays of up to two seconds round trip time (RTT). The coupling of the force and position through the actuator along with simultaneous sensing capabilities also showed robustness for contact with soft environments. Various estimation methods for HC parameter identification was investigated to improve the local side model. A novel application of Chebyshev polynomial approximation of the HC model with adaptive parameter estimation was also proposed to provide control along with decreasing the excitation requirements by using backsteping control with nonlinear swapping and the data accumulation concept. A simulated teleoperation system demonstrated the effectiveness of this approach with a smooth paramater update transition. Initial results from experiment also show promise for this approach in practice. Finally, a user study involving a pick and place task produced favorable results for the proposed semi-autonomous framework which significantly reduced task completion times.

Despite the advancements in the development of autonomous vehicles (AVs), there are still numerous complex situations in which AVs may encounter challenges. In recent years, the concept of teleoperation, which entails establishing a connection between a remote operator and the AV, has garnered substantial attention from both AV companies and governmental bodies as a viable safety backup method. However, a research gap is apparent when it comes to the remote manipulation of AVs positioned at a considerable distance. This gap involves a) AV with a temporal delay through real-time direct control within the constraints of current wireless communication technology in an unpredictable road environment, and b) enhancing the AV's inherent detection capabilities to augment its autonomous control abilities, thereby reducing the operator's workload. To address this research gap, this dissertation introduces an innovative teleoperation system. Initially, we devise a control system utilizing the wave variable approach as a communication method to alleviate the impact of signal latency. And Radial Basis Function Networks (RBFN) are employed to effectively manage the uncertain nonlinear dynamics of the vehicle. Subsequently, a saliency-based object detection (OD) algorithm, named SalienDet, is proposed to identify objects not present in the training sample set. SalienDet incorporates saliency maps generated without prior information into the neural network, enhancing image features for unfamiliar objects. This augmentation significantly aids the OD algorithm in detecting previously unknown objects, thereby empowering the AV to possess an improved perception ability. This advancement is particularly valuable when the operator imparts driving advice to the AV instead of exercising direct control. In conclusion, this dissertation makes a noteworthy contribution to AV teleoperation by furnishing a comprehensive solution that spans various aspects of AV teleoperation. / Doctor of Philosophy / This dissertation revolves around the teleoperation of autonomous vehicles (AVs), with the objective of formulating a comprehensive teleoperation system that encompasses two critical aspects: direct control and indirect control. In the initial segment of the dissertation, we introduce a real-time teleoperation direct control system based on neural networks. This framework plays a pivotal role in assisting operators in navigating AVs efficiently, especially in the face of challenges such as communication delays and complex external environments. Following this, we present a novel saliency-based object detection (OD) algorithm. This algorithm empowers the AV to recognize potential objects beyond its prior knowledge, thereby enhancing its level of autonomous control, particularly when operators opt not to exercise direct control over the remote AV. Our research findings delve into the essential facets of AV teleoperation. The developed teleoperation system serves as a valuable reference for future researchers and engineers dedicated to advancing autonomous vehicle technology.

The importance of cranes can not be understated, as they are essential in various fields. They require precision and heavy safety considerations when operated on-site since they are responsible for numerous accidents and even fatalities. Traditional methods for remote operation sometimes fail to depict the intricate scenarios that operators experience in on-site environments, resulting in worsened immersion and, potentially, performance. Accordingly, the immediate purpose of this thesis is to address this limitation by enhancing an existing crane simulator with virtual reality and audio integration. In order to validate the impact of the proposed improvements, the simulator underwent testing in an experimental environment that included 16 individuals. Each participant tested five versions of the simulator, all of which varied in their combination of machinery sounds, environmental sounds,  and VR, to assess the impact of the improvements by measuring their operating performance and on-site immersion. The research findings indicate that using virtual reality affects the realism and sense of presence of the simulator, enabling operators to be more immersed in the crane operation. The incorporation of sound, more specifically environmental and machine sounds, enriches the virtual environment by offering audio cues that accurately reflect real-world operations. The operating performance was, however, not significantly improved with the addition of VR and audio. The results obtained from questionnaires, interviews, and performance metrics show that the built-in improvements can potentially increase the safety and efficiency of crane operations.

The thesis focuses on the design of a control strategy for safety-critical remote teleoperation. The main goal is to make the controlled system track the desired velocity specified by a human operator while avoiding obstacles despite communication delays. Different methods adopting Control Barrier Functions (CBFs) and Model Predictive Control (MPC) have been explored and tested. In this combination, CBFs are used to define the safety constraints the system has to respect to avoid obstacles, while MPC provides the framework for filtering the desired input by solving an optimization problem. The resulting input is sent to the remote system, where appropriate low-level velocity controllers translate it into system-specific commands. The main novelty of the thesis is a method to make the CBFs robust against the uncertainties affecting the system’s state due to network delays. Other techniques are investigated to improve the quality of the system information starting from the delayed one and to formulate the optimization problem without knowing the specific dynamic of the controlled system. The results show how the proposed method successfully solves the safetycritical teleoperation problem, making the controlled systems avoid obstacles with different types of network delay. The controller has also been tested in simulation and on a real manipulator, demonstrating its general applicability when reliable low-level velocity controllers are available. / Avhandlingen fokuserar på utformningen av en kontrollstrategi för säkerhetskritisk fjärrstyrd teleoperation. Huvudmålet är att få det kontrollerade systemet att följa den önskade hastigheten som specificeras av en mänsklig operatör samtidigt som hinder undviks trots kommunikationsfördröjningar. Olika metoder som använder Control Barrier Functions (CBFs) och Model Predictive Control har undersökts och testats. I denna kombination används CBFs för att definiera de säkerhetsbegränsningar som systemet måste respektera för att undvika hinder, medan MPC utgör ramverket för filtrering av den önskade indata genom att lösa ett optimeringsproblem. Den resulterande indata skickas till fjärrsystemet, där lämpliga hastighetsregulatorer på låg nivå översätter den till systemspecifika kommandon. Den viktigaste nyheten i avhandlingen är en metod för att göra CBFs robust mot de osäkerheter som påverkar systemets tillstånd på grund av nätverksfördröjningar. Andra tekniker undersöks för att förbättra kvaliteten på systeminformationen med utgångspunkt från den fördröjda informationen och för att formulera optimeringsproblemet utan att känna till det kontrollerade systemets specifika dynamik. Resultaten visar hur den föreslagna metoden framgångsrikt löser det säkerhetskritiska teleoperationsproblemet, vilket gör att de kontrollerade systemen undviker hinder med olika typer av nätverksfördröjningar. Styrningen har också testats i simulering och på en verklig manipulator, vilket visar dess allmänna tillämpbarhet när tillförlitliga lågnivåhastighetsregulatorer finns tillgängliga.

The area of Human-Robot Interaction deals with problems not only related to robots interacting with humans, but also with problems related to humans interacting and controlling robots. This dissertation focuses on the latter and evaluates multi-sensory (vision, hearing, touch, smell) feedback interfaces as a means to improve robot-operator cognition and performance. A set of four empirical studies using both simulated and real robotic systems evaluated a set of multi-sensory feedback interfaces with various levels of complexity. The task scenario for the robot in these studies involved the search for victims in a debris-filled environment after a fictitious catastrophic event (e.g., earthquake) took place. The results show that, if well-designed, multi-sensory feedback interfaces can indeed improve the robot operator data perception and performance. Improvements in operator performance were detected for navigation and search tasks despite minor increases in workload. In fact, some of the multi-sensory interfaces evaluated even led to a reduction in workload. The results also point out that redundant feedback is not always beneficial to the operator. While introducing the concept of operator omni-directional perception, that is, the operator’s capability of perceiving data or events coming from all senses and in all directions, this work explains that feedback redundancy is only beneficial when it enhances the operator omni-directional perception of data relevant to the task at hand. Last, the comprehensive methodology employed and refined over the course of the four studies is suggested as a starting point for the design of future HRI user studies. In summary, this work sheds some light on the benefits and challenges multi-sensory feedback interfaces bring, specifically on teleoperated robotics. It adds to our current understanding of these kinds of interfaces and provides a few insights to assist the continuation of research in the area.

The demands of mobile robots system are growing rapidly in recent days. Autonomous and semi-autonomous mobile robots for indoor environments have become great helper to human, such as transport robot in warehouse, cleaning robot for apartment and various humanoids for personal assistance, education and space exploration. This thesis project work describes how to implement a mobile robot system for unstructured indoor environment. This mobile robot system is able to build a map of the environment, plan path and navigate robot between user-defined locations in the environment autonomously or teleoperated by human through GUI or keyboard on a remote control terminal. Additionally, this mobile robot will be equipped a laser pointer worked as an interaction interface between robot and human by showing robot intensions to operator or surroundings. The development of this project work will involve two rapid growing features, one is depth camera and the other is ROS. Use depth camera as the primary sensor has been accepted by more and more mobile robot platform. ROS, as the fast-growing robot software framework, is an inevitable path when developing robot systems.

Teleoperation of ground/aerial vehicle extends operator's ability (e.g. expertise, strength, mobility) into the remote environment, and haptic feedback enhances the human operator's perception of the slave environment. In my thesis, two cases are studied: wheeled mobile robot (MWR) haptic tele-driving over the Internet and unmanned aerial vehicle (UAV) haptic teleoperation over the Internet. We propose novel control frameworks for both dynamic WMR and kinematic WMR in various tele-driving modes, and for a "mixed" UAV with translational dynamics and attitude kinematics. The recently proposed passive set-position modulation (PSPM) framework is extended to guarantee the passivity and/or stability of the closed-loop system with time-varying/packet-loss in the communication; and proved performance in steady state is shown by theoretical measurements.For UAV teleoperation, we also derive a backstepping trajectory tracking control with robustness analysis. Experimental results for dynamic/kinematic WMR and an indoor quadrotor-type UAV are presented to show the efficacy of the proposed control framework.

Teleoperation of an unmanned vehicle is a challenging task for human operators especially when the vehicle is out of line of sight. Improperly designed and applied display interfaces directly affect the operation performance negatively and even can result in catastrophic failures. If these teleoperation missions are human-critical then it becomes more important to improve the operator performance by decreasing workload, managing stress and improving situational awareness. This research aims to develop electrical and control system of an unmanned ground vehicle (UGV) using an All-Terrain Vehicle (ATV) and validate the development with investigation of the effects of force feedback devices on the teleoperation performance. After development, teleoperation tests are performed to verify that force feedback generated from the dynamic obstacle information of the environment improves teleoperation performance. Results confirm this statement and the developed UGV is verified for future research studies. Development of UGV, algorithms and real system tests are included in this thesis.

This thesis focuses on the development of a system in which a team of heterogeneous mobile robots can cooperate to perform a wide range of tasks. In order that a group of heterogeneous robots can cooperate among them, one of the most important parts to develop is the creation of an architecture which gives support for the cooperation. This architecture is developed by means of embedding agents and interfacing agent code with native low-level code. It also addresses the implementation of resource sharing among the whole group of robots, that is, the robots can borrow capabilities from each-other.In order to validate this architecture, some cooperative applications have been implemented. The first one is an application where a group of robots must cooperate in order to safely navigate through an unknown environment. One robot with camera calculates the optical flow values from the images, and from these values calculates the "time to contact" values. This information is shared among the team so that any robot can navigate without colliding with the obstacles.The second cooperative application consists of enabling the team of heterogeneous robots to create a certain formation and navigate maintaining this formation. The application consists of two parts or stages. The first one is the creation of the formation, where a robot with the camera can detect where the rest of the robots are in the environment and indicates to them which is their initial position in the formation. In the second stage the robots must be able to navigate through an environment following the path that the robot with the laser indicates. Due to the odometry errors of the robots, the camera of one of the robots is used so that robots which lose their correct position in the formation can re-align themselves. Finally, in an attempt to facilitate access to the robots of the team and to the information that their accessories provide, a system for the teleoperation of the team has been implemented. This system can be used for teaching robotics or to facilitate the tasks of programming and debugging in the research tasks.

Haptic devices allow a human to interact physically with a remote or virtual environment by providing tactile feedback to the user. In general haptic devices can be classified in two groups according to the energetic nature of their actuators. Devices using electric motors, pneumatic or hydraulic cylinders or other similar actuators that can add energy to the system are considered "active." Devices using brakes, clutches or other passive actuators are considered "passive" haptic devices. The research presented here focuses on the use of passive haptic devices used during teleoperation, the remote control of a "slave" device by the haptic "master" device. An actuation scheme as well as three different control methods is developed for providing the user with haptic feedback. As a final step, the effectiveness of the controllers is compared to that of a commercially available active haptic device. Twenty subjects provide data that shows the usefulness of the passive device in three typical teleoperation tasks.