This thesis investigates stereo vision based techniques for supporting work with teleoperated mobile robots. In particular, the tasks of navigation and control of the robot in unknown environments as well as control of the manipulator are considered. Support is required with map overviews and images from novel viewpoints. Challenges are to create maps and novel images in real time, exclusively from image sequences of a calibrated stereo camera that is allowed to move arbitrarily in three dimensions. Furthermore, no assumptions are made about the environment. These requirements have led to research in the areas of stereo vision, camera motion estimation, mapping and novel view synthesis. Several contributions have been made in these areas. Firstly, the behaviour of stereo correlation is analysed and a new real time stereo algorithm derived, which has reduced matching errors compared to traditional algorithms. Secondly, a new robust real time camera motion estimation method is described, which exclusively uses stereo images and permits arbitrary camera motion. Thirdly, a new method for creating two-dimensional maps from images of a stereo camera under arbitrary three-dimensional motion is presented. Finally, novel view synthesis is performed in disparity space of source stereo images using a new method. All the proposed techniques have been integrated into the Immediate Reality Scanner (IRIS) system. The system performs incremental mapping, immediate virtual walkthroughs and dynamic novel views exclusively from a stereo camera under arbitrary three-dimensional motion. Mapping and immediate virtual walkthroughs can be performed in real time, concurrently to scanning the environment. The thesis describes camera calibration, stereo processing, camera motion estimation, map building, novel view synthesis and the integration into IRIS, including evaluations in all areas. It is concluded that IRIS does not only fulfil the requirements of the anticipated application on teleoperated mobile robots, but could also be used in a wide range of robotics and non-robotics applications. It is believed that IRIS is the first system that allows incremental mapping and immediate virtual walkthroughs in real time concurrently to scanning an unconstrained environment with an arbitrarily moving stereo camera

A novel interface for robotic teleoperation was developed to enable accurate and highly efficient teleoperation of the Industrial Reconfigurable Anthropomorphic Dual-arm (IRAD) system and other robotic systems. In order to achieve a revolutionary increase in operator productivity, the bilateral/master-slave approach must give way to shared autonomy and unilateral control; autonomy must be employed where possible, and appropriate sensory feedback only where autonomy is impossible; and today’s low-information/high feedback model must be replaced by one that emphasizes feedforward precision and minimal corrective feedback. This is emphasized for task spaces outside of the traditional anthropomorphic scale such as mobile manipulation (i.e. large task spaces) and high precision tasks (i.e. very small task spaces). The system is demonstrated using an anthropomorphically dimensioned industrial manipulator working in task spaces from one meter to less than one millimeter, in both simulation and hardware. This thesis discusses the design requirements and philosophy of this interface, provides a summary of prototype teleoperation hardware, simulation environment, test-bed hardware, and experimental results. / text

In this research work, a study has been carried out on the current development of teleoperation and the control strategies adopted to co-ordinate human operator input and computer autonomous control. Based on the findings, a human and computer cooperative control concept has been developed to overcome the shortcomings of the existing control strategies. It utilises the human and computer characteristic advantages to complement each other to improve task performance and increase task success rate. Some aspects of the tasks are shared between humans and computers while computer assistance is introduced into the human supervisory functions in this co-operative control. Traditional problems of teleoperation such as time delay and task dependence of high-level control have been addressed and overcome by utilising the distributed and parallel processing system architecture and an expandable knowledge base incorporated with generic and task specific modules. Guidelines and an ideal system model have been proposed to provide system developers with future design guidance. A telerobotic system has been built based on the developed human and computer cooperative control concept. It uses the client/server model to achieve distributed and parallel processing. Operators can perform tasks through direct manual control and computer assisted task-level control provided in the system. Multiple manual inputs have been introduced to suit different control conditions. They provide the operator with intuitive and hardware independent input methods to manipulate the remote robot. A computer vision system has been developed to provide not only the visual feedback of the remote environment but also assistance in object location and robot navigation. Task modules containing generic and specific task handling functions are used for achieving human and computer co-operation at both task planning and execution levels. The experiment results obtained from the system evaluation have shown that the task completion time in co-operative control mode is half the time required in human direct manual control. Task performance achieved in co-operative control mode is less sensitive to the difference between the operators' capability. In conclusion, the usability and effectiveness of the co-operative control, and the validity of the techniques of camera calibration and image processing has been proven through the demonstration task.

A novel control scheme for asymmetric bilateral teleoperation systems is developed based on linear models of the hardware, with two scenarios considered: i) only an upper bound on the time delay and ii) an upper and lower bounds on the time delay. Lyapunov based methods are used with linear matrix inequalities to prove that the system error is bounded between the manipulators. To ensure stability, a master side impedance matching controller is used. Simulations were conducted using the Matlab and Simulink platform to run the LMI code and simulate the system. Experiments were then conducted using actual hardware to verify the results, with deviations from imulation results. The variations were due to non-linearities in the hardware and model parameter approximation errors. Finally, suggestions for future work are made.

Teleoperation systems broaden human ability to perform a task in a real or virtual, local or remote environment. An emerging application of such systems is in dual-user teleoperation or haptic simulation systems in which two users collaboratively perform a task in a shared real or virtual environment. Examples of this application are in human haptic guidance for rehabilitation therapy and medical surgical training. In such collaborative systems, users interact with each other and a user's decision is affected by the other user's decisions. This interaction between users bring out the need for new control architecture design methods. In addition, the controller should maintain system stability and achieve desired performance under various operational conditions, including contact with a wide range of environments and being interfaced with two users displaying highly variable arm dynamics. To this purpose, a class of shared control architectures for human haptic guidance has been developed. The architectures feature a dominance factor that adjusts the supremacy of the trainer over the trainee in the execution of a task. To tackle the stability issue, a novel robust stability analysis framework for unconditional stability analysis of multi-user teleoperation/haptic systems has been proposed. In terms of performance these systems have been evaluated kinesthetically, referring to the dynamics felt by the users, or task-based, referring to the quantities that measure task efficiency and effort. For kinesthetic performance evaluation of dual-user systems as opposed to the single-user systems, there are two users interacting with the environment and with each other. In these systems it may be desirable for a user to not only feel the environment but also to sense the other user dynamics. Hence, some of the previously defined performance measures for single-user systems have been extended for dual-user systems. Furthermore, two novel performance measures for multilateral dual-user systems have been introduced. Finally, to assess the task-based performance of the proposed architectures, a user study has been conducted for trajectory following tasks on a developed dual-user haptic simulation testbed under various environmental conditions, such as different environment geometries, environment view points and environment dynamics. / Thesis (Ph.D, Electrical & Computer Engineering) -- Queen's University, 2010-11-22 18:38:05.142

Telepresence is defined as the sensation of being present at a remote robot task site while physically present at a local control station. This concept has received substantial attention in the recent past as a result of hypothesized benefits of "presence" experiences on human task performance with teleoperation systems. Human factors research, however, has made little progress in establishing a relationship between the concept of telepresence and teleoperator performance. This has been attributed to the multidimensional nature of telepresence, the lack of appropriate studies to elucidate this relationship, and the lack of a valid and reliable, objective measure of telepresence. Subjective measures (e.g., questionnaires, rating scales) are most commonly used to measure telepresence. Objective measures have been proposed, including behavioral responses to stimuli presented in virtual worlds (e.g. ducking virtual objects). Other research has suggested use of physiological measures, such as cardiovascular responses to indicate the extent of telepresence experiences in teleoperation tasks. The objective of the present study was to assess the utility of using measures of attention allocation and situation awareness (SA) to objectively describe telepresence. Attention and SA have been identified as cognitive constructs potentially underlying telepresence experiences. Participants in this study performed a virtual mine neutralization task involving remote control of a simulated robotic rover and integrated tools to locate, uncover, and dispose of mines. Subjects simultaneously completed two secondary tasks that required them to monitor for "low battery" signals associated with operation of the vehicle and controls. Subjects were divided into three groups of eight according to task difficulty, which was manipulated by varying the number, and spacing, of mines in the task environment. Performance was measured as average time to neutralize four mines. Telepresence was assessed using a "Presence" questionnaire. Situation awareness was measured using the Situation Awareness Global Assessment Technique. Attention was measured as a ratio of the number of ?low battery" signal detections to the total number of signals presented through the secondary task displays. Analysis of variance results revealed level of difficulty to significantly affect performance time and telepresence. Regression analysis revealed level of difficulty, immersive tendencies, and attention to explain significant portions of the variance in telepresence.

<p> The performance index in teleoperation, transparency, is often defined as linear scaling of force and position between the master/ operator and slave/ environment. Motivated by applications involving soft tissue manipulation such as robotic surgery, the transparency objective is generalized in this thesis to include static nonlinear and linear-time-invariant filter mappings between the master I slave position and force signals. Lyapunov-based adaptive motion/ force controllers are proposed to achieve the generalized transparency objectives. Using Lyapunov stability theory the mapped position and force tracking errors are shown to converge in the presence of dynamic uncertainty in the master I slave robots and user I environment dynamics. Given a priori known bounds on unknown dynamic parameters, a framework for robust stability analysis is proposed that uses stability of Lur'ePostinkov systems and Nyquist/Bode envelopes of interval plant systems. Methods for finding the required Nyquist/Bode envelopes are presented in this thesis. A comprehensive stability analysis is performed under different sets of generalized mappings. For nonlinear mapping of either position or force, robust stability depends on stability of an equivalent Lur'e-Postinikov system. Stability results of such systems are discussed in this thesis. In particular, the on and off-axis circle theorems are utilized. Using these theorems, sufficient teleoperation stability regions are obtained that are far less conservative than those obtained from passivity. In the special case of LTI filtered force and position mappings the exact robust stability regions are obtained by showing stability of the relevant closed-loop characteristic polynomial. The proposed robust stability test uses the phase values of a limited set of extremal polynomials. </p> <P> To demonstrate the utility of the generalized performance measures, a stiffness discrimination tele-manipulation task is considered in which the user compares and contrasts the stiffness of soft environments via haptic exploration in the presence and absence of visual feedback. Using adaptive psychophysical perception experiments a nonlinear force mapping is shown to enhance stiffness discrimination thresholds. The design guidelines for this enhanced nonlinear force mapping are reported in this thesis. Generalized nonlinear and linear filtered mappings are achieved in experiments with a two-axis teleoperation system where the details of implementation are given. </p> / Thesis / Master of Applied Science (MASc)

Both civilian and defense industry rely heavily on robotics which continues to gain a more prominent role. To exemplify, defense strategies in Middle East have relied upon robotic drones and teleoperative assistant robots for mission oriented tasks. These operations have been crucial in saving the lives of soldiers and giving us the edge in mitigating disasters. Future assistive robotics will have direct human interaction and will reside in normal human environments. As the advancement in technology continues to occur, there will be a focus towards eliminating the direct human control and replacing it with higher level autonomy. Further, advancements in electronics and electromechanical components will reduce the cost and makes the assistive robotics accessible to the masses. This thesis focuses on robotic teleoperation technology and the future high level control of assistant robotics. A dexterous 16 degree of freedom hand with bend sensors for precise joint positions was designed, modeled, fabricated and characterized. The design features a unique motor actuation mechanism that was 3D printed to reduce the cost and increase the modularity. The upper body was designed to be biomimetic with dimensions similar to that of a typical six foot tall male. The upper body of the humanoid consists of a 4 degree of freedom shoulder and upper arm with direct feedback at each joint. A theoretical nonlinear switching controller was designed to control these 4 degrees of freedom. The entire system was teleoperative controlled with an Xbox Kinect that tracks the skeletal points of a user and emulates these 3D points to the joints of humanoid upper body. This allows for a direct user control over a robotic assistive upper body with nothing more than a human emulating the desired movements. / Master of Science

Performance and stability of bilateral teleoperation control systems are adversely affected by variations in environment dynamics and time delay in communication channel. Prior relevant research in the literature has mainly yielded control algorithms that sacrifice performance in order to guarantee robust stability. In contrast, this thesis proposes methods to deal with these two main problems in order to maintain the stability without compromising performance. To handle changes in environment dynamics, a multiple model controller for teleoperation is introduced. It is assumed that the dynamics of the environment are governed by a model from a finite set of environment models at any given time with Markov chain switching between these models. The first-order generalized pseudo-Bayesian (GPB1) multi-model estimation technique is used to identify the effective model at each time step given the sensory observations. The control action is a weighted sum of mode-based control laws that are designed for each mode of operation. The second major problem in teleoperation systems that this thesis deals with is communication channel time delay. The constant time-delay problem is solved using two different methods, i.e. discrete-time and continuous-time predictive type Linear Quadratic Gaussian (LQG) controllers. The treatment of the problem in the discrete-time domain allows for the development of a finite dimension state-space model that explicitly encompasses the time delay. The robustness of the controller with respect to uncertainty in the system parameters is examined via Nyquist analysis. In continuous-time, a modified state transformation is proposed to obtain delay-free dynamics based on the original dynamics with delayed inputs and outputs. The application of the continuous-time LQG control synthesis to these reduced dynamics yields a control law that guarantees closed-loop stability and performance. Mode-based controllers are designed for each phase of operation, i.e. free motion/soft contact and contact with rigid environments. Performance objectives such as position tracking and tool impedance shaping for free motion/soft contact, as well as position and force tracking for contact with rigid environments are incorporated into the LQG control design framework. Simulation and experimental results are presented for each of the proposed controllers in various scenarios. These results demonstrate the effectiveness of the proposed methods in providing a stable transparent interface for teleoperation in free motion and in contact with rigid environments. / Thesis / Master of Applied Science (MASc)

<p>The subject of this thesis is teleoperation, and especially teleoperation with demanding time constraints due to significant dynamics inherent in the task. A comprehensive background is given, describing many aspects of teleoperation, from history and applications to operator interface hardware and relevant control theory concepts. Then follows a presentation of the research done by the author.</p><p>Two prototypical highly dynamic teleoperation tasks have been attempted: high speed driving, and ball catching. Systems have been developed for both, employing operator interfaces tailored to facilitate perception of the remote scene and including assistive features to promote successful task completion within the required time frame. Prediction of the state at the remote site as well as of operator action has been applied to address the problem of delays arising when using the Internet as the communication channel.</p> / <p>Detta arbete handlar om teleoperation, som skulle kunna översättas med fjärrstyrning, och speciellt sådan som ställer stränga tidskrav på grund av att uppgiftens natur inbegriper en avsevärd dynamik. Först ges en bred bakgrund där många aspekter av teleoperation belyses, från dess historia och användningsområden till hårdvara för användargränssnitt och relevant reglerteori. Sedan följer en presentation av författarens forskning på området.</p><p>Två prototypuppgifter har använts, som båda involverar snabba dynamiska förlopp: styrning av en mobil robot i hög hastighet och fångst av kastade bollar i luften. Teleoperationssystem har utvecklats för båda uppgifterna.</p><p>Användargränssnitt har skräddarsytts för att göra det lättare för operatören att uppfatta vad som händer med och omkring den styrda roboten, och aktiva hjälpmedel har byggts in för att ge större möjligheter att fullgöra uppgiften på tillgänglig tid. Modellering och prediktion av roboten och dess omgivning, men också av operatörens kommandon, har använts för att lösa de fördröjningsproblem som uppstår när internet används som kommunikationsmedium.</p> / Neurobotics