Telesurgery relies on fast and reliable data transmission between the surgeon and tele-operator side over lossy and delay constrained networks. Medical data involves audio, video, ECG and Force Feedback data. When these media streams are transmitted through best effort networks, the temporal information gets affected due to network constraints. Major network degradation is due to the Force Feedback device with rendering rate of 1 KHz, hence data is generated every millisecond. In our proposal we concentrate on improving the synchronization of force feedback device on varying networking conditions. Force feedback data is generated by operating a source (surgical) device which controls the movement of remote device. It has a great potential in improving telemedicine facilities, when included with the support of different multimedia services. The channel imposes delay and packet loss constraints for such devices which require unique solutions, unlike audio or video media, due to its high rendering rate. Current research supports Force Feedback in fiber optic communication, packet switched networks. However, such schemes are not feasible in supporting surgical telepresence system. While efforts are made to support force feedback media in wireless medium, few works have addressed delay synchronization and loss of data. There exists no previous work which has attempted to provide an efficient integrated solution where video and force feedback information have been supported by the same network. This thesis focuses in providing an integrated architecture that supports the force feedback data over a collaborative network and improves the data synchronization and packet loss prediction in the remote side over a varying network link. The goal will be to evaluate the support of such data types. We have implemented a Linear Packet Predictor Algorithm which predicts the missing packet value. Data generated from the source device are sent as UDP packets. UDP transmission is unreliable and hence we use an RTP over UDP to make it reliable. Each packet will have the current position of the device and force applied. We use a Microsoft Sidewinder Force Feedback joystick. The handle of the joystick is located at the center of the base. So we record the position of the device on both positive and negative axis moving in a two dimensional space. This device provides rotational movement and hence drastic change in position occurs within milliseconds. Once the packet arrives at the receiver side, the control unit checks for the sequence number of the packet. If continuity is missing then, the control unit passes the packet to the predictor algorithm which predicts the packet else it directly updates the packet to the Virtual Time Rendering Algorithm Another major issue is the delay jitter. On the source (server) side the intra time difference between two packets will be 1msec. But due to varying delay in the network the data packets arrive at the receiver with fluctuating intra time difference. In order to counter the delay jitter effect, we implement the Virtual Time Rendering algorithm which reads the time stamp value at which the packet was generated at the source and modifies the update time at the receiver side. In our work we do not control another device on the remote side, rather an applet which was developed using a Virtual Reality Markup Language in Matlab. Another challenge which is imposed when other multimedia is introduced with force feedback is the intra media synchronization. Real time video is captured from the applet side and given as feedback to the server side to improve the interactivity of the application. At every instant in time, different multimedia data produce data to be updated at the remote end. Since all the information are inter dependent with other media in time, efficient intra media synchronization is required. This thesis also focuses in providing an architecture which not only supports force feedback data but have a multiplexed model which allows an efficient transmission of all surgical information in real time. Each data occupies significant part of bandwidth in the network and the effect of multiplexing might affect the synchronization scheme of the force feedback device. Our architecture supports the efficient transmission of all types of multimedia information and also maintain the synchronization of the scheme. This method is unique with its methodical approach to support different multimedia information.

A quadrotor is an omnidirectional unmanned air vehicle that is suitable for indoor flight because of its ability to hover and maneuver in confined spaces. The remote operation of this type of vehicle is difficult due to a lack of sensory perception; typically, the view from the onboard camera is the only information transmitted to the pilot. This thesis proposes using force feedback exerted by the command input device on the hand of the pilot to assist in avoiding collisions while navigating in indoor environments. Five candidate algorithms are presented for calculating the forces to be felt by the pilot based on the quadrotor's position and velocity in the indoor environment. The candidates include a parametric algorithm based on the dynamics of the quadrotor, two time-to-impact algorithms, and two algorithms that employ virtual springs between the quadrotor and obstacles. A method of incorporating the position of the command input device to improve the usability and effectiveness of the algorithms is also presented. A framework for simulating the quadrotor dynamics, indoor environment, and force feedback algorithms is described. In the simulation, the pilot commands a simulated quadrotor, using a commercial haptic interface, as it flies in an indoor environment. The pilot receives force feedback cues as the quadrotor navigates around obstacles. Two methods of control were used for the simulation. In the first method, displacements of the haptic interface correspond to velocity commands to the quadrotor. In the second method, displacements of the input correspond to desired roll and pitch commands. Two user study experiments, one for each control method, were performed to compare the force feedback algorithms in simulation. The results of the velocity control experiment suggest that higher force levels help to avoid collisions and that the time to impact algorithm results in fewer collisions than having no force, but is not significantly better than the other algorithms. The results of the angle control experiment suggest that the time to impact algorithm is clearly the best in terms of hits and hit length and has no disadvantages compared to the other algorithms. Finally, to demonstrate the force feedback algorithms and software in a real-world environment, the system was interfaced with a physical quadrotor. The quadrotor system is described and the results of the tests are presented.

Indoor flight of unmanned aerial vehicles (UAVs) has many applications in environments in which it is undesirable or dangerous for humans to be, such as military reconnaissance or searching for trapped victims in a collapsed building. However, limited visual feedback makes it difficult to pilot UAVs in cluttered and enclosed spaces. Haptic feedback combined with visual feedback has shown to reduce the number of collisions of UAVs in indoor environments; however, it has increased the mental workload of the operator. This thesis investigates the potential of combining novel haptic and 3D audio feedback to provide additional information to operators of UAVs in order to improve performance and reduce workload. Many haptic feedback algorithms, such as Time to Impact (TTI)~cite{Brandt2009}, have been developed to help pilot UAVs. This thesis compares TTI with two new haptic feedback algorithms: Omni-Directional Dynamics Springs (ODDS) and Velocity Scaled Omni-Directional Dynamic Springs (VSODDS). These novel algorithms are based on the idea that dynamic springs are attached to the haptic controller in all directions. This thesis is unique by augmenting visual and haptic feedback with real-time 3D audio feedback. Continuous Directional Graded Threshold (CDGT) and Discrete Directional Graded Threshold (DDGT) are two novel algorithms that were developed to provide 3D audio warning cues to operators. To reduce sensory overload, these algorithms play a graded audio alert cue in the direction of velocity and when within a threshold distance of an obstacle. In order to measure operator workload, many researchers have used subjective measures, which suffer from subject bias, preconceptions, and ordering. Instead of using a subjective measure, experimental data is used to objectively measure operator workload using behavioral entropy, which works on the idea that humans work to reduce entropy by skilled behavior. QuadSim, a robust and versatile indoor quadrotor simulator, was developed as a test bed for visual, haptic, and 3D audio feedback. Using QuadSim, a human subject experiment was performed to determine the effectiveness of haptic and 3D audio feedback on operator performance and workload. The results of the study indicate that haptic feedback significantly reduced the number of collisions and collision length. Operator workload was decreased in the side-to-side direction by VSODDS but was adversely increased by TTI. Overall, VSODDS outperformed the other haptic algorithms. Unlike haptic feedback, audio feedback proved to be neither helpful nor harmful in improving performance or reducing workload.

No description available.

Research has shown that head-mounted displays can produce greater presence in a virtual environment than direct-view displays. It has also been shown that after vision, haptic response is one of the most important inputs for humans in a simulated environment. This research was designed primarily to determine the performance differences associated with different display types, levels of steering force feedback, and the interaction between these two factors in a low-to-medium fidelity, PC-based driving simulator. Participants drove on a simulated driving course during which both objective driving performance data were collected (lane deviation, speed control, steering wheel angle variance, and time to the complete course) as well as subjective self-report measures including questionnaires designed to tap immersive tendencies and perceived levels of presence. Results of the research show that the use of a head-mounted display can significantly impact driving performance in terms of speed control and lane deviation. Speed control was significantly improved (increased) and lane deviation was significantly improved (decreased) in three of the four roadway segments with the use of an HMD. Results for active steering force feedback, however, showed a significantly negative effect on driving performance with an increase in average lane deviation. Descriptive statistics showed that participants preferred the HMD and D-V equally and all but one participant preferred active steering force feedback. / Master of Science

A qualitative analysis of a rotary traveling wave-type ultrasonic motor (USM) used to supply feedback forces in force-feel systems is carried out. Prior to simulation, the subsystems and contact mechanics needed to define the motor's equations of motion are discussed along with the pitfalls of modeling a USM. A mathematical model is assembled and simulated in MATLAB Simulink. Accompanying the dynamic model, a new reduced model is presented from which predictions of USM performance can be made without a complicated dynamic model. Outputs from the reduced model are compared with those of the dynamic model to show the differences in the transient solution, agreement in the steady state solution, and above all that it is an efficient tool for approximating a motor's steady state response as a function of varying the motor parameters. In addition, the reduced model provides the means of exploring the USMs response to additive loading, loads acting in the direction of motor motion, where only resistive loads, those opposite to the motor rotation, had been considered previously. Fundamental differences between force-feel systems comprising standard DC brushless motors as the feedback actuators and the proposed system using the USM are explained by referencing the USM contact mechanics. Outputs from USM model simulations are explored, and methods by which the motor can be implemented in the force-feel system are derived and proven through simulation. The results show that USMs, while capable of providing feedback forces in feel systems, are far from ideal for the task. The speed and position of the motor can be controlled through varying stator excitation parameters, but the transient motor output torque cannot; it is solely a function of the motor load, whether additive or resistive. / Master of Science

Recent technology developments have made possible the creation of several commercial devices and a selected number of development platforms for the inclusion of haptics (the sense of touch) in virtual environments (VE). This thesis sought to investigate and develop a better understanding of whether or not haptics or sound substitutions improved manipulation performance or usability in VE applications. Twenty-four volunteers (12 males and 12 females) participated in a 2 (haptics) x 2 (sound) x 2 (gender) mixed factorial experiment in which they completed a VE manipulation task involving the assembly and disassembly of 5 interconnecting parts. Performance for the manipulation task was measured through completion time and the number of collisions made, as well as subjective measures of usability. Results indicated that completion times were slower and collision counts were higher for males with the addition of haptics (ptime = 0.03; pcollisions<0.05), while females exhibited a smaller increase in collision counts and no increase in completion time with the addition of haptics. Nonetheless, there were improved usability attributes when haptics were incorporated, more specifically, an increased sense of realism, perceived helpfulness and perceived utility in a design task (p<0.05 for all). Sound was found to be an effective substitute for haptics in most measures taken while the combination of sound and haptics versus either alone, did not demonstrate any signs of improving performance or any usability attributes. It is therefore recommended that sound substitution be used in VE manipulation tasks where the extra haptic information is desired, and minimizing completion time or collisions are the overall goal. Finally, for the utility of the feedback towards a design task, users ranked haptics as being more useful than sound, but ranked the combination of sound and haptics as being the best feedback condition (p<0.05). Further research is required to determine whether this belief is consistent with objective measures. / Master of Science

Depuis son invention en 1986, les microscopes à force atomique (AFM) ont été des puissants outils pour la caractérisation des matériaux et des propriétés des matériaux à l'échelle nanométrique. Cette thèse est entièrement dédiée à la mesure de l'interaction entre une sonde AFM et une surface avec une nouvelle technique AFM appelée Force Feedback Microscopy (FFM). La technique a été développée et utilisée pour l'étude d'échantillons biologiques. Le principe central de la technologie FFM est que la force totale moyenne appliquée à la pointe est égal à zéro. En conséquence, en présence d'une interaction pointe-échantillon, une force égale et contraire doit être appliqué à la pointe par une boucle de rétroaction . La force de réaction est ici appliquée à la pointe à travers le déplacement d'un petit élément piézoélectrique positionné à la base du levier AFM. La boucle de rétroaction permet d'éviter instabilités mécaniques tels que le saut au contact, permettant la mesure complète de la courbe d'interaction. En plus, il donne la possibilité de mesurer simultanément les parties élastique et inélastique de l'interaction.La technique a été appliquée à l'étude des interactions à l'interface solide/gaz, avec un intérêt particulier pour l'observation de la formation et de la rupture des ponts capillaires entre pointe et échantillon. Ensuite, on a focalisé notre attention aux interfaces solide/liquide. Dans ce contexte, courbes complètes de type DLVO sont caractérisées d'un point de vue élastique et dissipatif.Nous avons développé des nouveaux modes d'imagerie AFM pour l'étude des biomolécules. Images de phospholipides et de l'ADN à force constante ont été réalisées et certaines propriétés mécaniques comme le module de Young des échantillons ont été évaluées. En plus, nous avons réalisé une étude spectroscopique de l'élasticité et du coefficient d'amortissement de l'interaction entre des cellules vivantes de type PC-12 et une pointe AFM en silicium. L'étude montre que le FFM est un instrument capable de mesurer l'interaction à des fréquences qui ne sont pas nécessairement liées aux résonances caractéristiques du levier. L'étude spectroscopique pourrait avoir dans le futur des applications importantes pour l'étude des bio-molécules et des polymères. / Since its invention in 1986, the atomic force microscopes (AFMs) have been powerful tools for the characterization of materials and material properties at the nanoscale. The present thesis focuses on the measurement of the interaction between an AFM probe and a surface. A new AFM technique called Force Feedback Microscopy (FFM) has been developed and applied to the study of biological specimens. The central principle of the FFM is that the average total force acting on the tip is maintained equal to zero. It means that, in presence of a tip-sample interaction, a counteracting force has to be applied to the tip by a feedback loop. We apply a counteracting force to the tip by displacing the cantilever base with a small piezoelectric element. The feedback loop avoids mechanical instabilities such as jump to contact allowing the complete measurement of the interaction force. Moreover it is possible to simultaneously measure the elastic and inelasticcomponents of the interaction.The technique has been applied to the study of interactions at the solid/gas interface with a particular interest to the observation of the nucleation and rupture of capillary condensates between the tip and the sample. At the solid/liquid interface, complete DLVO force curves are characterized elastically and inelastically.We developed new AFM imaging modes for the study of biomolecules. Images of phospholipids and DNA at constant force have been acquired and the mechanical Young modulus of the samples has been evaluated when possible. In addition, a spectroscopic study of the elasticity and the damping factor of the interaction between living cells and the tip has been carried out. The study reveals that the FFM is an instrument capable of measuring the interaction at frequencies which are not necessarily linked to the cantilever eigenmodes. The spectroscopy study could have in the future important applications on the study of biomolecules and polymers.

Teleoperation of an unmanned vehicle is a challenging task for human operators especially when the vehicle is out of line of sight. Improperly designed and applied display interfaces directly affect the operation performance negatively and even can result in catastrophic failures. If these teleoperation missions are human-critical then it becomes more important to improve the operator performance by decreasing workload, managing stress and improving situational awareness. This research aims to develop electrical and control system of an unmanned ground vehicle (UGV) using an All-Terrain Vehicle (ATV) and validate the development with investigation of the effects of force feedback devices on the teleoperation performance. After development, teleoperation tests are performed to verify that force feedback generated from the dynamic obstacle information of the environment improves teleoperation performance. Results confirm this statement and the developed UGV is verified for future research studies. Development of UGV, algorithms and real system tests are included in this thesis.

Humans rely on all their senses when interacting with others in order to communicate and collaborate efficiently. In mediated interaction the communication channel is more or less constrained, and humans have to cope with the fact that they cannot get all the information that they get in face-to-face interaction. The particular concern in this thesis is how humans are affected by different multimodal interfaces when they are collaborating with another person in a shared virtual environment. One aspect considered is how different modalities affect social presence, i.e. people’s ability to perceive the other person’s intentions and emotions. Another aspect investigated is how different modalities affect people’s notion of being present in a virtual environment that feels realistic and meaningful. Finally, this thesis attempts to understand how human behavior and efficiency in task performance are affected when using different modalities for collaboration. In the experiment presented in articles A and B, a shared virtual environment that provided touch feedback was used, making it possible to feel the shape, weight and softness of objects as well as collisions between objects and forces produced by another person. The effects of touch feedback on people’s task performance, perceived social presence, perceived presence and perceived task performance were investigated in tasks where people manipulated objects together. Voice communication was possible during the collaboration. Touch feedback improved task performance significantly, making it both faster and more precise. People reported significantly higher levels of presence and perceived performance, but no difference was found in the perceived social presence between the visual only condition and the condition with touch feedback. In article C an experiment is presented, where people performed a decision making task in a collaborative virtual environment (CVE) using avatar representations. They communicated either by text-chat, a telephone connection or a video conference system when collaborating in the CVE. Both perceived social presence and perceived presence were significantly lower in the CVE text-chat condition than in the CVE telephone and CVE video conference conditions. The number of words and the tempo in the dialogue as well as the task completion time differed significantly for persons that collaborated using CVE text-chat compared to those that used a telephone or a video conference in the CVE. The tempo in the dialogue was also found to be significantly higher when people communicated using a telephone compared to a video conference system in CVEs. In a follow-up experiment people performed the same task using a website instead, with no avatar but with the same information content as before. Subjects communicated either by telephone or a video conference iv system. Results from the follow-up experiment showed that people that used a telephone completed tasks significantly faster than those that used a video conference system, and that the tempo in the dialogue was significantly higher in the web environments than in the CVEs. Handing over objects is a common event during collaboration in face-to face interaction. In the experiment presented in article D and E, the effects of providing touch feedback was investigated in a shared virtual environment in which subjects passed a series of cubic objects to each other and tapped them at target areas. Subjects could not communicate verbally during the experiment. The framework of Fitts’ law was applied and it was hypothesized that object hand off constituted a collaboratively performed Fitts’ law task, with target distance to target size ratio as a fundamental performance determinant. Results showed that task completion time indeed linearly increased with Fitts’ index of difficulty, both with and without touch feedback. The error rate was significantly lower in the condition with touch feedback than in the condition with only visual feedback. It was also found that touch feedback significantly increased people’s perceived presence, social presence and perceived performance in the virtual environment. The results presented in article A and E analyzed together, suggest that when voice communication is provided the effect of touch feedback on social presence might be overshadowed. However, when verbal communication is not possible, touch proves to be important for social presence. / QC 20100630