This dissertation describes the wave particles technique for simulating water surface waves and two way fluid-object interactions for real-time applications, such as video games. Water exists in various different forms in our environment and it is important to develop necessary technologies to be able to incorporate all these forms in real-time virtual environments. Handling the behavior of large bodies of water, such as an ocean, lake, or pool, has been computationally expensive with traditional techniques even for offline graphics applications, because of the high resolution requirements of these simulations. A significant portion of water behavior for large bodies of water is the surface wave phenomenon. This dissertation discusses how water surface waves can be simulated efficiently and effectively at real-time frame rates using a simple particle system that we call "wave particles." This approach offers a simple, fast, and unconditionally stable solution to wave simulation. Unlike traditional techniques that try to simulate the water body (or its surface) as a whole with numerical techniques, wave particles merely track the deviations of the surface due to waves forming an analytical solution. This allows simulation of seemingly infinite water surfaces, like an open ocean. Both the theory and implementation of wave particles are discussed in great detail. Two-way interactions of floating objects with water is explained, including generation of waves due to object interaction and proper simulation of the effect of water on the object motion. Timing studies show that the method is scalable, allowing simulation of wave interaction with several hundreds of objects at real-time rates.

The human hands are one of the most complex organs of the human body. As they enable us to grasp various objects in many different ways, they have played a crucial role in the rise of the human race. Being able to control hands as a human do is a key step towards friendly human-robots interaction and realistic virtual human simulations. Grasp generation has been mostly studied for the purpose of generating physically stable grasps. This paper addresses a different aspect: how to generate realistic, natural looking grasps that are similar to human grasps. To simplify the problem, the wrist position is assumed to be known and only the finger pose is generated. As the realism of a grasp is not easy to put into equations, data-driven machine learning techniques are used. This paper investigated the application of the deep neural networks to the grasp generation problems. Two different object shape representations (point cloud and multi-view depth images), and multiple network architectures are experimented, using a collected human grasping dataset in a virtual reality environment. The resulting generated grasps are highly realistic and human-like. Though there are sometimes some finger penetrations on the object surface, the general poses of the fingers around the grasped objects are similar to the collected human data. The good performance extends to the objects of categories previously unseen by the network. This work has validated the efficiency of a data-driven deep learning approach for human-like grasp synthesis. I believe the realistic-looking objective of the grasp synthesis investigated in this thesis can be combined with the existing mechanical, stable grasp criteria to achieve both natural-looking and reliable grasp generations. / Den mänskliga handen är en av de mest komplexa organen i människokroppen. Eftersom våra händer gör det möjligt för oss att hantera olika föremål på många olika sätt, har de spelat en avgörande roll i människans utveckling. Att kunna styra händer är ett viktigt steg mot interaktion mellan människor och robotar, samt skapa realistiska simuleringar av virtuella människor. Virtualla handgrepp har tidigare mest studerats för att generera fysiskt stabila grepp. I detta papper behandlas en annan aspekt: hur man genererar realistiska grepp som liknar en människas grepp. För att förenkla problemet antas att handledspositionen är känd och endast fingerkonfigurationen genereras. Eftersom hur realistiskt ett grepp är inte är lätt att beskriva i ekvationer, används istället data-driven maskininlärningsteknik. Detta papper undersöker tillämpningen av djupa neurala nätverken (Deep Neural Networks) för att generera grepp. Två olika representationer av former i 3D (punktmoln och bilder med djupinformation) och flera alternativa nätverksarkitekturer utvärderas med hjälp av en databas av mänskliga grepp samlad i en virtuell verklighetsmiljö. De resulterande genererade greppen är mycket realistiska och mänskliga. Även om det ibland förekommer något finger som penetrerar objektet, liknar den allmänna positioneringen av fingrarna den insamlade mänskliga datan. Denna goda prestanda gäller även för föremål i kategorier som aldrig tidigare setts av nätverket. I arbetet valideras också effektiviteten av ett data-drivet tillvägagångssätt baserat på djupa neurala nätverk för människoliknande syntes av grepp

Interacting with a Virtual Environment (VE) generally requires the user to correctly perceive the relative position and orientation of virtual objects. For applications requiring interaction in personal space, the user may also need to accurately judge the position of the virtual object relative to that of a real object, for example, a virtual button and the user's real hand. This is difficult since VEs generally only provide a subset of the cues experienced in the real world. Complicating matters further, VEs presented by currently available visual displays may be inaccurate or distorted due to technological limitations. Fundamental physiological and psychological aspects of vision as they pertain to the task of object manipulation were thoroughly reviewed. Other sensory modalities--proprioception, haptics, and audition--and their cross-interactions with each other and with vision are briefly discussed. Visual display technologies, the primary component of any VE, were canvassed and compared. Current applications and research were gathered and categorized by different VE types and object interaction techniques. While object interaction research abounds in the literature, pockets of research gaps remain. Direct, dexterous, manual interaction with virtual objects in Mixed Reality (MR), where the real, seen hand accurately and effectively interacts with virtual objects, has not yet been fully quantified. An experimental test bed was designed to provide the highest accuracy attainable for salient visual cues in personal space. Optical alignment and user calibration were carefully performed. The test bed accommodated the full continuum of VE types and sensory modalities for comprehensive comparison studies. Experimental designs included two sets, each measuring depth perception and object interaction. The first set addressed the extreme end points of the Reality-Virtuality (R-V) continuum--Immersive Virtual Environment (IVE) and Reality Environment (RE). This validated, linked, and extended several previous research findings, using one common test bed and participant pool. The results provided a proven method and solid reference points for further research. The second set of experiments leveraged the first to explore the full R-V spectrum and included additional, relevant sensory modalities. It consisted of two full-factorial experiments providing for rich data and key insights into the effect of each type of environment and each modality on accuracy and timeliness of virtual object interaction. The empirical results clearly showed that mean depth perception error in personal space was less than four millimeters whether the stimuli presented were real, virtual, or mixed. Likewise, mean error for the simple task of pushing a button was less than four millimeters whether the button was real or virtual. Mean task completion time was less than one second. Key to the high accuracy and quick task performance time observed was the correct presentation of the visual cues, including occlusion, stereoscopy, accommodation, and convergence. With performance results already near optimal level with accurate visual cues presented, adding proprioception, audio, and haptic cues did not significantly improve performance. Recommendations for future research include enhancement of the visual display and further experiments with more complex tasks and additional control variables.

As software systems become larger and more complex, developers require the ability to model abstract concepts while ensuring consistency across the entire project. The internet has changed the nature of software by increasing the desire for software deployment across multiple distributed platforms. Finally, increased dependence on technology requires assurance that designed software will perform its intended function. This thesis introduces the Formal Object Interaction Language (FOIL). FOIL is a new object-oriented modeling language specifically designed to address the cumulative shortcomings of existing modeling techniques. FOIL graphically displays software structure, sequential and concurrent behavior, process, and interaction in a simple unified notation, and has an algebraic representation based on a derivative of the π-calculus. The thesis documents the technique in which FOIL software models can be mathematically verified to anticipate deadlocks, ensure consistency, and determine object state reachability. Scalability is offered through the concept of behavioral inheritance; and, FOIL’s inherent support for modeling concurrent behavior and all known workflow patterns is demonstrated. The concepts of process achievability, process complete achievability, and process determinism are introduced with an algorithm for simulating the execution of a FOIL object model using a FOIL process model. Finally, a technique for using a FOIL process model as a constraint on FOIL object system execution is offered as a method to ensure that object-oriented systems modeled in FOIL will complete their processes based activities. FOIL’s capabilities are compared and contrasted with an extensive array of current software modeling techniques. FOIL is ideally suited for data-aware, behavior based systems such as interactive or process management software.

Les travaux décrits dans ce document abordent le problème du couplage dynamique homme-système haptique. Nous proposons une étude de ce couplage basée sur l'hypothèse d'un système hybride temporaire. Selon cette hypothèse, le système formé lors du couplage peut être considéré comme un système dynamique dont les deux parties ne peuvent pas être séparées. Ce sujet est pluridisciplinaire, se situant à l'intersection des sciences cognitives, de l'automatique et de l'haptique. La première partie du document comporte un état de l'art sur l'analyse du couplage dans ces trois domaines, une description de la problématique et de la méthode à utiliser pour notre étude ainsi qu'une proposition des typologies du geste. Lors de cette étude du couplage, nous nous intéressons à un groupe de gestes particuliers, notamment le geste périodique et le geste passif dans une situation de simulation haptique ainsi qu'aux modèles d'interaction capables de les générer. La méthode générale, consiste à définir des approches pour la modélisation du couplage main-système haptique pour ensuite réaliser une analyse du système couplé à partir d'une acquisition des données du système lors du couplage et en utilisant des méthodes d'identification de paramètres issus de l'automatique pour caractériser les modèles. La dernière partie, décrit la mise en place du dispositif pour l'analyse expérimentale du couplage en situation de simulation avec une interaction haptique. Ce dispositif permet l'acquisition des données du geste pour l'analyse. Nous présentons également, l'étude réalisée sur le simulateur haptique afin d'établir l'équivalence entre les paramètres virtuels introduits et issus du simulateur et des paramètres physiques réels. Ensuite, nous décrivons l'analyse expérimentale des différentes situations de couplage proposées. Les expériences effectuées lors de cette étude ont été réalisées sur la plateforme temps réel ERGON_X, conçue par l'ACROE/ICA. Les résultats de ces expériences ont permis de quantifier les modèles du geste et d'observer ses composantes, selon les modèles établis. Mots clés : haptique, interface haptique, interfaces homme-machine, simulation temps réel, couplage homme-objet, geste, modélisation physique, identification de paramètres. / The work described in this document deals with the problem of human-haptic system dynamic coupling. We propose a study of this kind of coupling based on the hypothesis of a temporary hybrid system. Under this hypothesis, the system formed during the coupling can be considered as a dynamic system in which the two parties that compose it cannot be separated. This is multidisciplinary topic, situated at the intersection of cognitive science, automation and haptics. The first part of the document includes a state of the art on the analysis of coupling in these three areas, the description of the problem and the methodology for the study as well as a proposal of gesture typology. In this study of coupling, we are interested in a particular group of actions, such as periodic movement and passive gesture in a situation of haptic simulation and also, in the interaction models able to generate them. The general method is to define the approaches for modeling the hand-haptic device coupling and then perform an analysis of the coupled system by acquiring system data during the coupling and using parameter identification methods to characterize the models. The final section describes the implementation of the device for the experimental analysis of coupling during simulation with a haptic interaction. This device allows data acquisition for gesture analysis. We also present the study of the haptic simulator to establish the equivalence between virtual parameters introduced to and returned by the simulator and real physical parameters. Then, we describe the experimental analysis of different proposed coupling situations. The experiments performed for this study were performed using the real-time platform ERGON_X, designed by ACROE / ICA. The results of these experiments were used to quantify gesture models and to observe its components, according to established models. Keywords: haptic, haptic interface, human-machine interfaces, real-time simulation, human-object coupling, gesture, physical modeling, parameter identification.

No description available.

Die automatische Erkennung von Mensch-Objekt-Interaktionen (HOI) spielt eine zentrale Rolle in der Mensch-Computer-Interaktion, der Verhaltenanalyse und zahlreichen KI-Anwendungen. Während aktuelle HOI-Methoden überwiegend auf rektilinearen Bildern trainiert wurden, gewinnen omnidirektionale Kameras mit Fischaugenobjektiven zunehmend an Bedeutung, da sie mit ihrem extrem weiten Sichtfeld deutlich größere Räume abdecken können, als hermkömmliche Kameras. Allerdings führen die starken optischen Verzerrungen dieser Objektive dazu, dass herkömmliche Computer-Vision-Algorithmen oft unzuverlässige Ergebnisse liefern. Diese Arbeit untersucht, inwiefern bestehende Methoden zur Erkennung von Mensch-Objekt-Interaktionen an die besonderen Eigenschaften von Fischaugenaufnahmen angepasst werden können. Durch den Einsatz von Transferlernen wurde ein bestehendes HOI-Modell auf zwei speziell erstellte Datensätze trainiert. Zudem wurde eine Annotationssoftware entwickelt, welches eine effiziente Beschriftung von Mensch-Objekt-Interaktionen ermöglicht und der Forschungsgemeinschaft zur Verfügung steht. Die experimentellen Ergebnisse zeigen, dass HOI-Modelle durch Transferlernen erfolgreich an die Verzerrungen omnidirektionaler Bilder angepasst werden können, wodurch eine Erkennungsgenauigkeit von bis zu 85% erreicht wurde.:1 Einleitung 1.1 Problemstellung 1.2 Zielsetzung 1.3 Forschungsfragen 2 Grundlagen 2.1 Mensch-Objekt-Interaktion 2.2 Verfahren für die HOI-Erkennung 2.2.1 Zweistufige Verfahren 2.2.2 Einstufige Verfahren 2.2.3 End-To-End Verfahren 2.2.4 Übersicht 2.3 Datensätze 2.3.1 HICO-DET 2.3.2 V-COCO 2.3.3 360Action 2.4 Augmentierungsmethoden 2.5 Faltungsnetze 2.6 Transferlernen 3 Verwandte Arbeiten 3.1 HOI-Erkennung in Echtzeit 3.2 HOI-Erkennung in 360° Aufnahmen 3.3 Transferlernen auf Fischaugenbildern 4 Auswahl des Modells 4.1 HoiTransformer 4.1.1 Backbone 4.1.2 Encoder 4.1.3 Decoder 4.1.4 Prediction Head 5 Datenerzeugung 5.1 Fischaugen-HOI-Datensatz 5.2 Datenaugmentierung 5.3 Annotationssoftware 5.3.1 HOI-Det-UI 5.3.2 HOI Labeling Tool 6 Methode 6.1 Daten 6.2 Evaluierungsmetrik 6.3 Trainingsaufbau 7 Ergebnisse und Auswertung 7.1 Quantitative Ergebnisse 7.2 Qualitative Ergebnisse 8 Schluss 8.1 Fazit 8.2 Ausblick / Human-object interaction (HOI) detection plays an important role in human-computer interaction, action analysis, and a wide range of AI-driven applications. While current HOI methods are mainly trained on perspective images, omnidirectional cameras are gaining importance due to their ultra-wide field of view, enabling coverage of significantly larger spaces compared to conventional cameras. However, the optical distortions inherent to fisheye lenses often cause traditional computer vision algorithms to produce unreliable results. This thesis explores how existing HOI detection methods can be adapted to address the unique challenges of fisheye images. Using the method of transfer learning, an established HOI model was fine-tuned on two custom datasets. Additionally, an annotation tool was developed for the labeling of HOI triplets, which has been made publicly available to support the research community. Experimental results demonstrate that HOI models can be successfully adapted to handle distortions in omnidirectional images through transfer learning, achieving a detection accuracy of up to 85%. This work highlights the feasibility of bridging the gap between conventional HOI frameworks and the demands of fisheye-based vision systems.:1 Einleitung 1.1 Problemstellung 1.2 Zielsetzung 1.3 Forschungsfragen 2 Grundlagen 2.1 Mensch-Objekt-Interaktion 2.2 Verfahren für die HOI-Erkennung 2.2.1 Zweistufige Verfahren 2.2.2 Einstufige Verfahren 2.2.3 End-To-End Verfahren 2.2.4 Übersicht 2.3 Datensätze 2.3.1 HICO-DET 2.3.2 V-COCO 2.3.3 360Action 2.4 Augmentierungsmethoden 2.5 Faltungsnetze 2.6 Transferlernen 3 Verwandte Arbeiten 3.1 HOI-Erkennung in Echtzeit 3.2 HOI-Erkennung in 360° Aufnahmen 3.3 Transferlernen auf Fischaugenbildern 4 Auswahl des Modells 4.1 HoiTransformer 4.1.1 Backbone 4.1.2 Encoder 4.1.3 Decoder 4.1.4 Prediction Head 5 Datenerzeugung 5.1 Fischaugen-HOI-Datensatz 5.2 Datenaugmentierung 5.3 Annotationssoftware 5.3.1 HOI-Det-UI 5.3.2 HOI Labeling Tool 6 Methode 6.1 Daten 6.2 Evaluierungsmetrik 6.3 Trainingsaufbau 7 Ergebnisse und Auswertung 7.1 Quantitative Ergebnisse 7.2 Qualitative Ergebnisse 8 Schluss 8.1 Fazit 8.2 Ausblick

Recent advances in camera technology have made it possible to build a comfortable, wearable system which can capture the scene in front of the user throughout the day. Products based on this technology, such as GoPro and Google Glass, have generated substantial interest. In this thesis, I present my work on egocentric vision, which leverages wearable camera technology and provides a new line of attack on classical computer vision problems such as object categorization and activity recognition. The dominant paradigm for object and activity recognition over the last decade has been based on using the web. In this paradigm, in order to learn a model for an object category like coffee jar, various images of that object type are fetched from the web (e.g. through Google image search), features are extracted and then classifiers are learned. This paradigm has led to great advances in the field and has produced state-of-the-art results for object recognition. However, it has two main shortcomings: a) objects on the web appear in isolation and they miss the context of daily usage; and b) web data does not represent what we see every day. In this thesis, I demonstrate that egocentric vision can address these limitations as an alternative paradigm. I will demonstrate that contextual cues and the actions of a user can be exploited in an egocentric vision system to learn models of objects under very weak supervision. In addition, I will show that measurements of a subject's gaze during object manipulation tasks can provide novel feature representations to support activity recognition. Moving beyond surface-level categorization, I will showcase a method for automatically discovering object state changes during actions, and an approach to building descriptive models of social interactions between groups of individuals. These new capabilities for egocentric video analysis will enable new applications in life logging, elder care, human-robot interaction, developmental screening, augmented reality and social media.