L’objectif de cette thèse est d’étudier l’utilisation d’Interfaces Cerveau-Ordinateur (ICOs) au sein de la Réalité Virtuelle (RV) et de la Réalité Augmentée (RA). Notre but est d’évaluer la compatibilité entre les systèmes basés sur une ICO et la RV/RA, de concevoir de nouveaux outils pour la visualisation de l’activité cérébrale basée sur la RV/RA, et finalement de proposer de nouveaux usages pour les ICOs, plus particulièrement en combinaison avec des vêtements intelligents. Afin de réaliser ces objectifs, nous avons tout d’abord réalisé une étude de faisabilité concernant l’association entre une ICO et la RV. Notre objectif était d’étudier l’influence de l’activité motrice sur une ICO. Nous avons conçu un système similaire à un jeu vidéo, servant comme support à une étude utilisateur montrant que l’ICO peut être utilisée avec succès, même lorsque les participants exécutent une activité musculaire exigeante. Dans un second temps, nous avons également proposé des outils de visualisation de l’activité cérébrale basés sur la RV/RA. Notre premier système nommé «Mind-Mirror» superpose un cerveau virtuel représentant l’activité cérébrale d’un utilisateur à l’image de celui-ci dans un miroir. Une étude utilisateur a montré qu’aucune perte significative de performance de l’ICO n’a été constatée, même avec une complexité additionnelle due à l’affichage basé sur la RA. Notre seconde contribution se nomme «Mind-Window» et étend les possibilités du Mind-Mirror en permettant plusieurs points de vue sur un même enregistrement d’activité cérébrale en utilisant des tablettes. Notre dernière contribution se nomme «Mind-Inside» et permet aux utilisateurs de visualiser en RV leur activité cérébrale en temps réel tout en étant immergés dans un cerveau virtuel. Enfin, nous avons étudié comment les ICOs et la RV/RA peuvent être appliquées au domaine des vêtements intelligents. Nous avons mis en place une plateforme d’expérimentation consistant en une cabine d’essayage virtuelle intégrant une ICO et permettant aux utilisateurs de porter des vêtements intelligents virtuels en RA. Poursuivant ces travaux, nous avons également conçu une «cape d’invisibilité» inspirée par l’univers de Harry Potter. Cette cape virtuelle permet aux utilisateurs de se camoufler en RA en utilisant leur état mental. Une étude utilisateur sur le contrôle de l’effet a mis en avant l’amélioration de l’expérience utilisateur et «l’impression d’avoir un superpouvoir». / The objective of this PhD thesis is to study the use of Brain- Computer Interfaces (BCIs) within Virtual Reality (VR) and Augmented Reality (AR). Our goal is to evaluate the compatibility between systems based on a BCI and VR/AR, to design new tools for the visualization of the brain activity based on VR/AR, and finally to propose new uses for the BCIs, and especially in combination with smart clothes. In order to fulfil these objectives, we have first designed and performed a feasibility study concerning the combination of a BCI and VR. Our objective was to study the influence of motor activity on a BCI. We have designed a system similar to a video game, serving as a base for a user study showing that this BCI can be successfully used, even when participants are performing a demanding muscular activity. We have also proposed three visualization tools for the brain activity based on VR/AR. Our first system called the «Mind- Mirror» which enables the visualization of our own brain activity «inside our own head» by superimposition. A user study has shown that no significant drop in BCI performance occurred, even with the additional complexity due to our AR-based display. Our second contribution is called «Mind- Window» and extends the Mind-Mirror’s possibilities by enabling one or multiple users to visualize the brain activity of another person as if her skull was transparent. Our last contribution is called «Mind-Inside» and allows users to visualize their brain activity in real-time while being immersed in a virtual brain. Finally, we have studied how BCIs and the VR/AR can be applied to smart clothes. We have designed an experimental platform comprising a dressing room integrating a BCI. Following this work, we proposed an «invisibility cloak» inspired by the Harry Potter universe. This virtual cloak allows users to camouflage themselves in AR using their mental state. Results from a preliminary study based on a simple videogame inspired by the Harry Potter universe could notably show that, compared to a standard control made with a keyboard, controlling the optical camouflage directly with the BCI could enhance the user experience and the feeling of «having a super-power».

Human-computer interaction (HCI) is being revolutionized by computational design and artificial intelligence. As the diversity of user interfaces shifts from personal desktops to mobile and wearable devices, yesterday’s tools and interfaces are insufficient to meet the demands of tomorrow’s devices. This dissertation describes my research on leveraging different physical channels (e.g., vibration, light, capacitance) to enable novel interaction opportunities. We first introduce FontCode, an information embedding technique for text documents. Given a text document with specific fonts, our method can embed user-specified information (e.g., URLs, meta data, etc) in the text by perturbing the glyphs of text characters while preserving the text content. The embedded information can later be retrieved using a smartphone in real time. Then, we present Vidgets, a family of mechanical widgets, specifically push buttons and rotary knobs that augment mobile devices with tangible user interfaces. When these widgets are attached to a mobile device and a user interacts with them, the nonlinear mechanical response of the widgets shifts the device slightly and quickly. Subsequently, this subtle motion can be detected by the Inertial Measurement Units (IMUs), which is commonly installed on mobile devices. Next, we propose BackTrack, a trackpad placed on the back of a smartphone to track finegrained finger motions. Our system has a small form factor, with all the circuits encapsulated in a thin layer attached to a phone case. It can be used with any off-the-shelf smartphone, requiring no power supply or modification of the operating systems. BackTrack simply extends the finger tracking area of the front screen, without interrupting the use of the front screen. Lastly, we demonstrate MoiréBoard, a new camera tracking method that leverages a seemingly irrelevant visual phenomenon, the moiré effect. Based on a systematic analysis of the moiré effect under camera projection, MoiréBoard requires no power nor camera calibration. It can easily be made at a low cost (e.g., through 3D printing) and ready to use with any stock mobile device with a camera. Its tracking algorithm is computationally efficient and can run at a high frame rate. It is not only simple to implement, but also tracks devices at a high accuracy, comparable to the state-of-the-art commercial VR tracking systems.

Studies of how users experience Virtual Reality (VR) have thus far failed to address the extent to which rendering resolution and rendering frame rate affect users’ sense of immersion in VR, including applications of VR involving simulators, treatments for psychological and mental disorders, explorations of new and nonexistent structures, and ways to better understand the human body in medical applications. This study investigated if rendering resolution affected users’ sense of immersion in VR. This was conducted by comparing the responses of two groups, relative to two measures of participant immersion: (a) participant’s sense of presence and (b) participant’s sense of embodiment. The treatment levels were (a) low 512 pixels per inch (ppi) and (b) high 2048 ppi rendering resolution. One potential moderating variable, game type, varied over three levels: narrative, objective, and situational. The participants were randomly assigned to a treatment level account for previous VR experience, neither participants nor the research observer knew the treatment level. Measurements were collected after each game via an Immersion tendency Questionnaire after each game. For each dependent measure, sample descriptive statistics—mean (M) and inter-quartile range (IQR) with a conventional significance level of 0.05—were evaluated to conclude the results. Data indicated that the rendering resolution did not affect user immersion, but the game type did affect immersion and the situational game type was determined to be significantly more immersive than the other game types.

Event-related brain potentials (ERPs) in electroencephalography are manifestations at the scalp of neural activity that is triggered by, and is involved in, the processing of specific events. This chapter focuses on braincomputer interfaces (BCIs) that use P300, an endogenous ERP component. The P300 is a positive potential that occurs over central-parietal scalp 250- 700 msec after a rare event occurs in the context of the oddball paradigm. This paradigm has three essential attributes: a subject is presented with a series of events (i.e., stimuli), each of which falls into one of two classes; the events that fall into one of the classes are less frequent than those that fall into the other class; and the subject performs a task that requires classifying each event into one of the two classes.

Computer-generated imagery is ubiquitous in today's society, appearing in advertisements, video games, and computer-animated movies among other places. Much of this imagery needs to be as realistic as possible, and animators have turned to techniques such as fluid simulation to create scenes involving substances like smoke, fire, and water. The Lattice-Boltzmann Method (LBM) is one fluid simulation technique that has gained recent popularity due to its relatively simple basic algorithm and the ease with which it can be distributed across multiple processors. Unfortunately, current LBM simulations also suffer from high memory usage and restrict free surface fluids to domains of fixed size. This thesis modifies the LBM to utilize a recursive run-length-encoded (RLE) grid data structure instead of the standard fixed array of grid cells, which reduces the amount of memory required for LBM simulations as well as allowing the domain to grow and shrink as necessary to accomodate a liquid surface. The modified LBM is implemented within the open-source 3D animation package Blender and compared to Blender's current LBM simulator using the metrics of memory usage and time required to complete a given simulation. Results show that, although the RLE-based simulator can take several times longer than the current simulator to complete a given simulation, the memory usage is significantly reduced, making an RLE-based simulation preferable in a few specific circumstances.

Before there were airplanes, cars, trains, boats, or bicycles, the primary means of transportation was on foot. Unfortunately, many of the trails used by ancient travelers have long since been abandoned. We present a software tool which can help visualize and predict where these forgotten trails might lie through the use of a human-centered cost metric. By comparing the paths generated by our software with known historical trails, we demonstrate how the tool can indicate likely trails used by ancient travelers. In addition, this new tool provides novel visualizations to better help the user understand alternate paths, effect of terrain, and nearby areas of interest. Such a tool could be used by archaeologists and historians to better visualize and understand the terrain and paths around sites of historical interest. This thesis is a continuation of previous work, with emphasis on the ability to generate paths which traverse several thousand kilometers. To accomplish this, various graph simplification and path approximation algorithms are explored to construct a real-time path finding algorithm. To this end, we show that it is possible to restrict the search space for a path finding algorithm while not disrupting accuracy. Combined with a multi-threaded variant of Dijkstra's shortest path algorithm, we present a tool capable of traversing the contiguous US, a dataset containing over 19 billion datapoints, in under three hours on a 2.5 Ghz dual core processor. The tool is demonstrated on several examples which show the potential archaeological and historical applicability, and provide avenues for future improvements.

Hardware/Software (HW/SW) interfaces are pervasive in computer systems. However, many HW/SW interface implementations are unreliable due to their intrinsically complicated nature. In industrial settings, there are three major challenges to improving reliability. First, as there is no systematic framework for HW/SW interface specifications, interface protocols cannot be precisely conveyed to engineers. Second, as there is no unifying formal model for representing the implementation semantics of HW/SW interfaces accurately, some critical properties cannot be formally verified on HW/SW interface implementations. Finally, few automatic tools exist to help engineers in HW/SW interface development. In this dissertation, we present an automata-theoretic approach to HW/SW co-verification that addresses these challenges. We designed a co-specification framework to formally specify HW/SW interface protocols; we synthesized a hybrid Büchi Automaton Pushdown System, namely Büchi Pushdown System (BPDS), as the unifying formal model for HW/SW interfaces; and we created a co-verification tool, CoVer that implements our model checking algorithms and realizes our reduction algorithms for BPDS. The application of our approach to the Windows device/driver framework has resulted in the detection of fifteen specification issues. Furthermore, utilizing CoVer, we discovered twelve real bugs in five drivers. These non-trivial findings have demonstrated the significance of our approach in industrial applications.

Networked virtual reality environments including virtual worlds devoted to entertainment, online socializing and remote collaboration have grown in popularity with the rise of commercially available consumer graphics hardware and the growing ubiquity of the Internet. These virtual worlds are typified by a persistent simulated three-dimensional space that communicates over a computer network, where users interact with the environment and each other through digital avatars. Development of these virtual worlds challenges the limits of the networking infrastructure, 3D streaming graphics techniques, and the distributed computing design of the virtual world systems that manages the simulation. In this dissertation, we explore solutions to different aspects of the overall problem of developing a general purpose, networked virtual environment, focusing on the networking and software system issues. Specifically, we show how to improve the networking infrastructure to better support the high packet-rate traffic that is typical of virtual worlds, efficiently stream terrain data for remote rendering, and construct a dynamically adaptive distributed systems framework suitable for virtual world simulations.

No description available.

Objective: To assess the reliability and usefulness of an EEG-based brain-computer interface (BCI) for patients with advanced amyotrophic lateral sclerosis (ALS) who used it independently at home for up to 18 months. Methods: Of 42 patients consented, 39 (93%) met the study criteria, and 37 (88%) were assessed for use of the Wadsworth BCI. Nine (21%) could not use the BCI. Of the other 28, 27 (men, age 28-79 years) (64%) had the BCI placed in their homes, and they and their caregivers were trained to use it. Use data were collected by Internet. Periodic visits evaluated BCI benefit and burden and quality of life. Results: Over subsequent months, 12 (29% of the original 42) left the study because of death or rapid disease progression and 6 (14%) left because of decreased interest. Fourteen (33%) completed training and used the BCI independently, mainly for communication. Technical problems were rare. Patient and caregiver ratings indicated that BCI benefit exceeded burden. Quality of life remained stable. Of those not lost to the disease, half completed the study; all but 1 patient kept the BCI for further use. Conclusion: The Wadsworth BCI home system can function reliably and usefully when operated by patients in their homes. BCIs that support communication are at present most suitable for people who are severely disabled but are otherwise in stable health. Improvements in BCI convenience and performance, including some now underway, should increase the number of people who find them useful and the extent to which they are used.