Recent rapid development of cost-effective, accurate digital imaging sensors, high-speed computational hardware, and tractable design software has given rise to the growing field of augmented reality in the computer vision realm. The system design of a 'Digital Whiteboard' system is presented with the intention of realizing a practical, cost-effective and publicly available spatial augmented reality system. A Microsoft Kinect sensor and a PC projector coupled with a desktop computer form a type of spatial augmented reality system that creates a projection based graphical user interface that can turn any wall or planar surface into a 'Digital Whiteboard'. The system supports two kinds of user inputs consisting of depth and infra-red information. An infra-red collimated light source, like that of a laser pointer pen, serves as a stylus for user input. The user can point and shine the infra-red stylus on the selected planar region and the reflection of the infra-red light source is registered by the system using the infra-red camera of the Kinect. Using the geometric transformation between the Kinect and the projector, obtained with system calibration, the projector displays contours corresponding to the movement of the stylus on the 'Digital Whiteboard' region, according to a smooth curve fitting algorithm. The described projector-based spatial augmented reality system provides new unique possibilities for user interaction with digital content.

I detta kandidatarbete undersöker vi hur projection mapping (projicering på en yta) kan användas för att skapa en rumslig berättelse eller upplevelse. Utifrån vår egna tolkning av calm technologys (lugn teknologi) principer har vi skapat ett designperspektiv som syftar till att göra designen lugn, genom att teknologin inte ska bli störande för åskådaren. Till vår hjälp har vi använt oss utav Keri Smiths metoder till att utforska och experimentera med omgivningen och material. För att sedan kunna utöka kreativiteten och hitta nya idéer inom projection mapping har vi använt oss av metoder såsom brainstorming, moodboards och skisser. Vi har även utgått från Stolterman och Löwgrens syn på en designprocess. Detta resulterade till tre gestaltningar där alla utgår från naturens lugna atmosfär. Under detta arbete har vi kommit fram till att det finns många olika sätt att berätta med ytor, det behöver inte bara vara ett fysiskt rum med väggar och tak. Ytor kan bli rum genom att manipulera fysiska föremål med projektorer. Genom att lägga på ett extra lager på objekten som till exempel en animation så kan det förstärka objektet i sig eller få en helt ny karaktär. Med enkla medel som olika mönster, färger och ljus. / In this bachelor thesis we will examine how projection mapping (projection on a surface) can be used as a spatial experience. From our own interpretation of calm technology’s principles, will we create a design perspective that could keep the design calm with respect to the technology, and not be a disturbing moment for the viewer. To guide us through this we have taken Keri Smith’s methods to explore and experiment the environment and different materials. To expand the creativity to find new possibilities in projection mapping, we have used methods like brainstorming, moodboards and sketches. To help our process we based our project on Stolterman and Löwgren’s view of the design process. This led to a result of three figurations where all of them are inspired by the atmosphere of our nature. Through this project we have reached that there’s a lot of ways you can narrate with spaces, it doesn't need to be a physical room where you can stand in. Spaces can transform by manipulating physical artifacts. By adding an extra layer onto these artifacts like an animation it can amplify it’s meaning or give it a new character. Only with simple means like different patterns, colors and light.

Spatial augmented reality is a particular kind of augmented reality technique that uses projector to blend the real objects with virtual contents. Coincidentally, as a means of 3D shape measurement, structured light illumination makes use of projector as part of its system as well. It uses the projector to generate important clues to establish the correspondence between the 2D image coordinate system and the 3D world coordinate system. So it is appealing to build a system that can carry out the functionalities of both spatial augmented reality and structured light illumination. In this dissertation, we present all the hardware platforms we developed and their related applications in spatial augmented reality and structured light illumination. Firstly, it is a dual-projector structured light 3D scanning system that has two synchronized projectors operate simultaneously, consequently it outperforms the traditional structured light 3D scanning system which only include one projector in terms of the quality of 3D reconstructions. Secondly, we introduce a modified dual-projector structured light 3D scanning system aiming at detecting and solving the multi-path interference. Thirdly, we propose an augmented reality face paint system which detects human face in a scene and paints the face with any favorite colors by projection. Additionally, the system incorporates a second camera to realize the 3D space position tracking by exploiting the principle of structured light illumination. At last, a structured light 3D scanning system with its own built-in machine vision camera is presented as the future work. So far the standalone camera has been completed from the a bare CMOS sensor. With this customized camera, we can achieve high dynamic range imaging and better synchronization between the camera and projector. But the full-blown system that includes HDMI transmitter, structured light pattern generator and synchronization logic has yet to be done due to the lack of a well designed high speed PCB.

Structured Light Imaging (SLI) is a means of digital reconstruction, or Three-Dimensional (3D) scanning, and has uses that span many disciplines. A projector, camera and Personal Computer (PC) are required to perform such 3D scans. Slight variances in synchronization between these three devices can cause malfunctions in the process due to the limitations of PC graphics processors as real-time systems. Previous work used a Field Programmable Gate Array (FPGA) to both drive the projector and trigger the camera, eliminating these timing issues, but still needing an external camera. This thesis proposes the incorporation of the camera with the FPGA SLI controller by means of a custom printed circuit board (PCB) design. Featuring a high speed image sensor as well as High Definition Multimedia Interface (HDMI) input and output, this PCB enables the FPGA to perform SLI scans as well as pass through HDMI video to the projector for Spatial Augmented Reality (SAR) purposes. Minimizing ripple noise on the power supply by means of effective circuit design and PCB layout, realizes a compact and cost effective machine vision sensing solution.

This thesis focuses on the application of the tangible user interface and spatial augmented reality principals for playing board games. For this purpose, a system called interactive table was developed. It is both hardware and software solution. The hardware part solves the installation of a depth camera and a projector located above the table. The software solution deals with calibration of the depth camera and projector system using known procedures for camera calibration on the one hand, on the other hand it solves tasks from the field of computer vision. This perceptional part is the core of the whole thesis and for the purposes of playing board games it provides information about the location of the physical game object, physical game board and the calculations of mask for masking distracting objects. An educational application was developed for the purposes of demonstration of the system options. The interactive table offers new possibilities for playing board games in a real-world environment by the combination of the augmented reality elements with real-world objects and the related new user experience.

This report covers the methodology, research, and design process of my Thesis Project I: The Mixed Reality Book. The project is a proof-of-concept system that adds contextual periphery effects to regular paper books, using Spatial Augmented Reality. The intention is to enhance reading experiences within public libraries – amongst children and students. In this brief study we investigate how Projected Periphery can create, improve, and augment reading by manipulating the physical book, and the area around it, using projections. Throughout the study, I conduct design engagements, rapid prototyping, and workshops with the intention of identifying meaningful interactions. Two primary contexts of use are identified and analysed with an emphasis on developing usable design conventions and laying the foundation for a Mixed Reality Book system. The result is a working prototype, analysis of the research and challenges, and an exploration of how this technology could be shaped further and deployed.

The upcoming new generation of autonomous vehicles for transporting materials in industrial environments will be more versatile, flexible and efficient than traditional Automatic Guided Vehicles (AGV), which simply follow pre-defined paths. However, freely navigating vehicles can appear unpredictable to human workers and thus cause stress and render joint use of the available space inefficient. This work addresses the problem of providing information regarding a service robot’s intention to humans co-populating the environment. The overall goal is to make humans feel safer and more comfortable, even when they are in close vicinity of the robot. A spatial Augmented Reality (AR) system for robot intention communication by means of projecting proxemic information onto shared floor space is developed on a robotic fork-lift by equipping it with a LED projector. This helps in visualizing internal state information and intents on the shared floors spaces. The robot’s ability to communicate its intentions is evaluated in realistic situations where test subjects meet the robotic forklift. A Likert scalebased evaluation which also includes comparisons to human-human intention communication was performed. The results show that already adding simple information, such as the trajectory and the space to be occupied by the robot in the near future, is able to effectively improve human response to the robot. This kind of synergistic human-robot interaction in a work environment is expected to increase the robot’s acceptability in the industry.

La Réalité Augmentée Spatiale (SAR) vise à superposer spatialement l'information virtuelle sur des objets physiques. Au cours des dernières décennies ce domaine a connu une grande expansion et est utilisé dans divers domaines, tels que la médecine, le prototypage, le divertissement etc. Cependant, pour obtenir des projections de bonne qualité, on doit résoudre plusieurs problèmes, dont les plus importants sont la gamme de couleurs réduite du projecteur, la lumière ambiante, la couleur du fond, et la configuration arbitraire de la surface de projection dans la scène. Ces facteurs entraînent des distorsions dans les images qui requièrent des processus de compensation complémentaires.Les projections intelligentes (smart projections) sont au cœur des applications de SAR. Composées d'un dispositif de projection et d'un dispositif d'acquisition, elles contrôlent l'aspect de la projection et effectuent des corrections à la volée pour compenser les distorsions. Bien que les méthodes actives de Lumière Structurée aient été utilisées classiquement pour résoudre ces problèmes de compensation géométrique, cette thèse propose une nouvelle approche non intrusive pour la compensation géométrique de plusieurs surfaces planes et pour la reconnaissance des objets en SAR s'appuyant uniquement sur la capture du contenu projeté.Premièrement, cette thèse étude l'usage de l'invariance couleur pour améliorer la qualité de la mise en correspondance entre primitives dans une configuration d'acquisition des images vidéoprojetées. Nous comparons la performance de la plupart des méthodes de l'état de l'art avec celle du descripteur proposé basé sur l'égalisation d'histogramme. Deuxièmement, pour mieux traiter les conditions standard des systèmes projecteur-caméra, deux ensembles de données de captures de projections réelles, ont été spécialement préparés à des fins expérimentales. La performance de tous les algorithmes considérés est analysée de façon approfondie et des propositions de recommandations sont faites sur le choix des algorithmes les mieux adaptés en fonction des conditions expérimentales (paramètres image, disposition spatiale, couleur du fond...). Troisièmement, nous considérons le problème d'ajustement multi-surface pour compenser des distorsions d'homographie dans les images acquises. Une combinaison de mise en correspondance entre les primitives et de Flux Optique est proposée afin d'obtenir une compensation géométrique plus rapide. Quatrièmement, une nouvelle application en reconnaissance d'objet à partir de captures d'images vidéo-projetées est mise en œuvre. Finalement, une implémentation GPU temps réel des algorithmes considérés ouvre des pistes pour la compensation géométrique non intrusive en SAR basée sur la mise en correspondances entre primitives. / Spatial Augmented Reality (SAR) aims at spatially superposing virtual information on real-world objects. Over the last decades, it has gained a lot of success and been used in manifold applications in various domains, such as medicine, prototyping, entertainment etc. However, to obtain projections of a good quality one has to deal with multiple problems, among them the most important are the limited projector output gamut, ambient illumination, color background, and arbitrary geometric surface configurations of the projection scene. These factors result in image distortions which require additional compensation steps.Smart-projections are at the core of PAR applications. Equipped with a projection and acquisitions devices, they control the projection appearance and introduce corrections on the fly to compensate distortions. Although active structured-light techniques have been so far the de-facto method to address such problems, this PhD thesis addresses a relatively new unintrusive content-based approach for geometric compensation of multiple planar surfaces and for object recognition in SAR.Firstly, this thesis investigates the use of color-invariance for feature matching quality enhancement in projection-acquisition scenarios. The performance of most state-of-the art methods are studied along with the proposed local histogram equalization-based descriptor. Secondly, to better address the typical conditions encountered when using a projector-camera system, two datasets of real-world projections were specially prepared for experimental purposes. Through a series of evaluation frameworks, the performance of all considered algorithms is thoroughly analyzed, providing several inferences on that which algorithms are more appropriate in each condition. Thirdly, this PhD work addresses the problem of multiple-surface fitting used to compensate different homography distortions in acquired images. A combination of feature matching and Optical Flow tracking is proposed in order to achieve a more low-weight geometric compensation. Fourthly, an example of new application to object recognition from acquired projections is showed. Finally, a real-time implementation of considered methods on GPU shows prospects for the unintrusive feature matching-based geometric compensation in SAR applications.

Cette dissertation décrit le design, implémentation et évaluation de nouvelles applications en Réalité Augmentée Spatiale (RAS). Ces applications sont concentrées sur l’amélioration du dessin physique tel que les dessins au crayons ou peintures en projetant des outils numériques.Tout d’abord, nous décrivons notre système de RAS et ses possibilités. Il prend en comptes les paramètres internes et externes d’un couple caméra/projecteur pour permettre une projection précise sur des feuilles de papier. De plus, il permet la détection du toucher des feuilles et de la position de la main au dessus de celles-ci grâce à une caméra de profondeur. Par conséquent, il permet la création d’écrans tactiles interactifs sur des feuilles de papier posées sur une table.Ensuite, nous nous penchons sur la création d’art visuel, plus précisément sur les premières étapes de la création quand l’artiste créer la structure. Nous offrons la possibilité de créer et d’éditer des lignes de construction numériques (LCN) projetées sur le papier. Ces outils sont des outils de Réalité Augmentée (RA), c’est-à-dire qu’ils s’ajoutent aux outils existants: toute l’interface utilisateur est sur la table, et l’utilisateur n’utilise jamais ni une souris, ni un clavier ni un écran. En plus des simples LCN (lignes et courbes), nous proposons une spécialisation pour les dessins spécifiques tels que les dessins en perspective, les dessins de personnages ainsi que les dessins à partir d’un modèle 3D. Nous proposons de nouvelles méthodes pour afficher et interagir avec des objets 3D sur papier. Nous proposons également de créer des dessins mixtes:art visuel interactif qui tire parti à la fois des possibilités physiques et numériques.Pour finir, nous décrivons des nouveaux usages pour notre système de RAS de nombreux contextes différents à travers des démonstrations publiques. L’acceptabilité de ce genre de système a été très bonne, et “magique” par la plupart des utilisateurs. Ils ont juste vu et interagis avec des feuilles de papier sans remarquer le système de projection et suivi. / This dissertation presents the design, implementation, and evaluation of novel SpatialAugmented Reality (SAR) applications. These applications focus on enhancing physical drawingsuch as pencil drawing or painting by the projection of digital tools.First, we describe our tabletop SAR system and its possiblities. It takes into account thecamera and projector internal and external parameters to enable precise projection onto trackedsheets of paper. In addition to this, we enable touch and hand detection over the table with adepth camera. Consequently, it enables the creation of interactive touch screens on the pieces ofpaper on a table.Next, we focus on the creation of visual art, more precisely the first steps of creation whenthe artist structures it. We enable the creation and edition of digital construction lines (DCL)that are projected onto the paper. These tools are Augmented Reality (AR) tools, i.e., theyadd to existing tools: the whole user interface is on the table and the user never uses a mouse,keyboard or screen. In addition to the simple DLC (lines and curves), we propose to specializethem for specific drawings such as perspective drawing, character drawing and drawing from a3D model. We propose a novel way to display and interact with 3D objects on paper. We alsopropose to create mixed media drawing: it enables the creation of interactive visual art thattakes advantage from both the physical and digital possiblities.Finally, we describe new usages for our SAR system in many different contexts throughpublic exhibitions. The acceptability of such systems was found very good and “magical” bymost of the users. They only saw and interacted with interactive pieces of paper without noticingthe projection and tracking system.