Usage of real objects as links between real and virtual information is one key aspect in augmented reality. A central issue to achieve this link is the estimation of the visuospatial information of the observed object, or in other words, estimating the object pose. Different objects can have different behaviors when used for interaction. This not only encompasses changes in position, but also folding or deformations. Traditional researches in the area solve those pose estimation problems using different approaches, depending on the type of the object. Additionally, some researches are based only on positional information of observed feature points, simplifying the object information. In this work, we explore the pose estimation of different objects by gathering more information from the observed feature points, and obtaining the local poses of such points, which are not explored in other researches. We apply this local pose estimation idea in two different capturing scenarios, reaching two novel approaches of pose estimation: one based on RGB-D cameras, and another based on RGB and machine learning methods. In the RGB-D based approach, we use the feature point orientation and near surface to obtain its normal; then, find the local 6 degrees-of-freedom (DoF) pose. This approach gives us not only the rigid object pose, but also the approximated pose of deformed objects. On the other hand, our RGB based approach explores machine learning with local appearance changes. Unlike other RGB based works, we replace the complex non-linear systems solvers with a fast and robust method, reaching local rotation of the observed feature points, as well as, full 6 DoF rigid object pose with dramatically lower real-time calculation demands. Both approaches show us that gathering local poses can bring information for the pose estimation of different types of objects. / O uso de objetos reais como meio de conexão entre informações reais e virtuais é um aspecto chave dentro da realidade aumentada. Uma questão central para tal conexão é a estimativa de informações visuo-espaciais do objeto, ou em outras palavras, a detecção da pose do objeto. Diferentes objetos podem ter diferentes comportamentos quando utilizados em interações. Não somente incluindo a mudança de posição, mas também sendo dobradas ou deformadas. Pesquisas tradicionais solucionam tais problemas de detecção usando diferentes abordagens, dependendo do tipo de objeto. Adicionalmente, algumas pesquisas se baseiam somente na informação posicional dos pontos de interesse, simplificando a informação do objeto. Neste trabalho, a detecção de pose de diferente objetos é explorada coletando-se mais informações dos pontos de interesse observados e, por sua vez, obtendo as poses locais de tais pontos, poses que não são exploradas em outras pesquisas. Este conceito da detecção de pose locais é aplicada em dois ambientes de capturas, estendendo-se em duas abordagens inovadoras: uma baseada em câmeras RGB-D, e outra baseada em câmeras RGB e métodos de aprendizado de maquinas. Na abordagem baseada em RGB-D, a orientação e superfície ao redor do ponto de interesse são utilizadas para obter a normal do ponto. Através de tais informações a pose local é obtida. Esta abordagem não só permite a obtenção de poses de objetos rígidos, mas também a pose aproximada de objetos deformáveis. Por outro lado, a abordagem baseada em RGB explora o aprendizado de máquina aplicado em alterações das aparências locais. Diferentemente de outros trabalhos baseados em câmeras RGB, esta abordagem substitui solucionadores não lineares complexos com um método rápido e robusto, permitindo a obtenção de rotações locais dos pontos de interesse, assim como, a pose completa (com 6 graus-de-liberdade) de objetos rígidos, com uma demanda computacional muito menor para cálculos em tempo-real. Ambas as abordagens mostram que a coleta de poses locais podem gerar informações para a detecção de poses de diferentes tipos de objetos.

Este trabalho apresenta os resultados da pesquisa e da aplicação de técnicas de GPGPU (General-Purpose computation on Graphics Processing Units) sobre o sistema de vídeo-avatar com realidade aumentada denominado AVMix. Com o aumento da demanda por gráficos tridimensionais interativos em tempo real cada vez mais próximos da realidade, as GPUs (Graphics Processing Units) evoluíram até o estado atual, como um hardware com alto poder computacional que permite o processamento de algoritmos paralelamente sobre um grande volume de dados. Desta forma, É possível usar esta capacidade para aumentar o desempenho de algoritmos usados em diversas áreas, tais como a área de processamento de imagens e visão computacional. A partir das pesquisas de trabalhos semelhantes, definiu-se o uso da arquitetura CUDA (Computer Unified Device Architecture) da Nvidia, que facilita a implementação dos programas executados na GPU e ao mesmo tempo flexibiliza o seu uso, expondo ao programador o detalhamento de alguns recursos de hardware, como por exemplo a quantidade de processadores alocados e os diferentes tipos de memória. Após a reimplementação das rotinas críticas ao desempenho do sistema AVMix (mapa de profundidade, segmentação e interação), os resultados mostram viabilidade do uso da GPU para o processamento de algoritmos paralelos e a importância da avaliação do algoritmo a ser implementado em relação a complexidade do cálculo e ao volume de dados transferidos entre a GPU e a memória principal do computador. / This work presents the results of research and application of GPGPU (General-Purpose computation on Graphics Processing Units) techniques on the video-avatar system with augmented reality called AVMix. With increasing demand for interactive three-dimensional graphics rendered in real-time and closer to reality, GPUs (Graphics Processing Units) evolved to the present state as a high-powered computing hardware enabled to process parallel algorithms over a large data set. This way, it is possible to use this capability to increase the performance of algorithms used in several areas, such as image processing and computer vision. From the research of similar work, it is possible to define the use of CUDA (Computer Unified Device Architecture) from Nvidia, which facilitates the implementation of the programs that run on GPU and at the same time flexibilize its use, exposing to the programmer some details of hardware such as the number of processors allocated and the different types of memory. Following the reimplementation of critical performance routines of AVMix system (depth map, segmentation and interaction), the results show the viability of using the GPU to process parallel algorithms in this application and the importance of evaluating the algorithm to be implemented, considering the complexity of the calculation and the volume of data transferred between the GPU and the computer\'s main memory.

<p>U disertaciji je predstavljeno istraživanje usmereno ka problemu unapređenja rada postojećih industrijskih SCADA sistema. U fokus istraživanja su postavljeni zadaci klasičnih SCADA sistema kao što su: prikupljanje podataka o postrojenju, skladištenje i analiza podataka, kao i prikaz podataka o postrojenju i na koji način se ovi zadaci mogu unaprediti kroz primenu postojećih mobilnih tehnologija, tehnologija Interneta stvari, kao i tehnologija proširene realnosti. Kao rezultat istraživanja predložen je novi model mobilnih nadzornih sistema sa proširenom realnošću i integrisanim industrijskim Internetom stvari. Predloženi model omogućava realizaciju novih sistema nadzora industrijskih postrojenja baziranih na Industriji 4.0 kao i jednostavnu integraciju u postojeće industrijske sisteme. Predloženi novi model nadzornih sistema je opšti model i njegova velika prednost je univerzalnost i otvorenost za primenu novih mobilnih tehnologija, tehnologija Interneta stvari, kao i tehnologija proširene realnosti. On takođe ima svojsvo fleksibilnosti zato što se može primeniti za različite strukture preduzeća, kao i različite proizvodne procese. Predloženi novi model omogućava sledljivost prikupljenih podataka o industrijskim postrojenjima i proizvodnim procesima dvosmerno, od senzora i aktuatora ka menadžmentu postrojenja, kao i od menadžmenta postrojenja ka senzorima i aktuatorima. Omogućena je implementacija različitih algoritama obrade podataka, nezavisnih od predloženog novog modela nadzornih sistema, sa ciljem detektovanja grešaka ili potencijalnih problema u radu industrijskog postrojenja. Predloženi model omogućava uvid u podatke o postrojenju ne samo licima zaduženim za nadzor i održavanje postrojenja, već i proizvođačima industrijske opreme koji dobijaju informacije o opremi u realnim uslovima omogućavajući unapređenje kvaliteta industrijske opreme. Za analizu i obradu podataka prikupljnih o industrijskom postrojenju i procesu model omogućava implementaciju više algoritama istovremeno, čime je moguće međusobno porediti algoritme sa različitim parametrima obrade podataka.</p> / <p>In this doctoral dissertation, a research oriented to the problem of improvement<br />existing industrial SCADA systems is presented. The research is focused on<br />the tasks of classical SCADA systems such as data acquisition, data storage<br />and analysis, and displays of acquired plant data and how these tasks can be<br />improved by applying existing mobile technologies, Internet technology, and<br />technology of augmented reality. As a result of the research, a new model of<br />mobile supervision systems with augmented reality and integrated industrial<br />Internet of Things was proposed. The proposed model enables the<br />implementation of new supervision systems in industrial plants based on<br />Industry 4.0 concept, as well as simple integration into existing industrial plants.<br />The proposed new model of supervision systems is a general model and its<br />great advantage is the universality and openness to the application of new<br />mobile technologies, the Internet of Things technology, and technology of<br />augmented reality. It also has its own flexibility because it can be applied to<br />different company structures and different production processes. The proposed<br />new model enables the traceability of collected data on industrial plants and<br />production processes in two ways, from sensors and actuators to plant<br />management, as well from plant management to sensors and actuators. It is<br />possible to implement different data processing algorithms, independent of the<br />proposed new model of control systems, with the aim of detecting errors or<br />potential problems in the operation of industrial plants or production processes.<br />The proposed model provides insight into plant data not only for plant<br />monitoring and maintenance personnel but also for industrial equipment<br />manufacturers who receive information on equipment in real-world conditions,<br />enabling improvement of the quality of industrial equipment. For the analysis<br />and processing of data collected on industrial plants and production processes,<br />the proposed model enables the implementation of several algorithms at the<br />same time, thus making it possible to compare different algorithms with different<br />data processing parameters.</p>

Cette thèse propose d’étudier les enjeux et les répercussions de l’introduction du dispositif de la réalité augmentée dans l’espace muséal assurant, de ce fait, deux fonctions principales : « médium de médiation » et « médium de création ».La première partie s’intéresse à la présentation de la réalité augmentée. Elle précise, en premier lieu, sa définition, son histoire d’invention, ses différents outils techniques, ainsi que ses multiples usages dans différents domaines d’application. Elle essaie, en deuxième lieu, de démontrer, à partir de l’analyse de quelques expériences artistiques pionnières, l’ancrage de la réalité augmentée dans l’histoire de l’art. La deuxième partie consacrée à la « mise en scène » de l’espace muséal, examine, d’abord, l’implication de la réalité augmentée dans la mise en place de deux stratégies de « médiation in situ » : « médiation documentaire » et « médiation par appropriation ». Ensuite, elle ambitionne, en se référant à quelques exemples de musées, de dégager les principaux avantages et inconvénients liés à l’adoption de la réalité augmentée comme « médium de médiation ». Axée sur la « mise en oeuvre » de l’espace muséal, la troisième partie aborde la réalité augmentée comme étant un « médium de création ». Elle présente, d’abord, l’histoire du passage de l’espace d’exposition muséal en « oeuvre-exposition ». Elle focalise, ensuite, son intérêt sur l’œuvre We AR in MoMA (2010) du collectif Manifest.AR. De l’analyse des étapes du processus de création de celle-ci, il en découle, enfin, la mise en évidence de notions corollaires telles que : hacktivisme, « musée portatif », « ville augmentée ». / This thesis proposes to study the issues and the repercussions of the introduction of the augmented reality device in the museum space, thus ensuring two main functions : "medium of mediation" and "medium of creation".The first part focuses on the presentation of augmented reality. It specifies, firstly, its definition, its history of invention, its various technical tools, and its multiple uses in different fields of application. Secondly, it try to demonstrate, from the analysis of some pioneering artistic experiences, the anchoring of augmented reality in the history of art. The second part devoted to the "staging" of the museum space, examines first, the involvement of augmented reality in the setting up of two "in situ mediation" strategies : "documentary mediation" and "mediation by appropriation". Then, it aims to identify, by referring to a few examples of museums, to the main advantages and disadvantages of adopting augmented reality as a "mediation medium". Axed on the "implementation" of the museum space, the third part studies augmented reality as a "medium of creation". First, it presents the history of the transition from the museum space of exhibition to a "work-exhibition". Then, it focus its interest on the work We AR in MoMA (2010) of the collective Manifest.AR. From the analysis of the stages of the creation process of this one, it results, finally, the highlighting of corollary notions such as : hacktivism, "handled museum", "augmented city".

The visualization of data helps to provide faster and deeper insight into the data. In this work, a system for visualizing and analyzing big data in an interactive mixed reality environment is proposed. Such a system can be used for representing different types of data such as temporal, geospatial, network graph, and high dimensional. Also, an implementation of this system for four different data types are created. The data types include network data, volumetric data, high dimensional, and spectral data for different mixed reality devices such as Microsoft HoloLens, Oculus Rift, Samsung Gear VR, and Android ARCore were created. It was shown that such a system could store and use billions of samples and represent millions of them at once.

Despite the rapidly growing adoption of augmented reality (AR) applications, existing methods for interacting with AR content are rated poorly, with surveyors of the area calling for better means of interaction, while researchers strive to create more natural input methods, mainly focusing on gesture input. This thesis aims to contribute to the aforementioned efforts by recognizing that technologies for consumer-grade smartphone-based pose estimation have been rapidly improving in recent years and due to their increased accuracy may have untapped potential ready to be utilized for user input. To this end, a rudimentary system for pose based input is integrated into prototype applications, which are constructed with both pose based input and touch input in mind. In this work, pose, pose estimation, and posed based input refer to using the distance and orientation of the user (or more precisely, the distance and orientation of their device) in relation to the AR content. Using said prototypes within a user interaction study allowed the identification of user preferences which indicate the approaches that future efforts into utilizing pose for input in an AR context ought to adopt. By comparing questionnaire answers and logged positional data across four prototype scenarios, it can be clearly identified that to perceive pose input as intuitive, the AR experiences shouldn’t employ a scale which is so large that it requires substantial shifts in the position of the user, as opposed to merely shifts in the position of the user’s device.

近年來GPS導航軟體正蓬勃發展，但市面上的導航軟體大多只能帶領使用者到達旅遊景點，無法更進一步提供旅遊資訊與行程建議。因此，本研究結合行動裝置與「3D擴增實境」（3D Augmented Reality），試圖規劃一套新型的導覽模式。我們以淡水為例，結合當地古蹟景點與歷史典故，使導覽系統能提供豐富的數位內容。在設計的過程中，本研究建構出「新科技敘事模式」，在「故事」、「影音效果」、與「互動機制」之間取得平衡，讓使用者體驗到故事、感官刺激、和旅遊合而為一的導覽經驗。此外，在「3D擴增實境」上，本研究建構了不同精細度的3D模型，並且在行動裝置上測試其效能。結果發現，「局部精化」的新形態建模概念，能夠兼顧美觀與運算效能。最後，本研究針對行動裝置硬體效能的負荷進行了權重測試，並獲得GPU（Render）＞CPU（PR）的結論。 / In the past few years we have witnessed the rapid growth in the sales of GPS related products on the market. Nonetheless, most navigation software solely provides route planning rather than travel information or tour guidance. This research aims to combine mobile devices and 3D augmented reality (AR) to create a novel form of navigational experience. Taking the famous tourist spot Tamsui as an illustration, materials adapted local monuments and historical allusions are re-arranged creatively to provide substantial digital contents with helpful navigation information. During the design process, this research creates different modes of narration, enabling users to undergo a brand new navigation experience through the blending of various media sources, including story, video and interaction. Additionally, this research constructs 3D models of different levels of detail and examines their efficiency on mobile devices. The experimental results indicate that 3D model built with partial fineness provides a balance between artistic fidelity and computational efficiency. Also, the results from the experiment suggest that the loading of the GPU (responsible for rendering the model) is greater than that of the CPU (responsible for pattern recognition).

Local feature detection and its applications in different problems are quite popular in vision research. In order to analyze a scene, its invariant features, which are distinguishable in many views of this scene, are used in pose estimation, object detection and augmented reality. However, required performance metrics might change according to the application type / in general, the main metrics are accepted as accuracy and computational complexity. The contributions in this thesis provide improving these metrics and can be divided into three parts, as local feature detection, local feature description and description matching in different views of the same scene. In this thesis an efficient feature detection algorithm with sufficient repeatability performance is proposed. This detection method is convenient for real-time applications. For local description, a novel local binary pattern outperforming state-of-the-art binary pattern is proposed. As a final task, a fuzzy decision tree method is presented for approximate nearest neighbor search. In all parts of the system, computational efficiency is considered and the algorithms are designed according to limited processing time. Finally, an overall system capable of matching different views of the same scene has been proposed and executed in a mobile platform. The results are quite promising such that the presented system can be used in real-time applications, such as augmented reality, object retrieval, object tracking and pose estimation.

A seamless blending of the real and virtual worlds is key to increased immersion and improved user experiences for augmented reality (AR). Photorealistic and non-photorealistic rendering (NPR) are two ways to achieve this goal. Non-photorealistic rendering creates an abstract and stylized version of both the real and virtual world, making them indistinguishable. This could be particularly useful in some applications (e.g., AR/VR aided machine repair, or for virtual medical surgery) or for certain AR games with artistic stylization. Achieving temporal coherence is a key challenge for all NPR algorithms. Rendered results are temporally coherent when each frame smoothly and seamlessly transitions to the next one without visual flickering or artifacts that distract the eye from perceived smoothness. NPR algorithms with coherence are interesting in both general computer graphics and AR/VR areas. Rendering stylized AR without coherence processing causes the final results to be visually distracting. While various NPR algorithms with coherence support have been proposed in general graphics community for video processing, many of these algorithms require thorough analysis of all frames of the input video and cannot be directly applied to real-time AR applications. We have investigated existing NPR algorithms with coherence in both general graphics and AR/VR areas. These algorithms are divided into two categories: Model Space and Image Space. We present several NPR algorithms with coherence for AR: a watercolor inspired NPR algorithm, a painterly rendering algorithm, and NPR algorithms in the model space that can support several styling effects.

Som en del av forskningsprojektet Young Operator 2020 så syftar den här studien till att vara ett första steg mot en produkt, som med hjälp av av tekniken augmented reality ska vara en del av framtidens tillverkningsindustri. Studien utgår från kognitionsvetenskap och människa-datorinteraktion för att ha användaren och användarens interaktion med prototypen i fokus. Eftersom olika typer av användare kan behöva olika saker av ett system, kommer den här studien också att utgå från teorier om kompetensnivåer. Ett system som är användbart för exempelvis en novis behöver inte nödvändigtvis vara användbart för en expert. En prototyp har skapats som en del av ett annat projekt och den utgår från det arbetssätt som operatörer på Volvo GTO i Skövde har idag. Prototypen använder sig av augmented reality för att visualisera instruktioner. Det är tänkt att prototypen bland annat ska hjälpa noviser att närma sig expertanvändare. Ett kompetitivt användbarhetstest har gjorts där prototypen jämförts med nuvarande system på Volvo GTO. Resultatet av studien visar att prototypen i dagsläget inte kan anses vara användbar. Flera rekommendationer har dock presenterats som ett bidrag till vidareutveckling av prototypen. / Young Operator 2020