The research presented in this thesis concerns the contribution of virtual human (or avatar) fidelity to social interaction in virtual environments (VEs) and how sensory fusion can improve these interactions. VEs present new possibilities for mediated communication by placing people in a shared 3D context. However, there are technical constraints in creating photo realistic and behaviorally realistic avatars capable of mimicking a person's actions or intentions in real time. At the same time, previous research findings indicate that virtual humans can elicit social responses even with minimal cues, suggesting that full realism may not be essential for effective social interaction. This research explores the impact of avatar behavioral realism on people's experience of interacting with virtual humans by varying the interaction fidelity. This is accomplished through the creation of Cinemacraft, a technology-mediated immersive platform for collaborative human-computer interaction in a virtual 3D world and the incorporation of sensory fusion to improve the fidelity of interactions and realtime collaboration. It investigates interaction techniques within the context of a multiplayer sandbox voxel game engine and proposes how interaction qualities of the shared virtual 3D space can be used to further involve a user as well as simultaneously offer a stimulating experience. The primary hypothesis of the study is that embodied interactions result in a higher degree of presence and co-presence, and that sensory fusion can improve the quality of presence and co-presence. The argument is developed through research justification, followed by a user-study to demonstrate the qualitative results and quantitative metrics.This research comprises of an experiment involving 24 participants. Experiment tasks focus on distinct but interrelated questions as higher levels of interaction fidelity are introduced.The outcome of this research is the generation of an interactive and accessible sensory fusion platform capable of delivering compelling live collaborative performances and empathetic musical storytelling that uses low fidelity avatars to successfully sidestep the 'uncanny valley'. This research contributes to the field of immersive collaborative interaction by making transparent the methodology, instruments and code. Further, it is presented in non-technical terminology making it accessible for developers aspiring to use interactive 3D media to pro-mote further experimentation and conceptual discussions, as well as team members with less technological expertise. / Master of Science / The work presented in this thesis explores social interactions and collaboration between users within the context of an immersive game platform. Improving the quality of these interactions is often challenging in terms of creating relatable virtual representations of the user that can also accurately capture user performances and behavioral intentions in real time. This research focuses on changing modes of performance capture to affect the quality of interactions between users. The immersive game platform, Cinemacraft, uses a Minecraft style game engine to propose how interaction qualities of a shared virtual space can be used to further involve a user as well as simultaneously offer a stimulating experience. The platform can accurately capture the users’ posture, limb movement, facial expressions and lip-synced mouth states and comes with an array of live cinematic production tools. The primary hypothesis of the study is that more natural modes of performance capture would result in a higher quality of interaction. Also, an additional level of intelligence to incorporate voice capture to improve tracking of users’ facial performance would yield the highest quality of interactions. The argument is developed through research justification, followed by a user-study involving 24 participants, to demonstrate the qualitative results and quantitative metrics. The outcome of this research is the generation of an interactive and accessible immersive game platform capable of delivering compelling live collaborative performances and empathetic musical storytelling. This research contributes to the field of immersive collaborative interaction by making transparent the methodology, instruments and code. Further, it is presented in non-technical terminology making it accessible for developers aspiring to use interactive 3D media to promote further experimentation and conceptual discussions, as well as team members with less technological expertise.

Indiana University-Purdue University Indianapolis (IUPUI) / Semantic segmentation is a machine learning task that is seeing increased utilization in multiples fields, from medical imagery, to land demarcation, and autonomous vehicles. Semantic segmentation performs the pixel-wise classification of images, creating a new, seg- mented representation of the input that can be useful for detected various terrain and objects within and image. Recently, convolutional neural networks have been heavily utilized when creating neural networks tackling the semantic segmentation task. This is particularly true in the field of autonomous driving systems. The requirements of automated driver assistance systems (ADAS) drive semantic seg- mentation models targeted for deployment on ADAS to be lightweight while maintaining accuracy. A commonly used method to increase accuracy in the autonomous vehicle field is to fuse multiple sensory modalities. This research focuses on leveraging the fusion of long wave infrared (LWIR) imagery with visual spectrum imagery to fill in the inherent perfor- mance gaps when using visual imagery alone. This comes with a host of benefits, such as increase performance in various lighting conditions and adverse environmental conditions. Utilizing this fusion technique is an effective method of increasing the accuracy of a semantic segmentation model. Being a lightweight architecture is key for successful deployment on ADAS, as these systems often have resource constraints and need to operate in real-time. Multi-Spectral Fusion Network (MFNet) [1] accomplishes these parameters by leveraging a sensory fusion approach, and as such was selected as the baseline architecture for this research. Many improvements were made upon the baseline architecture by leveraging a variety of techniques. Such improvements include the proposal of a novel loss function categori- cal cross-entropy dice loss, introduction of squeeze and excitation (SE) blocks, addition of pyramid pooling, a new fusion technique, and drop input data augmentation. These improve- ments culminated in the creation of the Fast Thermal Fusion Network (FTFNet). Further improvements were made by introducing depthwise separable convolutional layers leading to lightweight FTFNet variants, FTFNet Lite 1 & 2. 13 The FTFNet family was trained on the Multi-Spectral Road Scenarios (MSRS) and MIL- Coaxials visual/LWIR datasets. The proposed modifications lead to an improvement over the baseline in mean intersection over union (mIoU) of 2.92% and 2.03% for FTFNet and FTFNet Lite 2 respectively when trained on the MSRS dataset. Additionally, when trained on the MIL-Coaxials dataset, the FTFNet family showed improvements in mIoU of 8.69%, 4.4%, and 5.0% for FTFNet, FTFNet Lite 1, and FTFNet Lite 2.

Sistemas computacionais empregados na Agricultura de Precisão (AP) são dedicados, atualmente, a prover relativa amostragem, precisão e nível de processamento de dados requeridas pelas práticas agrícolas, que não são comuns a agricultura convencional, elevando custos com a produção e pesquisas direcionadas ao sensoriamento remoto, para o mapeamento e inspeção das linhas de plantio. São tarefas a priori realizadas com o uso de sensores proprioceptivos e exteroceptivos, instrumentação embarcada, informações geográficas e implementos existentes na produção e cultivo, que auxiliam as atividades do agricultor durante as etapas de plantação, maturação, conservação e colheita de determinada cultura. Ainda, estas tarefas são auxiliadas com uso de robôs agrícolas móveis terrestres, como plataformas veiculares autônomas para a locomoção entre as linhas de plantio na aquisição de dados do campo. Dadas estas informações e relacionando o grau de investimento e desenvolvimento destas tecnologias, o objetivo deste trabalho é auxiliar a inspeção, quantificação e qualificação de culturas agrícolas (citricultura) de determinada área de plantio através da análise e identificação de dados por fusão sensorial, associada a processamento de imagens digitais, termografia óptica e sensores de fluxo óptico e refletância, baseado na extração de objetos de cenas naturais reais, identificando itens como frutos, gramíneas, caules, ramos, folhas e galhos, provendo assim um conjunto de dados qualitativos e quantitativos da cultura analisada. A partir de um sistema de visão computacional e fusão sensorial, com câmeras GigE/IP-CMOS, conjunto de sonares, câmera térmica, sensores de fluxo óptico e sensores de refletância, são embarcados nas laterais da estrutura veicular, com a mesma relação do centro geométrico cinemático do robô agrícola móvel. Após a aquisição dos dados, são aplicadas técnicas de processamento de imagens e sinais para a segmentação de regiões não homogêneas, e reconhecimento de padrões por classificadores estatísticos. Tais técnicas são programadas em OpenCV e MATLAB, de forma offline. Classificadores cognitivos que realizam as correspondências de padrões e classes são pretendidos na combinação de técnicas de fusão de dados ponderada, para que, durante a locomoção do robô agrícola móvel, as etapas de processamento de imagens e combinação de parâmetros para a classificação sejam manipuláveis para análise a posteriori, conflitando os dados existentes com prováveis alterações decorrentes na cultura e na biomassa. / Computer systems are used in Precision Agriculture (PA) to provide relational sampling, accuracy and data processing required for agricultural practices and schemes, which are not common to conventional agriculture, demanding higher costs of production and research directed to the remote sensing for mapping and inspection of crop rows. Tasks are carried out using a priori On-the-Go sensors, and proprioceptive and exteroceptive ones, embedded instrumentation, geographic information and existing implements on production and farming, which these activities ensure during the steps of planting, maturation, maintenance and harvesting of a particular culture. Also, these tasks are aided with the use of terrestrial agricultural mobile robots such as autonomous vehicle platforms for locomotion between the crop rows in the acquisition of field data. Given this information and relating the investment and development of these technologies, the goal of this work is to assist the inspection, quantification and qualification of agricultural crops (citrus) of planting area through analyzing and identifying data for sensor fusion, associated with digital image processing, optical thermal imaging and optical flow sensors and reflectance, based on the extraction of real natural scenes objects, identifying items such as fruits, grasses, stems, branches and leaves, thus providing a qualitative and quantitative set of data analyzed culture. As of a computer vision system and sensor fusion, Gigabit Ethernet cameras, sonars, thermal camera, infrared optical flow sensors and monochrome CMOS sensors are embedded in two sides of the vehicle structure, with the same geometric center ratio of the agricultural mobile robot. After data acquisition, image and signal processing techniques are applied for non homogeneous region segmentation, and pattern recognition through statistical classifiers. Such techniques are programmed in MATLAB and OpenCV, embedded in a computer platform. Cognitive classifiers that perform pattern and classes matching are intended on a combination of weighted data fusion techniques, that during locomotion of the agricultural mobile robot, the steps of image processing and combination of parameters for classification are manipulated for analysis retrospectively, conflicting existing data with likely changes resulting in culture and biomass.

Sistemas computacionais empregados na Agricultura de Precisão (AP) são dedicados, atualmente, a prover relativa amostragem, precisão e nível de processamento de dados requeridas pelas práticas agrícolas, que não são comuns a agricultura convencional, elevando custos com a produção e pesquisas direcionadas ao sensoriamento remoto, para o mapeamento e inspeção das linhas de plantio. São tarefas a priori realizadas com o uso de sensores proprioceptivos e exteroceptivos, instrumentação embarcada, informações geográficas e implementos existentes na produção e cultivo, que auxiliam as atividades do agricultor durante as etapas de plantação, maturação, conservação e colheita de determinada cultura. Ainda, estas tarefas são auxiliadas com uso de robôs agrícolas móveis terrestres, como plataformas veiculares autônomas para a locomoção entre as linhas de plantio na aquisição de dados do campo. Dadas estas informações e relacionando o grau de investimento e desenvolvimento destas tecnologias, o objetivo deste trabalho é auxiliar a inspeção, quantificação e qualificação de culturas agrícolas (citricultura) de determinada área de plantio através da análise e identificação de dados por fusão sensorial, associada a processamento de imagens digitais, termografia óptica e sensores de fluxo óptico e refletância, baseado na extração de objetos de cenas naturais reais, identificando itens como frutos, gramíneas, caules, ramos, folhas e galhos, provendo assim um conjunto de dados qualitativos e quantitativos da cultura analisada. A partir de um sistema de visão computacional e fusão sensorial, com câmeras GigE/IP-CMOS, conjunto de sonares, câmera térmica, sensores de fluxo óptico e sensores de refletância, são embarcados nas laterais da estrutura veicular, com a mesma relação do centro geométrico cinemático do robô agrícola móvel. Após a aquisição dos dados, são aplicadas técnicas de processamento de imagens e sinais para a segmentação de regiões não homogêneas, e reconhecimento de padrões por classificadores estatísticos. Tais técnicas são programadas em OpenCV e MATLAB, de forma offline. Classificadores cognitivos que realizam as correspondências de padrões e classes são pretendidos na combinação de técnicas de fusão de dados ponderada, para que, durante a locomoção do robô agrícola móvel, as etapas de processamento de imagens e combinação de parâmetros para a classificação sejam manipuláveis para análise a posteriori, conflitando os dados existentes com prováveis alterações decorrentes na cultura e na biomassa. / Computer systems are used in Precision Agriculture (PA) to provide relational sampling, accuracy and data processing required for agricultural practices and schemes, which are not common to conventional agriculture, demanding higher costs of production and research directed to the remote sensing for mapping and inspection of crop rows. Tasks are carried out using a priori On-the-Go sensors, and proprioceptive and exteroceptive ones, embedded instrumentation, geographic information and existing implements on production and farming, which these activities ensure during the steps of planting, maturation, maintenance and harvesting of a particular culture. Also, these tasks are aided with the use of terrestrial agricultural mobile robots such as autonomous vehicle platforms for locomotion between the crop rows in the acquisition of field data. Given this information and relating the investment and development of these technologies, the goal of this work is to assist the inspection, quantification and qualification of agricultural crops (citrus) of planting area through analyzing and identifying data for sensor fusion, associated with digital image processing, optical thermal imaging and optical flow sensors and reflectance, based on the extraction of real natural scenes objects, identifying items such as fruits, grasses, stems, branches and leaves, thus providing a qualitative and quantitative set of data analyzed culture. As of a computer vision system and sensor fusion, Gigabit Ethernet cameras, sonars, thermal camera, infrared optical flow sensors and monochrome CMOS sensors are embedded in two sides of the vehicle structure, with the same geometric center ratio of the agricultural mobile robot. After data acquisition, image and signal processing techniques are applied for non homogeneous region segmentation, and pattern recognition through statistical classifiers. Such techniques are programmed in MATLAB and OpenCV, embedded in a computer platform. Cognitive classifiers that perform pattern and classes matching are intended on a combination of weighted data fusion techniques, that during locomotion of the agricultural mobile robot, the steps of image processing and combination of parameters for classification are manipulated for analysis retrospectively, conflicting existing data with likely changes resulting in culture and biomass.

<p>  </p> <p>Semantic segmentation is a machine learning task that is seeing increased utilization in multiples fields, from medical imagery, to land demarcation, and autonomous vehicles. Semantic segmentation performs the pixel-wise classification of images, creating a new, segmented representation of the input that can be useful for detected various terrain and objects within and image. Recently, convolutional neural networks have been heavily utilized when creating neural networks tackling the semantic segmentation task. This is particularly true in the field of autonomous driving systems.</p> <p>The requirements of automated driver assistance systems (ADAS) drive semantic segmentation models targeted for deployment on ADAS to be lightweight while maintaining accuracy. A commonly used method to increase accuracy in the autonomous vehicle field is to fuse multiple sensory modalities. This research focuses on leveraging the fusion of long wave infrared (LWIR) imagery with visual spectrum imagery to fill in the inherent performance gaps when using visual imagery alone. This comes with a host of benefits, such as increase performance in various lighting conditions and adverse environmental conditions. Utilizing this fusion technique is an effective method of increasing the accuracy of a semantic segmentation model. Being a lightweight architecture is key for successful deployment on ADAS, as these systems often have resource constraints and need to operate in real-time. Multi-Spectral Fusion Network (MFNet) [ 1 ] accomplishes these parameters by leveraging a sensory fusion approach, and as such was selected as the baseline architecture for this research.</p> <p>Many improvements were made upon the baseline architecture by leveraging a variety of techniques. Such improvements include the proposal of a novel loss function categorical cross-entropy dice loss, introduction of squeeze and excitation (SE) blocks, addition of pyramid pooling, a new fusion technique, and drop input data augmentation. These improvements culminated in the creation of the Fast Thermal Fusion Network (FTFNet). Further improvements were made by introducing depthwise separable convolutional layers leading to lightweight FTFNet variants, FTFNet Lite 1 & 2.</p>

Disertace se zabývá aplikaci neuronových sítí pro detekci objektů na multimodální data v robotice. Celkem cílí na tři oblasti: tvorbu datasetu, zpracování multimodálních dat a trénování neuronových sítí. Nejdůležitější části práce je návrh metody pro tvorbu rozsáhlých anotovaných datasetů bez časové náročného lidského zásahu. Metoda používá neuronové sítě trénované na RGB obrázcích. Užitím dat z několika snímačů pro vytvoření modelu okolí a mapuje anotace z RGB obrázků na jinou datovou doménu jako jsou termální obrázky, či mračna bodů. Pomoci této metody autor vytvořil dataset několika set tisíc anotovaných obrázků a použil je pro trénink neuronové sítě, která následně překonala modely trénované na menších, lidmi anotovaných datasetech. Dále se autor v práci zabývá robustností detekce objektů v několika datových doménách za různých povětrnostních podmínek. Práce také popisuje kompletní řetězec zpracování multimodálních dat, které autor vytvořil během svého doktorského studia. To Zahrnuje vývoj unikátního senzorického zařízení, které je vybavené řadou snímačů běžně užívaných v robotice. Dále autor popisuje proces tvorby rozsáhlého, veřejně dostupného datasetu Brno Urban Dataset. Na závěr autor popisuje software, který vznikl během jeho studia a jak je tento software užit při zpracování dat v rámci jeho práce (Atlas Fusion a Robotic Template Library).