Este trabalho tem como objetivo discutir o estado da arte e propor melhorias em uma área de grande potencial e de crescimento recente na computação: a análise do estado emocional do usuário de um sistema de informação e a aplicação dessa observação em cenários ricos de interação homem-máquina. Para foco em um escopo tratável, escolheu-se o estudo das técnicas atuais de captação afetiva através de três sensores diferentes (expressões faciais, expressões vocais e contexto semântico) e de como sua combinação poderia resultar em dados mais robustos sobre o estado emocional do usuário, em uma aplicação conhecida como Fusão Multimodal. Nesse sentido, o trabalho levanta não apenas a bibliografia recente sobre o estado da arte da aquisição de dados nas três áreas e em fusão, mas também aplica algumas das ferramentas existentes (tanto comerciais como abertas) para melhor compreensão do nível atual da tecnologia acessível ao público acadêmico. Dada a inexistência de uma aplicação facilmente disponível para análise de emoções no contexto semântico, desenvolveu-se uma ferramenta dedicada à pesquisa, nomeada emoCrawler, a qual utiliza redes sociais para avaliação do caráter emocional que palavras indiretamente podem acarretar em uma discussão. O emoCrawler busca avaliar as emoções evocadas em usuários de Twitter e as compara com um dicionário emocional conhecido, baseado em contribuições acadêmicas diversas, gerando um resultado quantitativo útil ao escopo do trabalho. Por fim, o trabalho lançou mão da ferramenta emoCrawler e das outras ferramentas selecionadas para análise de expressões faciais e vocais na criação de um sistema de fusão sensorial que aumente a confiabilidade global da emoção detectada. Para isso, inicialmente um corpus de áudio e vídeo disponível foi tratado e reduzido para o escopo da pesquisa. Em seguida, diferentes técnicas de fusão baseadas em algoritmos de classificação foram aplicadas (redes neurais, Naive Bayes e Support Vector Machine), com abordagens distintas de treinamento para busca de resultados melhores dos que os encontrados nas aplicações unimodais. Os resultados verificados de fato indicam para uma melhor taxa de detecção quando é aplicado o algoritmo de fusão. Nesse processo, a presença de dados fornecidos pelo emoCrawler parece ter contribuído positivamente. / This work aims at discussing the state of the art and proposing improvements in a field of great potential and recent growth in computing: the analysis of the emotional state of the user of an information system and the application of such observations in rich scenarios of human-machine interaction. In a tractable scope of research, this work evaluates the current techniques of affective observation applied on three different inputs (facial expressions, speech features and semantic context) and comprehends how their combination can result in more robust affirmations on the actual emotional status of the user an application called Multimodal Fusion. In this sense, the work presents not only the recent literature on the state of the art in data acquisition in the three aforementioned areas and in fusion, but also applies some of the existing tools (both commercial and open) for a better understanding of the current level of technology available to the academic audience. Given the lack of an application readily available for the analysis of semantic context in emotions, a dedicated piece of software to the research was developed, entitled emoCrawler, which uses social networks as a means to evaluate the emotional signature that words may leave in a discussion. emoCrawler aims at evaluating emotions evoked in Twitter users and comparing them with an emotion dictionary built around academic contributions, generating a quantitative result useful to this research. Finally, this work made use of emoCrawler and the other selected tools for both facial and speech expressions in the creation of a sensor fusion system to enhance the global reliability of the affective detection. To achieve this, an audiovisual database was initially processed and condensed. Then, different fusion techniques based on classification algorithms with distinct learning approaches (neural networks, naïve Bayes and Support Vector Machine) were applied in the pursuit of better outcomes than the unimodal applications. The final results indeed point to a better rate of detection when fusion algorithms are applied. In this process, the data provided by emoCrawler seems to have contributed positively.

Several recent maritime accidents suggest that modern technology sometimes can make it difficult for mariners to navigate safely. A review of the literature also indicates that the technological remedies designed to prevent maritime accidents at times can be ineffective or counterproductive. To understand why, problem-oriented ethnography was used to collect and analyse data on how mariners understand their work and their tools. Over 4 years, 15 ships were visited; the ship types studied were small and large archipelago passenger ships and cargo ships. Mariners and others who work in the maritime industry were interviewed. What I found onboard were numerous examples of what I now call integration work. Integration is about co-ordination, co-operation and compromise. When humans and technology have to work together, the human (mostly) has to co-ordinate resources, co-operate with devices and compromise between means and ends. What mariners have to integrate to get work done include representations of data and information; rules, regulations and practice; human and machine work; and learning and practice. Mariners largely have to perform integration work themselves because machines cannot communicate in ways mariners see as useful. What developers and manufacturers choose to integrate into screens or systems is not always what the mariners would choose. There are other kinds of ‘mistakes’ mariners have to adapt to. Basically, they arise from conflicts between global rationality (rules, regulations and legislation) and local rationality (what gets defined as good seamanship at a particular time and place). When technology is used to replace human work this is not necessarily a straightforward or successful process. What it often means is that mariners have to work, sometimes very hard, to ‘construct’ a cooperational human-machine system. Even when technology works ‘as intended’ work of this kind is still required. Even in most ostensibly integrated systems, human operators still must perform integration work. In short, technology alone cannot solve the problems that technology created. Further, trying to fix ‘human error’ by incremental ‘improvements’ in technology or procedure tends to be largely ineffective due to the adaptive compensation by users. A systems view is necessary to make changes to a workplace. Finally, this research illustrates the value problem-oriented ethnography can have when it comes to collecting information on what users ‘mean’ and ‘really do’ and what designers ‘need’ to make technology easier and safer to use.

Este trabalho tem como objetivo discutir o estado da arte e propor melhorias em uma área de grande potencial e de crescimento recente na computação: a análise do estado emocional do usuário de um sistema de informação e a aplicação dessa observação em cenários ricos de interação homem-máquina. Para foco em um escopo tratável, escolheu-se o estudo das técnicas atuais de captação afetiva através de três sensores diferentes (expressões faciais, expressões vocais e contexto semântico) e de como sua combinação poderia resultar em dados mais robustos sobre o estado emocional do usuário, em uma aplicação conhecida como Fusão Multimodal. Nesse sentido, o trabalho levanta não apenas a bibliografia recente sobre o estado da arte da aquisição de dados nas três áreas e em fusão, mas também aplica algumas das ferramentas existentes (tanto comerciais como abertas) para melhor compreensão do nível atual da tecnologia acessível ao público acadêmico. Dada a inexistência de uma aplicação facilmente disponível para análise de emoções no contexto semântico, desenvolveu-se uma ferramenta dedicada à pesquisa, nomeada emoCrawler, a qual utiliza redes sociais para avaliação do caráter emocional que palavras indiretamente podem acarretar em uma discussão. O emoCrawler busca avaliar as emoções evocadas em usuários de Twitter e as compara com um dicionário emocional conhecido, baseado em contribuições acadêmicas diversas, gerando um resultado quantitativo útil ao escopo do trabalho. Por fim, o trabalho lançou mão da ferramenta emoCrawler e das outras ferramentas selecionadas para análise de expressões faciais e vocais na criação de um sistema de fusão sensorial que aumente a confiabilidade global da emoção detectada. Para isso, inicialmente um corpus de áudio e vídeo disponível foi tratado e reduzido para o escopo da pesquisa. Em seguida, diferentes técnicas de fusão baseadas em algoritmos de classificação foram aplicadas (redes neurais, Naive Bayes e Support Vector Machine), com abordagens distintas de treinamento para busca de resultados melhores dos que os encontrados nas aplicações unimodais. Os resultados verificados de fato indicam para uma melhor taxa de detecção quando é aplicado o algoritmo de fusão. Nesse processo, a presença de dados fornecidos pelo emoCrawler parece ter contribuído positivamente. / This work aims at discussing the state of the art and proposing improvements in a field of great potential and recent growth in computing: the analysis of the emotional state of the user of an information system and the application of such observations in rich scenarios of human-machine interaction. In a tractable scope of research, this work evaluates the current techniques of affective observation applied on three different inputs (facial expressions, speech features and semantic context) and comprehends how their combination can result in more robust affirmations on the actual emotional status of the user an application called Multimodal Fusion. In this sense, the work presents not only the recent literature on the state of the art in data acquisition in the three aforementioned areas and in fusion, but also applies some of the existing tools (both commercial and open) for a better understanding of the current level of technology available to the academic audience. Given the lack of an application readily available for the analysis of semantic context in emotions, a dedicated piece of software to the research was developed, entitled emoCrawler, which uses social networks as a means to evaluate the emotional signature that words may leave in a discussion. emoCrawler aims at evaluating emotions evoked in Twitter users and comparing them with an emotion dictionary built around academic contributions, generating a quantitative result useful to this research. Finally, this work made use of emoCrawler and the other selected tools for both facial and speech expressions in the creation of a sensor fusion system to enhance the global reliability of the affective detection. To achieve this, an audiovisual database was initially processed and condensed. Then, different fusion techniques based on classification algorithms with distinct learning approaches (neural networks, naïve Bayes and Support Vector Machine) were applied in the pursuit of better outcomes than the unimodal applications. The final results indeed point to a better rate of detection when fusion algorithms are applied. In this process, the data provided by emoCrawler seems to have contributed positively.

Made available in DSpace on 2016-04-04T18:27:47Z (GMT). No. of bitstreams: 1 Marcos de Toledo Benassi.pdf: 2077139 bytes, checksum: fb6293f043c90566fbb8dea6710e711c (MD5) Previous issue date: 1997-06-24 / Pontif?cia Universidade Cat?lica de Campinas / In our contemporary society the computer has been used in a wide range of activities, in which man can set up relationships with this machine. This man-machine interactions have been studied by several sciences, that contribute with the understanding of these interactions, aiming to build better computer software s interfaces. The hypertext is a kind of software that can organize the information in a non-linear way, using a particular interface to allow the access to its contents. The aim of this research was to understand how subjects interact with a hypertext system created for this work, in order to determine the style of interaction: linear, non-linear or a blend of both. In addition, another aim was to correlate these categories of interactions with answers gave by the subjects to some questions, in order to verify if some relationship can be established among the quality of the answers and the interaction style. Using the X2 test was possible to conclude that the group of subjects used the non-linear style of interaction. The Spearman s rs coefficient found no significant correlations among the quality of subjects answers and the interaction style. / O computador tem se introduzido na sociedade contempor?nea em uma grande variedade de atividades, nas quais o homem com ele estabelece rela??es, quer sejam de trabalho que de lazer. Estas intera??es homem-m?quina tem sido objeto de estudo de um conjunto de ci?ncias, que contribuem para a compreens?o de como estas intera??es se processam, visando aprimorar as interfaces oferecidas pelos programas de computador para sua utiliza??o. Um dos muitos tipos de programas dispon?veis ? o hipertexto, que permite uma modalidade n?o-linear de organiza??o de informa??es, provendo uma interface peculiar para o acesso ao seu conte?do. Este trabalho teve como objetivo a compreens?o das modalidades de intera??o estabelecidas por sujeitos com um hipertexto criado para esta pesquisa, visando determinar se esta seria linear, n?o-linear ou indiferenciada. Adicionalmente, foram tra?adas correla??es entre quest?es respondidas pelo sujeitos visando verificar a exist?ncia de rela??es entre sua qualidade e a modalidade predominante de intera??o do grupo de sujeitos com o produto. Atrav?s do registro automatizado das intera??es e posterior an?lise estat?stica utilizando-se a prova de X2, verificou-se que o grupo de sujeitos estabeleceu, no contexto experimental desta pesquisa, uma intera??o predominantemente n?o-linear, interagindo com o hipertexto explorando a peculiaridade da m?dia. Utilizando-se o coeficiente Rn de Spearman foi tra?ada a correla??o entre este padr?o predominante de intera??o e a qualidade da resposta, n?o se obtendo resultados estatisticamente significativos.

Traditional design theories have focused on technical functions and more or less disregard a product’s user involvement. The existing methods of ergonomic designare mostly intended for analysis activities. There is a need for new dynamic methods that focus on user-product interactions. The aim of this research work is to develop design methods for user-product interactions, which should support synthesis activities in early product development phases. An observation study and a questionnaire survey were carried out in order to investigate product developers’ work and relation to the users for providing background information about the research problem. Furthermore, student projects in product development were followed, giving essential input. After the theories and methods were developed, a retrospective interview study was carried out in order to confirm the need for the developed methods. The studies showed, for instance, that companies use few formal methods and almost none of these are directed towards the user. It is also indicated that the product developers’ contact with users decreases with increasing company size. Few companies have a defined procedure for defining their intended users. Six methods are developed. They embrace three ways of classifying the users and their relations to products and other users (User identification, Use profile and User relations), an analysis of the users’ Activities, goals and motives behind their use of the product, a scenario technique (User-technical process scenario, UTPS), which shows the user process in parallel with the technical process, and a hierarchical decomposition of technical functions and user actions, which is named the Functionaction tree (FAT). All the methods, apart from FAT, were tested in real product development teams. All the tested methods stimulate communication between the group members of various competencies in the design group. Most of the methods are easy to apply and are valuable for understanding the design problem. The UTPS is also useful for comparing design solutions and generates new ideas about the design task. The other tested methods did not generate many new ideas, but the reason is probably that they were mainly tested on products that are already on the market. Thus, the methods are most valuable in the early design stages, when trying out a product idea or a concept. / QC 20100618

Orientador: Hermes Renato Hildebrand / Tese (doutorado) - Universidade Estadual de Campinas, Instituto de Artes / Made available in DSpace on 2018-08-22T09:23:29Z (GMT). No. of bitstreams: 1 Gregolin_MairaValencise_D.pdf: 3943869 bytes, checksum: e510258e58d7bae6614659520ba86238 (MD5) Previous issue date: 2012 / Resumo: Esta tese tem como tema central as articulações entre a comunicação móvel e a cultura participativa e suas consequências sobre as transformações da ação social contemporânea. Para investigar essas articulações, traçamos como objetivo geral compreender as inter-relações entre o desenvolvimento das mídias digitais, o fenômeno da convergência e a emergência da sociedade participativa. A fim de efetivar esse objetivo geral, tomamos como objeto de estudos dois fenômenos típicos da cultura participativa, denominados por Jenkins (2008) como "convergência corporativa" e "convergência alternativa". Do primeiro fenômeno, analisamos a telenovela Viver a Vida, exibida pela Rede Globo de Televisão (2009-2010); do segundo, analisamos o ativismo digital ou ativismo transmídia durante a Primavera Árabe - a fim de estudar as conexões entre a comunicação móvel e a emergência de práticas de ação social, no interior de uma sociedade participativa. A discussão inicia-se com o delineamento das transformações no desenvolvimento das mídias, focalizando as mutações tecnológicas que operaram a passagem dos dispositivos fixos aos móveis. A análise da telenovela Viver a Vida evidenciou mudanças no formato tradicional desse gênero televisivo, instaurando-se uma narrativa que se desenvolve entre a televisão e o meio digital; ao mesmo tempo, transforma-se a relação entre os consumidores e a telenovela, com novas formas de se assistirem aos capítulos (que migram para o computador e os dispositivos móveis) e de interagir com a narrativa (o telespectador pode acompanhar a história que se desenrola complementarmente tanto na televisão quanto nos sites e blogs). Quanto à análise do ativismo digital durante a Primavera Árabe, salientamos sua inextricável ligação com as redes sociais digitais e analisamos os deslocamentos nas práticas de ativismo político propiciadas pelo uso de mídias móveis e do compartilhamento de fotos e textos via smartphone. A análise mostra que os ativistas conseguem burlar a censura imposta aos meios de comunicação tradicionais e dão visibilidade aos acontecimentos, denunciando crimes por meio de imagens chocantes, que expõem ao olhar do mundo as atrocidades cometidas pelos governos ditatoriais. Usando desses mesmos recursos da telefonia móvel e das redes sociais digitais, os governos ditatoriais postam imagens e textos que exaltam seus feitos e condenam as ações dos manifestantes e rebeldes, travando uma batalha materializada nas imagens e textos que circulam nas redes sociais e que dão dimensão planetária aos acontecimentos. O compartilhamento de imagens e textos, via smartphone, salienta novas práticas estabelecidas pelo uso do aplicativo Instagram, tanto em termos de recursos de tratamento das imagens, quanto no que diz respeito à instantaneidade de postagem de fotografias e textos (legendas, etiquetas) em plataformas e redes sociais digitais. O uso do Instagram está modificando a produção e circulação de textos e imagens realizadas por fotógrafos amadores, na medida em que oferece recursos para refinamento das imagens realizadas pelo smartphone, tratadas pelo aplicativo e postadas em plataformas e redes sociais. O uso desse aplicativo está modificando também as práticas de produção no campo do fotojornalismo, sendo cada vez mais utilizado por fotógrafos profissionais de agências de notícias e bancos de imagens, instalando nova estética para a produção fotográfica na imprensa. Demonstramos, assim, que na ecologia contemporânea das mídias, deslocam-se as fronteiras entre o político e o estético e a comunicação móvel se transforma em arena para novas formas de ação social / Abstract: This thesis is focused on the links between mobile communication and participatory culture and their consequences on the contemporary transformations of social action. To investigate these articulations, we traced as the general objective to understand the inter-relationships between the development of digital media, the phenomenon of convergence and the emergence of participatory society. In order to accomplish this general goal, we take as the object of study two phenomena typical of participatory culture, called by Jenkins (2008) as "corporate convergence" and " alternative convergence". For the first phenomenon, we analyze the soap opera "Viver a Vida", aired by Globo Television Network (2009-2010). For the second, we investigate the digital activism or transmedia activism during the Arab Spring - to study the connections between mobile communication and the emergence of social activism within a participatory society. The discussion begins with the outlining of the changes in media development, focusing on the technological development that operated the transition from fixed to mobile devices. The analysis of the soap opera "Viver a Vida" showed changes in the traditional format of this television genre, establishing up a narrative that develops between television and digital media; while transforming the relationship between consumers and the soap opera, with new forms of watching chapters (which migrate to the computer and mobile devices) and interact with the narrative (the viewer can follow the story as it unfolds on television, complemented by the websites and blogs). As we analyze the digital activism during the Arab Spring, we emphasize its inextricable link with social networks and observe the shifts in practices of political activism enabled by the use of mobile media, by sharing photos and texts via smartphone. The analysis shows that activists can circumvent the censorship imposed on traditional media and provide visibility to events, denouncing crimes through shocking images that expose to the world the atrocities committed by dictatorial governments. Using the same mobile and social resources, dictatorial governments post images and texts that exalt their accomplishments and condemn the actions of the protesters and rebels, waging a battle materialized in texts and images circulating on social networking and giving global dimension to the events. The sharing of images and texts via smartphones highlights new practices established by using the Instagram app, both in terms of resources for processing images, as in the case of the instant posting of photos and text (captions, tags) on platforms and digital social networks. The use of Instagram is changing the production and circulation of texts and images taken by amateur photographers, as it provides image refinement features for smartphones, handled by the application and posted on social networks and platforms. The use of this application is also modifying production practices in the field of photojournalism, being increasingly used by professional photographers from news agencies and image banks, introducing new aesthetics for the photographic production in the press. We demonstrate, therefore, that the ecology of contemporary media move the boundaries between the political and the aesthetic, as mobile communication becomes arena for new forms of social action / Doutorado / Artes Visuais / Doutora em Artes

Orientadores Eleri Cardozo, Diogo Coutinho Soriano / Dissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de Computação / Made available in DSpace on 2018-08-24T05:15:26Z (GMT). No. of bitstreams: 1 SuarezUribe_LuisaFernanda_M.pdf: 6173860 bytes, checksum: 3a1e038c4b2f0b3d7f6c4981c268de63 (MD5) Previous issue date: 2013 / Resumo: Os potenciais evocados visuais de estado em regime permanente (SSVEP) são uma resposta cerebral medida ao capturar sinais cerebrais através de Eletroencefalograma (EEG) usando-se uma touca com eletrodos posicionados no escalpo mediante estimulação visual externa. Um sistema que detecta a resposta SSVEP gerada em um sujeito estimulado com fontes de luz (LEDs) piscando a frequências diferentes foi desenvolvido nesta dissertação com o intuito de criar uma interface homen-máquina. Para tanto, foram usados quatro estímulos visuais codificando quatro possíveis comandos, mais o estado de repouso, a serem classificados e identificados a partir da resposta observada no EEG de um sujeito e da devida análise espectral deste sinal. As características que determinam se a resposta SSVEP está presente foram estudadas através de diferentes heurísticas para a seleção de descritores (coeficientes associados a densidade espectral de potência), os quais foram posteriormente passados para um classificador linear para a determinação do comando associado ao estímulo. Para a seleção de características, o índice Davies Bouldin (DB) foi utilizado. No que concerne a análise de frequência realizado para os três sujeitos estudados foi possível observar a presença da resposta SSVEP nas frequências de estimulação, mas tal como esperado, com forte presença de ruído e com grande variabilidade entre os sujeitos. Apesar da variabilidade das características selecionadas pela estratégia adotada, obteve-se aqui resultados de classificação próximos a 90% de acerto para cada classe. Estes resultados de classificação indicaram que esta metodologia de processamento pode ser usada num sistema de BCI em tempo real, dado que o atual sistema foi projetado para operar somente com dados que foram gravados off-line / Abstract: Steady State Visual Evoked Potentials (SSVEP) is a brain response measured by capturing brain signals generated by external visual stimulation, through electroencephalogram (EEG) using a cap with electrodes placed on the scalp. A system for detection SSVEP response generated in a subject stimulated with light sources (LEDs) flashing at different frequencies has been developed in this dissertation in order to implement a human machine interface. Thus, we used four visual stimuli encoding four possible commands, besides idle state, to be classified and identified from the response observed in the EEG of a subject through the spectral analysis of this signal. The characteristics that determine if a SSVEP response is present were studied through different heuristics for selecting descriptors (coefficients associated with power spectral density), which were then passed to a linear classifier for the determination of the stimulus associated command. Davies Bouldin (DB) index was the method used for the selection of features. From the frequency analysis carried out for three subjects it was observed the presence of SSVEP response on the stimulation frequencies, but in some cases, as expected, was blurred by noise, which differ among individuals. Despite the variability of the features selected by the strategy adopted, it was obtained classification results with accuracy nearly 90\% for each class. These classification results indicated that this processing methodology could be used in a BCI system in real time, as the current system was designed to operate only with data that were recorded off-line / Mestrado / Engenharia de Computação / Mestra em Engenharia Elétrica

Nous modélisons une chaîne de traitement de documents comme un processus de décision markovien, et nous utilisons l’apprentissage par renforcement afin de permettre à l’agent d’apprendre à construire des chaînes adaptées à la volée, et de les améliorer en continu. Nous construisons une plateforme qui nous permet de mesurer l’impact sur l’apprentissage de divers modèles, services web, algorithmes, paramètres, etc. Nous l’appliquons dans un contexte industriel, spécifiquement à une chaîne visant à extraire des événements dans des volumes massifs de documents provenant de pages web et d’autres sources ouvertes. Nous visons à réduire la charge des analystes humains, l’agent apprenant à améliorer la chaîne, guidé par leurs retours (feedback) sur les événements extraits. Pour ceci, nous explorons des types de retours différents, d’un feedback numérique requérant un important calibrage, à un feedback qualitatif, beaucoup plus intuitif et demandant peu, voire pas du tout, de calibrage. Nous menons des expériences, d’abord avec un feedback numérique, puis nous montrons qu’un feedback qualitatif permet toujours à l’agent d’apprendre efficacement. / Information extraction (IE) is defined as the identification and extraction of elements of interest, such as named entities, their relationships, and their roles in events. For example, a web-crawler might collect open-source documents, which are then processed by an IE treatment chain to produce a summary of the information contained in them.We model such an IE document treatment chain} as a Markov Decision Process, and use reinforcement learning to allow the agent to learn to construct custom-made chains ``on the fly'', and to continuously improve them.We build a platform, BIMBO (Benefiting from Intelligent and Measurable Behaviour Optimisation) which enables us to measure the impact on the learning of various models, algorithms, parameters, etc.We apply this in an industrial setting, specifically to a document treatment chain which extracts events from massive volumes of web pages and other open-source documents.Our emphasis is on minimising the burden of the human analysts, from whom the agent learns to improve guided by their feedback on the events extracted. For this, we investigate different types of feedback, from numerical rewards, which requires a lot of user effort and tuning, to partially and even fully qualitative feedback, which is much more intuitive, and demands little to no user intervention. We carry out experiments, first with numerical rewards, then demonstrate that intuitive feedback still allows the agent to learn effectively.Motivated by the need to rapidly propagate the rewards learnt at the final states back to the initial ones, even on exploration, we propose Dora: an improved version Q-Learning.

La reconnaissance d'émotions est un domaine jeune mais dont la maturité grandissante implique de nouveaux besoins en termes de modélisation et d'intégration dans des modèles existants. Ce travail de thèse expose un modèle conceptuel pour la conception d'applications interactives sensibles aux émotions de l'utilisateur. Notre approche se fonde sur les résultats conceptuels issus de l'interaction multimodale: nous redéfinissons les concepts de modalité et de multimodalité dans le cadre de la reconnaissance passive d'émotions. Nous décrivons ensuite un modèle conceptuel à base de composants logiciels s'appuyant sur cette redéfinition: la branche émotion, facilitant la conception, le développement et le maintien d'applications reconnaissant l'émotion de l'utilisateur. Une application multimodale de reconnaissance d'émotions par la gestuelle a été développée selon le modèle de la branche émotion et intégrée dans un système d'augmentation de spectacle de ballet sensible aux émotions d'un danseur. / Computer-based emotion recognition is a growing field which develops new needs in terms of software modeling and integration of existing models. This thesis describes a conceptual framework for designing emotionally-aware interactive software. Our approach is based upon conceptual results from the field of multimodal interaction: we redefine the concepts of modality and multimodality within the frame of passive emotion recognition. We then describe a component-based conceptual model relying on this redefinition. The emotion branch facilitates the design, development and maintenance of emotionally-aware systems. A multimodal, interactive, gesture-based emotion recognition software based on the emotion branch was developed. This system was integrated within an augmented reality system to augment a ballet dance show according to the dancer's expressed emotions.

Au-delà des outils d’analyse statistique traditionnellement mobilisés par les chercheurs en humanités numériques, il existe de nombreux instruments de visualisation (bases de données 3D, moteurs de jeu vidéo, réalité virtuelle et augmentée, etc.) qui ont gagné ces dernières années une importance croissante, ceux-là permettant de spatialiser les connaissances de façon intuitive, de donner à voir des vestiges en lien avec les enquêtes humanistes ou encore de toucher un public dépassant le strict milieu universitaire. Cette thèse a pour but de préciser sur la base de quels modèles heuristiques et épistémologiques, les ingénieurs travaillant dans le domaine des interactions homme-machine ont pensé ces dispositifs. En Europe et jusqu’à l’époque baroque, plusieurs méthodes, désignées du nom d’arts de la mémoire, reposant sur l’immersion dans des espaces mentaux, avaient été mobilisées pour organiser les connaissances. Leur centralité dans la vie intellectuelle leur a donné une postérité double : elles ont structuré tant la manière de penser l’organisation des savoirs (logique et computation) que de les donner à voir (avènement du paradigme perspectif et de la muséologie). Cet héritage hybride les a constituées comme des repères pour le développement des nouveaux médias, notamment de l’infographie tridimensionnelle. Pour saisir les enjeux socio-économiques et esthétiques liés à l’utilisation de ces technologies dans le champ universitaire, leur mise au point dans les secteurs industriels du cinéma et du jeu vidéo, qui ont nourri leur évolution, est resituée dans la thèse. Cette enquête permet de définir des prescriptions utiles à leur déploiement au sein des humanités numériques. / Beyond the instruments of statistical analysis enlisted by researchers working in the field of digital humanities, a growing number of visual and spatial tools (3D databases, video game engines, virtual and augmented reality, etc.) have gained in interest over the past few years. These new approaches allow for the spatialized presentation of knowledge in novel and intuitive ways that increasingly appeal to a public beyond academia. The goal of this thesis is to pinpoint and explore the various heuristic and epistemological models engineers working in the field of human-computer interaction have used to develop these tools. Up until the Baroque period in Europe, a suite of methods grouped under the rubric of “the arts of memory” relied on similar modes of spatialization of immersive mental spaces in order to activate memory and to organize knowledge. The centrality of these arts in the formation of intellectual life has granted them a double posterity: they at once structured the way one organized knowledge – thus becoming instrumental in the refinement of logic and computation; while also serving as means to exhibit and present knowledge – allowing for the advent of the perspective paradigm and museology. This hybrid heritage establishes the arts of memory as landmarks for the further development of new media, and notably computer-generated images. To understand the stakes involved with the use of these technologies in scholarly practice, this thesis also addresses how the film and video game industries gradually shaped their evolution. Finally, this inquiry provides useful prescriptions for their deployment within the growing interdisciplinary field of digital humanities.