Dans le domaine de l'Interaction Homme-Machine, nos travaux concernent l'interaction multiutilisateur et multimodale. Exploitant les avancées importantes en terme de modalités d'interaction, de nombreux collecticiels reposent sur une combinaison de modalités, comme les systèmes multiutilisateurs autour d'une table augmentée. Néanmoins ces systèmes constituent des réalisations ad-hoc en l'absence d'outils de conception. Face à constat, nos contributions conceptuelles et logicielles concernent la phase de spécification de l'interaction multiutilisateur et multimodale. Nous présentons une notation de spécification notée COMM (COllaborative et MultiModale) qui est une extension de CTT par l'introduction de deux concepts, le rôle interactif et la tâche modale, et par la spécialisation des opérateurs temporels, en s'appuyant sur les relations de Allen. La notation COMM est instrumentée par un éditeur logiciel de spécifications COMM, nommé e-COMM ((http://iihm.imag.fr/demo/editeur/). COMM et e-COMM ont été utilisés dans un projet DGA de taille conséquente pour spécifier l'interaction au sein d'un poste de commande de drones militaires. / Multi-user multimodal interactive systems involve multiple users that can use multiple interaction modalities. Although multi-user multimodal systems are becoming more prevalent (especially multi-user multimodal systems involving multi-touch surfaces), their design is still ad-hoc without properly keeping track of the design process. Addressing this issue of lack of design tools, our doctoral research is dedicated to the specification of multi-user multimodal interaction. The doctoral research contributions include the COMM (Collaborative and MultiModal) notation and its on-line editor for specifying multi-user multimodal interactive systems. Extending the CTT notation, the salient features of the COMM notation include the concepts of interactive role and modal task as well as a refinement of the temporal operators applied to tasks using the Allen relationships. The COMM notation and its on-line editor e-COMM (http://iihm.imag.fr/demo/editeur/) have been successfully applied to a large scale project dedicated to a multimodal military command post for the control of unmanned aerial vehicles (UAV) by two operators.

Le sujet de cette thèse de doctorat consiste à proposer une architecture ambiante d’interaction homme-robot-environnement. Dans le cadre de la robotique d’aide à la personne dépendante. Cette architecture va permettre aux robots « Ubiquitous Networked Robots » de prendre en compte le contexte évolutif pour fournir continuellement du service à l'utilisateur. L'architecture proposée utilise le concept d'Ontologie du domaine pour la description de l'environnement. Nous avons choisi d'utiliser l'outil open source PROTEGE qui va nous permettre de définir l'ontologie ainsi que les moteurs de fusion et de fission. Les entrées multimodales seront fusionnées puis subdivisées en tâches élémentaires et envoyées comme commandes au fauteuil roulant muni d'un bras manipulateur. Cette architecture sera validée par des spécifications et des simulations via des réseaux de Pétri temporels et stochastiques. / The subject of this thesis is to provide an ambient architecture for the human-robotenvironment interaction, as part of thedependent person robotics help. This architecture will enable the robot to take into account the changing context and continually provide a service to the user. The architecture uses the concept of ontology for the descriptionof the environment. We have chosen to use the open source PROTEGE because it allows the definition of the ontology and the fusion and fission engines. Indeed, multimodal inputs will be merged and subdivided into elementary tasks and sent tocontrol the wheelchair with the manipulated arm. This architecture will be validated by specifications and simulations via temporal and stochastic Petri nets.

The purpose of the research is to identify how pupils and teachers experience different kind of study material in school. It is also important to know which factors affect these experiences. This is what my study will examine. The key-factor to be able to interpret different kinds of multimodal literacy is to understand its textual genre. Pupils and teachers need a certain level of technological skills to be able to create multimodal text and to have an estimation of the purpose of different text genres. To analyze the research material a hermeneutic method is used with a phenomenographic focus of the experience that the survey data and interviewed informants provide for the research. Results shows that some experience is similar between the teachers and pupils concerning the different study materials, then again preference might differ when it comes to preparations for tasks or lectures. Something all informants agree upon, is that the computer as digital tool is positive since it is used in every subject in school and it is never lost or forgotten at home by pupils.

Os processos de representação do conhecimento químico permeiam os mais diversos processos de ensino, aprendizagem e construção acadêmica da Química. Assim, nas múltiplas formas de se representarem fenômenos e nos mais diversos modos de se expressar sobre eles, foram caracterizadas diferentes mídias escritas, visuais e audiovisuais produzidas por alunos em uma sequência didática sobre o tema Transformações. As categorias que emergiram da análise dos materiais foram inspiradas na Gramática do Design Visual (GDV) proposta por Gunther Kress e Theo van Leeuwen e pela perspectiva da Semiótica Social de Michael Halliday. Assim, adota-se no trabalho o enfoque da abordagem multimodal da comunicação, segundo o qual diversos modos orquestrados (textos, imagens, gestos, sons e outros) potencializam a carga de significados de uma representação. Neste sentido, as análises desta Dissertação confirmam que as representações são mais informativas quando os diversos modos se complementam em objetos audiovisuais, propiciando características dinâmicas e conceituais mais amplas do que aquelas possíveis nas expressões escritas e em imagens estáticas. Na perspectiva do ensino de Química, os resultados deste trabalho permitem concluir que as abordagens multimodais, ao facilitarem a expressão das representações mentais dos alunos, podem ter grande importância para a ressignificação de múltiplos aspectos associados a elas e para incentivar a discussão coletiva das ideias, contribuindo para a melhoria da aprendizagem nesse campo do conhecimento. / Representation of the chemical knowledge underlies the processes of teaching and learning Chemistry as well as its development and communication. There are multiple forms of representing the chemical phenomena in the most diverse ways: written texts, visual and audiovisual media, drawings. In the present work, we developed a Teaching/Learning Sequence to High School students concerning Transformations in a multimodal approach. The analysis of the material produced by the students was done on the grounds of the Grammar of Visual Design (GDV) proposed by Gunther Kress and Theo van Leeuwen and in the perspective of Social Semiotics. The categories that emerged from this analysis indicate that the multimodal approach facilitates the communication of the students because orchestrated modes of expression (texts, images, gestures, sounds and others) potentiate the meanings of a representation. In this sense, the results of this Dissertation confirm that the representations are more informative when the diverse modes complement each other in audiovisual objects, providing broader conceptual and dynamic characteristics than those possible in written expressions or in still images. From the perspective of teaching chemistry, the results of the multimodal approach implemented during this graduate work show that it is advantageous, by facilitating the expression of students´ mental representation. This aspect is essential for the re-signification of multiple elements associated with the representations as promotes the discussion of ideas, contributing to the improvement of learning in this field of knowledge.

CoordenaÃÃo de AperfeiÃoamento de Pessoal de NÃvel Superior / FundaÃÃo Cearense de Apoio ao Desenvolvimento Cientifico e TecnolÃgico / Este trabalho tem o objetivo geral de descrever os gÃneros multimodais animados (GEMUA), considerando seus aspectos enunciativos e sua natureza multimodal. A pesquisa caracteriza-se epistemologicamente como interpretativa, de base empÃrica e de carÃter indutivo no sentido lato do termo. Ã luz dos pressupostos de Bakhtin (2006, 2003, 2002) sobre a relaÃÃo sujeito e linguagem e gÃneros discursivos, dos estudos sobre multimodalidade de Kress & van Leeuwen (1996), Kress (2010) e das discussÃes teÃricas acerca dos elementos nÃo-linguÃsticos na comunicaÃÃo humana de Fiorillo e Torres (2003), Cestero (1999), Poyatos (1994), Davis (2009), Albaladejo (2007) e Marcushi (1986), esta pesquisa se concentra em buscar respostas para a seguinte problemÃtica: como se caracterizam os gÃneros multimodais animados? O estudo do corpus, gÃneros multimodais animados, mostrou que ele apresenta dois possÃveis caminhos de observaÃÃo: o eixo da composiÃÃo e o eixo do funcionamento. A tese concentra a anÃlise no eixo da composiÃÃo, cujos dados mostraram que os GEMUA precisam ser analisados partindo de aspectos (i) linguÃsticos, (ii) sonoros, (iii) proxÃmicos, (iv) cinÃsicos e (v) imagÃticos. A maneira encontrada para operacionalizar os parÃmetros de observaÃÃo foi elaborar um fichamento analÃtico e submeter cada uma das 150 amostras do corpus discretizando cada aspecto composicional. A anÃlise mostrou que os GEMUA sÃo gÃneros que apresentam composiÃÃo altamente dinÃmica e plÃstica, alÃm de indicar a rica potencialidade desses gÃneros em mÃltiplos contextos sÃcio-discursivos.

O objetivo desta dissertação é analisar o processo de visualidade, quando da leitura de textos multimodais publicados em jornais brasileiros. A hipótese de nosso trabalho é que, contrário à maneira tradicional de abordar a imagem como documento, isto é, com seu significado estático e inerente a ela, acreditamos que a imagem é uma construção sócio histórica, carregando traços de seu contexto de produção; como tal a leitura e a interpretação dessa imagem serão influenciadas pelo contexto sócio histórico no qual se inserem seus intérpretes/produtores. O trabalho consistiu na contraposição entre análises de imagens jornalísticas e conceitos da Semiótica, Análise de Discurso, Antropologia Visual e Comunicação Visual. / The objective of this dissertation is to analyze visual multimodal texts published in Brazilian newspapers. The hypothesis is that, contrary to the traditional approach to the image as a document with a fixed static and inherent meaning, we believe that the image is a socio historic construction carrying traces of its context of production; as such the reading and interpretation of the image is influenced by the socio-historic context to which its interpreters/producers belong. Our methodology consisted of counterposing the analysis of journalistic images and concepts from Semiotics, Discourse Analysis, Visual Anthropology and Visual Communications.

A presente tese tem por objeto de estudo a construção do discurso publicitário da era digital sob a ótica da Retórica Clássica e da Nova Retórica em sinergia com construtos teóricos da Linguística Aplicada e com a Semiótica peirceana e teoria bakhtiniana. Esta investigação tem como corpora exemplares dos portais de notícias online dos jornais Estado de São Paulo, Folha de São Paulo e O Globo, além de páginas de sites correlatos ao assunto investigado. O cenário discursivo analisado são as eleições municipais de 2012 à prefeitura de São Paulo, durante o período do segundo turno disputado entre Fernando Haddad do PT e José Serra do PSDB. Levando em consideração este contexto, o estudo objetivou elucidar as características dos gêneros emergentes do meio digital empregados na construção do discurso publicitário, suas características multimodais, dispositivos semióticos harmonizados e as estratégias argumentativas articuladas em prol da construção do éthos de ambos os candidatos na busca da adesão dos espíritos do eleitorado. Para percorrer este caminho, os saberes clássicos aristotélicos e as reflexões contemporâneas de Perelman, Peirce e Bakhtin foram de extrema relevância, resultando na compreensão e interpretação dos fenômenos discursivos contemplados, transcendendo o labor meramente taxonômico rumo à compreensão das experiências colaterais que fundamentam a construção de sentido proveniente das relações dialógicas entre os elementos enunciativos e, por fim, revelando os aspectos ideológicos e sociais contidos na trilogia retórica da atividade discursiva investigada. A metodologia construída ao longo deste trabalho foi desenvolvida a partir de diferentes procedimentos fundamentados no arsenal teórico estudado e nas necessidades interpretativas iminentes, na medida em que as abduções despontavam, tinha início um processo constante de deduções e induções. Um movimento contínuo entre os juízos analíticos e o sintético a priori e a posteriori, ou seja, um processo ensaístico em estado evolutivo permanente na busca pelo sentido, ou pelo que Peirce denominava como retórica ou metodêutica o estudo da transferência de informação e dos métodos de pesquisa a serviço da busca pela Verdade. / The following thesis seeks to analyze the construction of the discourse of the advertising used in the digital age. The discourse is built on Classic and New Rhetoric, in conjunction with theoretical constructs of Applied Linguistics and Peirces Semiotic and Bakhtinian theory. The study draws on online news portals of O Estado de São Paulo, Folha de São Paulo and O Globo, as corpora in addition to pages of sites considered pertinent to the subject under investigation. The research context is the second round of 2012 São Paulo municipal elections between candidates Fernando Haddad of the Worker´s Party (in portuguese, PT) and José Serra of the Brazilian Social Democratic Party (PSDB). With that in mind, the study aimed to elucidate the characteristics of emerging genres of digital media employed in the construction of advertising discourse, its multimodal characteristics, harmonized semiotic devices and the argumentative strategies articulated in favor of the construction of the ethos instilled by both candidates to win the heart of faithful constituents. The classic Aristotelian precepts and the contemporary reflections of Perelman, Peirce and Bakhtin were extremely relevant, resulting in the understanding and interpretation of the contemplated discursive phenomena, transcending the basic taxonomic study towards an understanding of collateral experiences that underlie the construction of meaning emanated from the dialogical relations between the enunciative elements, revealing the ideological and social aspects contained in the rhetoric trilogy of the discursive activity investigated. The methodology generated in the course of the thesis developed from different procedures based on the desired theoretical framework and its imminent interpretative requirements. A constant process of deduction and induction was set in motion as abductions became evident. This resulted in a continuous movement between pre- and post- analytical and synthetic judgment, in other words, in an essay-like process in permanent evolutionary state in search of meaning, or what Peirce referred to as rhetoric or methodeutic, the study of the transfer of information and research methods at the service of the search for the Truth.

Philosophiae Doctor - PhD / The thesis explores the role of semiotic structuring of health information in relation to language, multimodality and health literacy and the affordances for agentive participation among consumers of two leading South African medical schemes - Discovery Health Medical Scheme (Discovery Health) and the Government Employees Medical Scheme (GEMS). The focus is on who has access to health information, how this information is constructed and what the semiotic health habitat looks like for citizen-consumers. Through a virtual ethnographic approach the thesis explores the design of genres of health information artefacts: application forms, application guides, a comic book, and a variety of website images. The choice to study the commercial package of a private health industry is aimed at finding and defining codes of practice in health communication that could be replicable in the public health sector. A new perspective emerging out of the thesis is how semiotic structuring of style, stance-taking, and choice of registers affects reading positions, and how these determine with what voice citizenconsumers can engage with this information.

This thesis explores the role of multimodality in language learners’ comprehension, and more specifically, the effects on students’ audio-visual comprehension when different orchestrations of modes appear in the visualization of vodcasts. Firstly, I describe the state of the art of its three main areas of concern, namely the evolution of meaning-making, Information and Communication Technology (ICT), and audio-visual comprehension. One of the most important contributions in the theoretical overview is the suggested integrative model of audio-visual comprehension, which attempts to explain how students process information received from different inputs. Secondly, I present a study based on the following research questions: ‘Which modes are orchestrated throughout the vodcasts?’, ‘Are there any multimodal ensembles that are more beneficial for students’ audio-visual comprehension?’, and ‘What are the students’ attitudes towards audio-visual (e.g., vodcasts) compared to traditional audio (e.g., audio tracks) comprehension activities?’. Along with these research questions, I have formulated two hypotheses: Audio-visual comprehension improves when there is a greater number of orchestrated modes, and students have a more positive attitude towards vodcasts than traditional audios when carrying out comprehension activities. The study includes a multimodal discourse analysis, audio-visual comprehension tests, and students’ questionnaires. The multimodal discourse analysis of two British Council’s language learning vodcasts, entitled English is GREAT and Camden Fashion, using ELAN as the multimodal annotation tool, shows that there are a variety of multimodal ensembles of two, three and four modes. The audio-visual comprehension tests were given to 40 Spanish students, learning English as a foreign language, after the visualization of vodcasts. These comprehension tests contain questions related to specific orchestrations of modes appearing in the vodcasts. The statistical analysis of the test results, using repeated-measures ANOVA, reveal that students obtain better audio-visual comprehension results when the multimodal ensembles are constituted by a greater number of orchestrated modes. Finally, the data compiled from the questionnaires, conclude that students have a more positive attitude towards vodcasts in comparison to traditional audio listenings. Results from the audio-visual comprehension tests and questionnaires prove the two hypotheses of this study.

Cette thèse propose une exploration synchronique du paradigme des quotatifs en anglais oral, suite à l’avènement dans ce groupe fonctionnel de deux nouvelles formes, GO et BE LIKE, apparues il y a moins de cinq décennies. Elle s’inscrit dans la continuité de plusieurs études antérieures visant à rendre compte du système quotatif de l’anglais depuis les phases précoces de ce changement linguistique désormais avéré. À partir d’un corpus de données originales, cette thèse examine les différentes fonctions pragmatiques, interactionnelles et discursives des quotatifs afin de mettre au jour certains facteurs de distribution de ces marqueurs en discours. L’approche adoptée est à la fois polysystémique, pluri-théorique et multimodale : la langue est un système complexe, dans lequel plusieurs sous-systèmes interagissent pour participer à la construction de l’interaction. Afin de rendre compte de cette multiplicité, l’application conjointe de plusieurs systèmes d’analyse et de plusieurs approches théoriques est indispensable. Cette thèse tient également compte de la dimension multimodale de la langue orale. Elle examine, qualitativement et quantitativement, les fonctions accomplies par les quotatifs en termes de représentation, de réitération d’événements antérieurs et de performance. Elle propose également un tour d’horizon des diverses stratégies narratives attribuables aux quotatifs lorsqu’ils participent conjointement à la construction de séquences dialoguées. Ce travail montre qu’en définitive, ce n’est que par la combinaison de plusieurs grilles d’analyse qu’il est possible de réduire le nombre d’événements imprévisibles dans les productions de la langue orale en interaction. / This thesis gives a synchronic account of the quotative paradigm in oral English, subsequently to the recent arrival of the new introducers GO and BE LIKE less than five decades ago. It stands in the wake of earlier studies that have aimed at analysing the quotative system of English since the earliest phases of this recently attested change in progress. The study was carried out on a corpus of original data, the Cambridge Student Corpus, which contains semi-guided dyadic conversations between British and Irish native speakers. This thesis examines, from an empirical point of view, the various pragmatic, interactional and discourse functions of quotatives in order to shed light on their complementary distribution in oral interaction. The chosen approach is polysystemic, multi-theoretic and multimodal. Language is a complex system, within which several sub-systems interact to participate in building spoken interaction. In order to account for this complexity, it is necessary to apply several methods of analysis and various linguistic theories simultaneously. This thesis also takes into account the multimodal aspects of oral interaction. It gives a qualitative and quantitative account of quotatives with respect to their functions of representation, reiteration of prior events and multimodal performance. It also examines various narrative strategies that these markers may implement as they participate together in the elaboration of sequences of dialogue. This work illustrates the notion that only through a combination of different methods can the analyst reduce the number of unexplained events that occur in oral interaction.