No ensino médio, o conceito de interações interpartículas é sistemático e classificatório em detrimento de um tratamento mais crítico acerca do sistema estudado. A correta compreensão deste conceito é importante para que o aluno possa explicar diversas propriedades físicas das substâncias (MENDES, 2007). Nesta perspectiva, a abordagem multimodal oferece suporte aos métodos de avaliação, elaboração de estratégias e pesquisas na área de ensino, pois o uso de diversas linguagens facilita a compreensão do objeto de estudo pelos seres humanos (MAYER, 2003). Neste trabalho são analisadas as potencialidades e as limitações dos modelos dos estudantes de ensino médio em interações interpartículas e suas interfaces com as propriedades físicas das substâncias em um minicurso oferecido na Faculdade de Educação da Universidade de São Paulo (USP). Os modelos expressos pelos estudantes são fomentados pela metodologia de ensino e aprendizagem do minicurso, a qual é suportada pelo uso de diversos modos de comunicação nos moldes da abordagem multimodal (MAYER, 2003). Os modelos são analisados de acordo com a frequência de acionamento de variáveis explicativas utilizadas pelo estudante para explicar e/ou predizer determinado fenômeno no contexto de resolução de problemas, enquanto que as limitações explicativas do modelo são contabilizadas qualitativamente via análise de conteúdo. Os resultados apontam que o uso de múltiplos modos de comunicação e informação se mostrou eficiente no fomento da produção discursiva em uma sala de aula heterogênea e auxiliou os estudantes durante o processo de aprendizagem, pois os modelos expressados evoluíram ao longo do minicurso em termos de complexidade, com cada vez menos limitações. / In high school, the concept of interparticle interactions is systematic and classificatory as opposed to a more critical treatment on the studied system. The correct understanding of this concept is important for the student to explain the physical properties of substances (MENDES, 2007). In this perspective, the multimodal approach supports the methods of evaluation, strategizing and research in the area of education, since the use of different languages facilitates the understanding of the study object by humans (MAYER, 2003). This paper analyzes the potential and limitations of high school students\' models in interparticle interactions and their interfaces with the physical properties of substances in a short course offered at the Faculdade de Educação at Universidade de São Paulo (USP). The models expressed by students are fostered by the teaching and learning methodology of the course, which is supported by the use of different modes of communication in the molds of multimodal approach (MAYER, 2003). The models are analyzed according to the frequency of use of explanatory variables used by the student to explain or predict a given phenomenon in problem solving context, while the explanatory limitations of the model are counted qualitatively through content analysis. The results appoint that the use of multiple modes of communication and information are efficient in fostering the discursive production in a heterogeneous classroom and helped students during the learning process, because the expressed models evolved over the course in terms of complexity and with fewer limitations.

Os documentos oficiais e a concretização das propostas no livro didático analisado abordam a ambiguidade lexical como algo a ser evitado e corrigido através de estratégias de desambiguação, ignorando a importância da natureza polissêmica da língua como recurso expressivo na comunicação cotidiana e como instrumento fundamental do humor, especialmente as variantes de humor popular e infantil. O presente trabalho busca abordar a ambiguidade lexical, polissemia e homonímia, no gênero tirinha cômica, texto multimodal e humorístico, visando observar a eficácia da proposta de leitura e produção como exercício de contextualização, levando os alunos a observarem, explicarem e produzirem texto, utilizando-se dos diversos significados possíveis das palavras em uso, em situação concreta de enunciação. A proposta se dá dentro do conceito de multimodalidade, trabalhando a ambiguidade lexical e o letramento visual, objetivando que o aluno não apenas observe e compreenda as tirinhas apresentadas, mas também que produza a sua própria, compondo o texto com os elementos verbal e visual. As atividades foram divididas em dez passos: pesquisa e compartilhamento de HQs, atividade diagnóstica, análise da ambiguidade lexical em tirinha cômica no livro didático, jogo digital Quem ri seus males espanta Piadas, consulta às diferentes acepções no dicionário, escrita do glossário de unidades lexicais polissêmicas ou homônimas, elaboração do roteiro da tirinha, autoavaliação, produção final de uma tirinha cômica e apreciação dos trabalhos das turmas. / The analised official papers and textbook proposals approach lexical ambiguity as something to be avoided and correct through desambiguation strategies, ignoring the importance of the polissemic nature of language as an expressive resource of the language in daily communication and as a fundamental tool of comedy and humor, especially popular and childrens jokes and puns. This paper aims to approach lexical ambiguity, polissemy and homonyms, within the comic strip genre, which is a multimodal and comic text genre, with the objetive of observing how effective this proposal of reading and writing can be to help students comprehend, learn and explain lexical ambiguity, by using a word in multiple possible meanings in the comic strip, in concrete enunciation. This proposal uses multimodality, by combining lexical ambiguity and visual litteracy, aiming to enable the student not only to observe and understand the comic strips presented, but also to produce their own by using verbal and visual components. The sequence is divided in ten steps: research and sharing comics, diagnostic evaluation, lexical ambiguity in the textbook, digital game Quem ri seus males espanta Piadas, searching for different meanings of words in the dictionary, writing a glossary of polissemic unities, production of a scratch comic, self evaluation, final production and appreciation of the classes works.

Orientador: Roxane Helena Rodrigues Rojo / Dissertação (mestrado) - Universidade Estadual de Campinas, Instituto de Estudos da Linguagem / Made available in DSpace on 2018-08-23T14:44:34Z (GMT). No. of bitstreams: 1 Santos_FrancineEloisa_M.pdf: 9241322 bytes, checksum: 0ab99f2484b0f48f594c1aad424c976a (MD5) Previous issue date: 2013 / Resumo: Essa dissertação de mestrado discute a questão da pedagogia dos multiletramentos na nova Proposta Curricular do Estado de São Paulo, implantada em 2008, e investiga como se dá a concretização dessa nos Cadernos do Professor de Língua Portuguesa do Ensino Fundamental/Ciclo II. Com base na teoria da pedagogia dos multiletramentos e diante dos novos desafios da contemporaneidade dos textos que apresentam uma multiplicidade de linguagens, semioses e mídias; perguntamo-nos se (e como) a Proposta Curricular do Estado de São Paulo abordaria essa questão e se essa também estaria presente nos Cadernos do Professor. Temos como corpus dessa pesquisa, então, os 16 Cadernos do Professor de Língua Portuguesa do Ensino Fundamental II produzidos pela Secretaria de Educação do Estado de São Paulo e publicados em 2009. Olhamos também neste trabalho para a questão da diversidade cultural, como e se é abordada nesta Proposta Curricular, também sob a ótica dos multiletramentos / Abstract: This dissertation discusses the pedagogy of multiliteracies in the new Curriculum Proposal of the State of São Paulo, deployed in 2008, and investigates how it works in the textbooks for Portuguese Language Teacher of Elementary Education / Cycle II. Based on the theory of multiliteracies pedagogy and facing the new challenges of contemporary texts that present a multiplicity of languages, semioses and media, we wonder if (and how) the Curricular Proposal of the State of São Paulo would address this issue and if this would also be present in the teacher's textbooks. We have a corpus of this research, then, the 16 books of the Portuguese Language Teacher of Elementary Education II produced by the Education Department of the State of São Paulo and published in 2009. In this dissertation, we also look at the question of cultural diversity, and how it is addressed in this Curriculum Proposal, also from the perspective of multiliteracies / Mestrado / Lingua Materna / Mestra em Linguística Aplicada

This thesis is concerned with exploring political Internet Memes within the discourse of the debate leading up to the Swiss referendum “For the Effective Expulsion of Foreign Criminals” (Durchsetzungsinitiative). During the campaign, an unprecedented amount of political agitation occurred online, which resulted in the distribution of a number of Internet Memes. A corpus of 25 Memes with content opposing the referendum was built and analyzed from a medialinguistic perspective with focus on their multimodal nature and their argumentation. As research on political Memes in the German language area has previously not been conducted, this study fills the need to apply German Bildlinguistik and multimodality theory on the investigation of a current online-phenomenon. To this purpose, the theoretical tools of multimodality theory, political communication studies and discourse analysis were applied. While the focus of this analysis lies on the collected data, the role of political memes within Web 2.0 in general were also investigated. The investigation of the corpus shows that images are just as important to the building of an argument as textual elements are and should be given equal investigative attention. Political Internet Memes are very well integrated into their medium, namely Web 2.0. They actively engage the readers in their argumentation with a number of semiotic resources, such as vernacular language or vectors pointing at the reader. This makes Memes very apt instruments within the online political discourse.

Multimodal theory has developed out of social semiotics and can be seen as a response to the rise in the use of new technologies for the creation, distribution and consumption of media texts and the need to find new ways of describing and explaining their role in representation and communication. Its development is historical. It is a response to change over time. The incorporation of the visual into social semiotics marks a key moment in the development of multimodal theory. Visual literacy is discussed in relation to changes in modes of representation and a critique of this concept is provided. This is conducted in relation to how the visual modality has been integrated into social semiotics as a platform for research into multimodal communication more generally. Framing is developed along three main lines of enquiry (semiotic, cognitive and affective) as alternative ways of accounting for some of these shifts in communication and each are presented in the form of case studies. Framing and its close relationship with composition in media texts is discussed and this understanding, one that emphasise proximity as a multimodal principle, is applied to the visual design of content, the realisation of context through the provision visual cues, and later to embodiment and urban space. The three case studies, the application of framing to a range of media texts, the critical judgements made about the role visual in contemporary theory and the application of these concepts to multimodality are presented as part of an intellectual journey.

Les interfaces homme-machine (IHM) multimodales offrent à l’utilisateur la possibilité de combiner les modalités d’interaction afin d’augmenter la robustesse et l’utilisabilité de l’interface utilisateur d’un système. Plus particulièrement, en sortie, les IHM multimodales permettent au système de restituer à l’utilisateur, l’information produite par le noyau fonctionnel en combinant sémantiquement plusieurs modalités. Dans l’optique de concevoir de telles interfaces pour des systèmes critiques, nous avons proposé un modèle formel de conception des interfaces multimodales en sortie. Le modèle proposé se décompose en deux modèles : le modèle de fission sémantique qui décrit la décomposition de l’information à restituer en informations élémentaires, et le modèle d’allocation qui spécifie l’allocation des modalités et médias aux informations élémentaires. Nous avons également développé une formalisation B Événementiel détaillée des deux modèles : fission sémantique et allocation. Cette formalisation a été instanciée sur des études de cas puis généralisée dans un processus de développement B Événementiel cadre dans lequel s’inscrivent les modèles de fission sémantique et d’allocation. Cette formalisation a permis de procéder à la vérification de propriétés de sûreté, de vivacité et d’utilisabilité. / Multimodal Human-Computer Interfaces (HCI) offer to users the possibility to combine interaction modalities in order to increase user interface robustness and usability. Specifically, output multimodal HCI allow system to return to the user, the information generated by the functional core by combining semantically different modalities. In order to design such interfaces for critical systems, we proposed a formal model for the design of output multimodal interfaces. The proposed model consists of two models: the semantic fission model describes the decomposition of the information to return into elementary information and the allocation model specifies the allocation of the elementary information with modalities and media. We have also developed a detailed Event B formalization for the two models: semantic fission and allocation. This formalization has been instantiated on case studies and generalized in an Event B development process framework including semantic fission and allocation models. This formalization allows to carry out safety, liveness and usability properties verification.

Philosophiae Doctor - PhD / This thesis is a multimodal discourse analysis of the media representations of Zimbabwean immigrants in the South African media. The aim of the investigation is to illustrate how Zimbabwean immigration and Zimbabwean immigrants are portrayed in the print media in South Africa. For the theoretical and analytical framework, the study mainly combines critical discourse analysis (CDA) (Fairclough, 1992, 1995, Van Dijk, 1988, 1991, 1993, Wodak, 1999) and multimodal analysis approaches (Kress and Van Leeuwen, 2006). These approaches are augmented by insights from the cognitive theory of metaphor (Lakoff and Johnson, 1980) and from sociologists such as Stanley Cohen (1972) who have researched on moral panic. The study analyses the coverage of Zimbabwean immigrant stories in the South African print media from the year 2000 to date. A total of five hundred and seventy five articles were randomly selected from the SA Media Database. The SA Media Database is a comprehensive collection of all newspapers published in South Africa. These data are supplemented by articles from newspaper cuttings collected during the course of the study. The focus of the analysis is on how the arrival of Zimbabwean immigrants is ‘problematized’ and transformed into a discursive crisis through the construction of anti-immigrant metaphors and generation of a moral panic. The data are grouped into emerging themes, and data analysis is guided by a multimodal critical discourse analysis approach in which the verbal and visuals are read as text. Although the findings of this study generally support earlier studies which argue that the media representation of Zimbabwean immigrants is negative, the multimodal analysis suggests a more balanced and positive image. Thus, although this study supports studies that show that media discourses represent Zimbabwean immigrants as ‘others,’ and often as criminals, the multimodal analysis of the images of Zimbabwean immigrants suggests that media discourse is much more complex. In the media, the Zimbabwean immigrants are presented as either victims or abusers of the system in South Africa. This reflects a broader discourse on migration which constructs Zimbabweans as ‘aggressors and victims.’ Through discourses of moral panic, the analysis of metaphors, the representation of female immigrants, and the multimodal analysis of language and visual data, this thesis shows an extensive deployment of discursive strategies used for the representation of us and them, characteristic of media discourse on migration. On the other hand, the South African media, through visual images, portrays a sympathetic view toward Zimbabwean immigrants and their difficulties. Thus, while the study supports Woods and King (2001), who note that media discourses represent immigrants as ‘others’ and often as criminals, the multimodal analysis of the images of Zimbabwean immigrants tend to be multivocal, in that they tend to also depict migrants as victims of circumstances beyond their control. Therefore, one conclusion that can be made is that the verbal and visual texts in the South African media do not always tell the same story. In some cases, two or more stories are being told at the same time. The study also concludes that multimodality offers the tools through which the different voices, some of which are contradictory, can be read and heard. Images evoke readers’ schemas and frames of experiences, for instance, of pain and human suffering. In fact, visual images are presented as authentic and objective pieces of evidence, not as representations of reality, but, in a sense, as reality itself (Dauber, 2001).Thirdly, the images of Zimbabwean immigrants convey additional information, beyond the journalist’s intention. With images, readers can quickly elicit a strong emotional response. This is different from a textual description. Therefore, images are not only excellent communicators, but also quickly affect us mentally and emotionally. In this study, for instance, images of Zimbabwean immigrants in long queues at the immigration department enduring cold weather, or bloodied victims of attack, or an image of a helpless Zimbabwean child eating from an empty platter seemingly abandoned and alone, by design or default, draw sympathy from the readers. Because multimodal images tend to tell more than one story, such images can also elicit anger and resentment from the readers.Lastly, this study contributes to our understanding of Zimbabwean immigration to South Africa by focusing on how the media multimodally constructs representations of Zimbabwean immigrants in the South African media. Thus, this study also fills a gap that exists in the study of the interplay between verbal texts and immigration images in the media in African contexts; and specifically contributes to the understudied representations of immigrants in South Africa. Another significant contribution is that this is the only study that has combined cognitive theory of metaphor, the sociological notion of moral panic, critical discourse analysis and current theorization on multimodality to comprehensively account for media representations of immigrants. This study points to the need for a multi-semiotic approach to the analysis of the verbal texts and images of immigration in the South African media for a comprehensive appreciation of the relationship between the verbal and visual texts.

L'utilisation d'images de modalités différentes est très répandue dans la résolution de problèmes liés aux applications de la télédétection. La raison principale est que chaque image d'une certaine modalité contient des informations spécifiques qui peuvent être intégrées en un modèle unique, afin d'améliorer notre connaissance à propos d'une scène spécifique. A cause du grand volume de données disponibles, ces intégrations doivent être réalisées de manière automatique. Cependant, un problème apparaît dès les premiers stades du processus : la recherche, dans des images de modalités différentes, de régions en correspondance. Ce problème est difficile à résoudre car la décision de regrouper des régions doit nécessairement reposer sur la part d'information commune aux images, même si les modalités sont différentes. Dans cette thèse, nous nous proposons donc d'apporter une contribution à la résolution de ce problème / The use of several images of various modalities has been proved to be quite useful for solving problems arising in many different applications of remote sensing. The main reason is that each image of a given modality conveys its own part of specific information, which can be integrated into a single model in order to improve our knowledge on a given area. With the large amount of available data, any task of integration must be performed automatically. At the very first stage of an automated integration process, a rather direct problem arises : given a region of interest within a first image, the question is to find out its equivalent within a second image acquired over the same scene but with a different modality. This problem is difficult because the decision to match two regions must rely on the common part of information supported by the two images, even if their modalities are quite different. This is the problem that we wish to address in this thesis

Interpreting studies has so far tended to concentrate on simultaneous interpreting over the consecutive mode. Note-taking – an integral part of consecutive interpreting – has therefore received very little scholarly attention. As an indispensable tool in consecutive interpreting, note-taking plays an important role in supporting the interpreter’s memory. This study argues, however, that the interpreter's notes should not be viewed merely as a memory storage tool, but as a third visual language with its own logic and meaning-making practices that need interpreting. The way in which interpreters read their notes is explored here from the perspective of Social Semiotics for two reasons. Firstly, Social Semiotics conceptualises signs as meaning-making resources which are realized in specific communicative contexts to convey specific communicative intentions – unlike previous approaches to note-taking, that have tended to categorise signs as static constituents of relatively finite sign codes. Secondly, Social Semiotics not only accounts for how written language is used in notes, but also how the pictorial component of communication is encoded and interpreted through interpreter’s notes. The interpreter, as a viewer, has to make use of semiotic resources deployed in the notes in order to reconstruct the information given by the speaker and to produce the target speech for the audience. Therefore, the interpreters’ note-reading stage, based on the interaction between signs, can be conceptualised by reference to the concept of visual grammar. This study draws on visual grammar (Kress and van Leeuwen 2006) to analyse interpreter’s notes with a view to gaining a better understanding of how linguistic and visual semiotic resources are deployed in the process of note-taking. Insight into interpreters’ meaning-making practices and note-taking patterns is gained through an experimental study of the notes produced by nine qualified, practising conference interpreters, during a consecutive interpreting task from English into Chinese. The patterns identified in my data set are then compared with the established prescriptive approaches to note-taking training – which are typically based on relatively stable correspondences between note-taking signs/symbols and their meaning. The analysis focuses on certain elements of the source speech (concepts that can be noted down through the use of vectors, geometrical shapes, specific classificational structures, margin, and salience) as reflected in the notes. The way in which interpreters read their notes involves the interaction between two core modes, such as image and language, and a range of sub-modes, such as vectors, geometrical shapes, composition, framing, salience and calligraphy. The results of the analysis indicate that the way in which interpreters arrange the contents of their notes reflects the depth of the information processing effort required by the note-taking process. The findings suggest that the narrative structure in notes seems to assist interpreters in retrieving information at a micro, lexical level, whereas the visual structure would appear to assist interpreters in retrieving information at a macro, contextual level, e.g. in representing the hierarchies of information value, constructing the structure of rendition, and showing the importance of specific signs.

A suposta ubiquidade de sistemas decomponíveis foi interpretada por Holland (1975) como o principal motivo para o desempenho dos algoritmos genéticos (Genetic Algorithms (GAs)). A hipótese de Building Blocks (BBs) sugere que algoritmos genéticos mais eficientes poderiam ser implementados, contudo, apenas anos depois essas ideias puderam ser avaliadas experimentalmente no contexto de algoritmos de estimação de distribuição (Estimation of Distribution Algorithms (EDAs)). EDAs utilizam modelos probabilísticos, estimados a partir da população, para inferir características do espaço de busca que poderiam ser utilizadas para implementar operadores de reprodução mais eficazes. Tanto em problemas mono- quanto multi-objetivo, EDAs emergiram sob a premissa de que a eficácia dos operadores de reprodução seria proporcional à representatividade dos modelos probabilísticos utilizados. No entanto, estudos recentes tem demonstrado que a dificuldade em se construir modelos confiáveis pode tornar essa premissa inviável. Ou seja, para certos problemas de otimização os modelos probabilísticos utilizados seriam, em geral, de baixa qualidade e, portanto, não produziriam operadores eficazes. Esta tese trata das limitações encontradas na construção de modelos probabilísticos (linkage learning) sob a perspectiva da multimodalidade dos problemas em questão. A análise teórica considerou problemas aditivamente separáveis, enquanto a generalização das conclusões foi investigada em instâncias do modelo NK-landscapes e do problema da mochila multidimensional (Multidimensional Knapsack Problem (MKP)). Os resultados indicaram que a acurácia dos modelos probabilísticos é se relaciona inversamente ao grau de multimodalidade da função objetivo e que, em casos de extrema multimodalidade a construção de modelos probabilísticos confiáveis pode ser tornar infactível. Este resultado poderia inviabilizar o uso de EDAs no contexto multiobjetivo, devido a intrínseca multimodalidade de tais problemas. No entanto, observou-se que apesar da ausência de estatísticas confiáveis sobre cada uma das funções objetivo, a correlação entre elas se torna estatisticamente observável e útil aos operadores de reprodução na manutenção da diversidade e controle convergência da população. / The supposed ubiquity of nearly-decomposable systems was interpreted by Holland (1975) as the rationale for the performance of Genetic Algorithms (GAs), the Building Block (BB) hypothesis. His seminal studies suggest more efficient GAs as viable, but only later on his ideas have become practically tangible in the context of Estimation of Distribution Algorithms (EDAs). EDAs employ probabilistic modeling so as to infer properties of the search space (BBs) that could be useful for the effectiveness of reproduction operators. In both, single- and multi-objective contexts, EDAs have emerged on the assumption there is a correlation between how much information a model can conceive and how effective reproduction operators can be. However, more recent results suggest the difficulties in producing accurate linkage models can prevent such a relation to be true. In other words, for some optimization problems linkage learning might not be able to produce accurate linkage models, hence EDAs would not outperform GAs. This thesis addresses the limits of linkage learning in the context of single- and bi-objective problems, regarding the influence of multimodality on the accuracy of the linkage models and the efficiency of EDAs. A theoretical analysis was performed in terms of additively separable functions and general conclusions are assessed through experimentation with instances of the NK-model and the Multidimensional Knapsack Problem (MKP). The results indicated that the accuracy of the linkage models tends to decrease as a result of increasing multimodality, which weakens pairwise dependencies and might lead to pairwise independence in extreme cases. Since most EDAs rely on bivariate statistics to estimate multivariate distributions, their applicability is limited to optimization problems within a certain range of multimodality. In multi-objective problems, on the other hand, some EDAs have shown better performance than GAs, which seemed as a contradiction since multi-objective problems are inherently multimodal. Our results suggest that in such cases the correlation among the objective functions becomes statistically evident, as a consequence, linkage learning models such correlation instead of problems substructures, which is useful to obtain a better exploration of extreme regions of the objective space.