Using a multimodal approach to literacy, this thesis explores student dialogue and responses from viewing wordless and image-rich books to answer the following questions: What impact does visual literacy instruction have on students’ learning and achievement in reading and viewing comprehension? What are grade five students’ perspectives on being involved in reading and viewing wordless and image-rich books in the classroom? Action research methodology is employed by the teacher researcher in a grade five classroom. Qualitative data sourced from whole class observations, small group reading interviews, student work samples, a colleague’s observational notes, and quantitative data from reading assessments reveal insights into the affordances of presenting visually rich texts to students. The author focuses the discussion on reading comprehension strategies, visual elements, and the experience of reading wordless and image-rich books. He concludes that students are able to use deeper level reading comprehension strategies and articulate their understanding while viewing images.

The present paper focusses on the genre of recipes, its structure and multimodality. The aim of this thesis is to present the genre of recipes from theoretical perspective and to make a practical analysis about the structure of recipes and multimodality based on Oliver’s and Nicholson’s recipes. For the practical part Jamie Oliver’s recipe book Happy Days with the Naked Chef, Beata Nicholson’s recipe book Beatos Virtuvė: Kepinių Knyga and their websites are analysed. Recipes for the analysis are chosen randomly and qualitative method is used as all of them are described and compared in order to find differences and similarities. The paper presents the concept of genre and genre analysis that is discussed by many authors and explained in great details. Further, the genre of recipes and its historical change is looked at thus the influence on the structure of recipes made by new innovations can be noticed. In addition, the ideas suggested by various authors show that this genre is structured and has clearly distinguished parts that should be included in every recipe. Finally, the issue of multimodality is presented and it is of special importance today since not only textual but also visual information is found in books and web-sites. The structural analysis of Oliver’s and Nicholson’s recipes supports the ideas of various scholars that this genre is highly structured. The main difference is found between the recipes in printed and online sources. In recipe books the... [to full text] / Šiame darbe apžvelgiamas receptų žanras, jo struktūra ir multimodalumas. Darbo tikslas yra pristatyti receptų žanro teoriją ir analizuoti receptų struktūrą ir multimodalumą, remiantis Džeimio Oliverio ir Beatos Nicholson receptais. Praktinėje dalyje analizuojama Oliverio receptų knyga „Happy days with the naked chef“ ir Nicholson receptų knyga „Beatos virtuvė: Kepinių knyga“ bei šių autorių internetiniai puslapiai su receptais. Receptai yra pasirinkti atsitiktinai ir kokybinė analizė yra pritaikyta, nes jie visi yra apibūdinami ir palyginami tam, kad būtų galima atrasti skirtumus ir panašumus. Šiame darbe pristatomas ir išsamiai apibūdinamas daugelio autorių aptartas žanro terminas ir žanro analizė. Taip pat apžvelgiamas receptų žanras ir jo istorinė kaita ir čia galima pastebėti, kad patobulinimai technikos srityje padarė įtaką receptų struktūrai. Be to, įvairių autorių mintys, kaip turėtų atrodyti receptai, atskleidžia, kad šis žanras turi aiškią struktūrą ir jo dalys gali būti gana tiksliai apibrėžtos. Galiausiai yra pristatoma multimodalumo tema, kuri šiandien yra labai svarbi, nes vaizdinės informacijos yra vis dažniau pateikiama knygose ir internetiniuose puslapiuose. Oliverio ir Nicholson receptų, esančių knygose ir internetiniuose puslapiuose, struktūros analizė patvirtina įvairių autorių mintis, kad šis žanras turi aiškias dalis. Daugiausiai skirtumų randama tarp receptų, esančių knygose ir internetiniuose puslapiuose. Receptų knygose tekstas susideda iš šių dalių... [toliau žr. visą tekstą]

Esta é uma investigação em que essencialmente se propõe o uso da combinação de mediadores em complexidade crescente permitindo a prática de diferentes formas de linguagens, as quais explicitam padrões de aprendizagem de saberes sobre química. Dentre os mediadores estão os signos que, se organizando em palavras e expressões, adquirem significados, os quais permitem que fenômenos da Ciência da Natureza sejam circunscritos, compreendidos e trazidos para o mundo dos humanos na forma de tecnologias na perspectiva da melhoria da qualidade de vida. Estas tecnologias que se manifestam na forma de bens e serviços exigem respostas sociais vigorosas quanto ao seu grau de uso e manutenção que demandam mais do que apenas concordância ou discordância, requerem o uso da argumentação fundamentada, a qual encontra na escola seu mais fértil campo de desenvolvimento que não pode se olvidar de recorrer também às tecnologias para construir propostas que auxiliem jovens estudantes nesse percurso. Desse desafio surge a proposta da aprendizagem sobre Química mediada pela multimodalidade, termo a princípio genérico, mas que entendemos como a combinação entre modalidades de explicitação da aprendizagem por gêneros do discurso que circulem em meio impresso e não-impresso (audiovisual), utilizando a linguagem narrativa e expositiva contendo aspectos conceituais e matemáticos, derivados de atividades tais como experimentos, jogos, projetos, construção e operação de aparatos escolar-científicos, dentre outros, e com apoio das tecnologias da informação e comunicação. Nesta investigação o uso da aprendizagem multimodal se materializa ao longo dos dois últimos anos do Ensino Médio regular em ciclos de complexidade crescente em turmas diversas em uma escola pública. A produção dos estudantes ao final de cada estágio foi analisada e classificada na busca da obtenção de padrões da forma de pensamento e do nível de processamento da informação gráfica, bem como de possíveis restrições na resolução de problemas químicos.Os resultados sugerem que a aprendizagem mediada pela multimodalidade na perspectiva apresentada contribui para a evolução das formas de pensamento, bem como do nível de processamento da informação gráfica, entretanto quando da introdução de um novo gênero do discurso escrito, surgem inflexões na curva crescente de aprendizagem revelando dispersão de padrões, a qual indica a necessidade da intervenção do professor com ações de retomada de conceitos e revisão monitorada. As condições de produção e a cultura escolar se mostram como elementos externos com forte poder de restrição ao processo de aprendizagem. / This is an investigation in which essentially it is proposed the use of the combination of mediators in increasing complexity allowing the practice of different forms of languages, which state standards of knowledge learning about chemistry. Among the mediators are the signs that, organizing themselves in words and expressions, acquire meanings, which allow phenomena of the Science of Nature to be circumscribed, understood and brought into the human world in the form of technologies in the perspective of the improvement of the quality of life. These technologies that are manifest in the form of goods and services require vigorous social responses regarding their level of use and maintenance that require more than just agreement or disagreement, they require the use of grounded argumentation, which finds in school its most fertile field of development and one can not forget to also appeal to the technologies to build proposals to assist young students in this course. From this challenge arises the proposal of learning about Chemistry mediated by multimodality, term at first generic, but that we understand as the combination between modalities by explicitation of the learning by speech genres that circulate in printed and unprinted (audiovisual) media, using the narrative and expositive language containing conceptual and mathematical aspects, derived from activities such as experiments, games, projects, construction and operation of educational and scientific apparatus, among others, and with the support of information and communication technologies. In this research the use of multimodal learning materializes along the last two years of regular high school in cycles of increasing complexity in diverse classes in a public school. The production of the students at the end of each stage was analyzed and classified in the search of the obtainment of patterns of thought form and of level of processing graphic information, as well as possible restrictions to solve chemical problems. The results suggest that learning mediated by multimodality in the presented perspective contributes to the evolution of the forms of thought, as well as of the level of processing of graphical information, but when introducing a new genre of written discourse, inflections emerge in the increasing learning curve revealing dispersion of patterns, which indicates the need for teacher intervention with actions of retaking of concepts and monitored review. The production conditions and the school culture show themselves as external elements with strong power of restriction to the learning process.

Magister Artium - MA / This study demonstrates how advertisers re-voice and re-perform others' gestures and actions (Prior and Hengst, 2010). The focus is on the mobility of semiosis across boundaries and practices. It uses Multimodal Discourse Analysis (Kress and Van Leeuwen, 1996, 2001, 2006) Semiotic Remediation/Resemiotization (Iedema, 2003, 2010; Prior and Hengst, 2010) as the theoretical/analytical framework. The idea is to explore how semiotic elements are remediated through intertextual references and multimodality and how semiotic remediation is employed in the process of re-creation and re-purposing of objects and messages in the selected television advertisements. Drawing on MTN, Vodacom, Kentucky Fried Chicken (KFC) and Nando's television advertisements, the study shows how popular, historical, cultural and political discourse is reproduced and re-lived as a creative idea in the selected television advertisements in the process of re-branding. In this regard, resemiotization or semiotic remediation is seen as social practice and an integral part of the marketing strategy in the South African television advertising industry. Upon examination, the study establishes that some selected television advertisements have been extensively re-worked and re-purposed. Therefore, resemiotization and/or semiotic remediation are found to be resourceful tools for the marketing discourse. Thus, the study found that South African advertising discourse depends primarily on societal discourses such as politics, history, cultural traditions and popular culture as its base for creativity. In terms of language use in South African advertising, the study has revealed that television advertisements are moving towards a localised language practice and/or localised English.

Drawing and coloring have been part of young students text making as  longas the writing system has been used, but with the increased use of digital tools and an enlarged focus on accountability of today there is a reinforced educational interest to understand what constitutes multimodal student texts in the context of classroom practice. This thesis project overall aim is to highlight and discuss the opportunities and difficulties in the assessment of language and knowledge-developing multimodal text work in a multilingual educational context.Conceptually the study is grounded in sociocultural theories, in sociosemiotic theory and in multiliteracies research. The methods used consist of qualitative multimodal text analysis and semi-structured interviews with students and their teacher.The three empirical studies were carried out, each having a different perspective. The first study looks at the text production of students in an integrated work of drawing, coloring and writing. The second study focuses how students reason when they choose to draw, write or both draw and write when they report to their teacher what they have learned. The third study discusses what the teacher highlights when assessing her students’ multimodal text productions. Overall, the results show that the semiotic resources, images and color, dominate students’ text productions and that the teacher attaches great importance to the illustrations, but that she, despite the best intentions, has trouble using multimodal criteria when assessing the students’ different ways of expressions and semiotic resources into a whole. It seems to be problematic for the teacher to allow students to freely interpret and independently design the task while she at the same time intends to make an overall assessment of how the content is presented. The results also indicate that it is difficult  for  the  students  to  verbalize  their  thoughts on the assessment and in practice; the teacher more often is focused on assessing abilities relating to how thoroughly the students carry out the process of documenting rather than encouraging the students to develop and express their knowledge. Finally, the thesis concludes with discussing the content of an ongoing need for research, especially regarding the consequences it may have for younger students, whatever language background they have. / <p>Ytterligare delarbeten</p><p>Borgfeldt, E., &amp; Lyngfelt, A. (2017). ”Jag ritade först sen skrev jag”. Elevperspektiv på multimodal textproduktion i årskurs 3. Forskning om undervisning och lärande 2017: 1 vol. 5, s. 64-88. http://www.forskul.se/tidskrift/nummer18/jag_ritade_forst_sen_skrev_jag_ __elevperspektiv_pa_multimodal_textproduktion_i_arskurs_3Borgfeldt, E. (2017).</p><p>Multimodal textproduktion i årskurs 3 – analys av en lärares bedömning. Educare: 2017: 1, s. 118-151. Malmö: Lärande och samhälle, Malmö högskola. https://www.mah.se/upload/FAKULTETER/LS/Dokument%20LS/Educa re%2017.1%20muep.pdf</p>

This dissertation examines the concept of haole, a Native Hawaiian articulation of whiteness, in two multimodal texts related to the proposed construction of the Thirty Meter Telescope on Mauna-a-Wākea in Hawai'i. Specifically, I analyze the Aloha 'Aima mural at the University of Hawai'i and the 'Imiloa web interface, as two examples of delinking whiteness through decolonial enactment. Building on this analysis, this project theorizes a decolonial haole rhetoric by redefining the rhetorical character of whiteness from outside a Eurowestern frame. Haole is an example of decolonial enactment because it responds to cultural rhetorics by engaging with the loci of enunciation rather than attending to meaning. Haole rhetoric is a form of whiteness that responds to Hawai'i's specific history of, and resistance to, colonialism. I argue for white settler allyship that works from the rhetorical ground of colonial wounds to counteract the colonial control of meaning. In this dissertation, I have developed a haole methodology that includes the following tenets: 1) any presence of whiteness manifests coloniality; 2) Indigenous knowledges are always taken on their own terms and never through Eurowestern frames of thought 3) the aim is always to inform meaning-making practices for white settlers, as opposed to Indigenous people; and 4) engagement of cultural rhetorics aim of epistemic disobedience, or delinking, from settler logics. Shifting whiteness from communicative identity to inhabiting whiteness as a decolonial enactment manifests whiteness in the peripheries of decolonial futures. An embrace of decolonial haole rhetoric and methodology maintains the colonial history between white settlers and Native Hawaiians, where rather than feigning innocence, white settlers instead consider histories of colonialism as sites on which to build responsible relationalities that may be productive for efforts of decolonization. This project demonstrates how Indigeneity can inform haole epistemological delinking, and how obligations, friendship, and intimacies are forged in colonial situations. / Doctor of Philosophy / This dissertation examines the concept of haole, a Native Hawaiian word for whiteness, in various texts, such as public murals and web interfaces, related to the proposed construction of the Thirty Meter Telescope on Mauna Kea in Hawai'i. Specially, this project theorizes a decolonial haole rhetoric as a new way to be white that is obligated to of cultural knowledges of place and history of whiteness. I argue for settler allyship that understands whiteness not from its own understanding, but instead from the meanings and knowledges of Hawaiian conceptions of whiteness. In this dissertation, I have developed a methodology that includes the following tenets: 1) any presence of whiteness manifests coloniality; 2) Indigenous knowledges are always taken on their own terms and never made relatable towards Eurowestern thought; 3) the aim is always to inform meaning-making practices for white settlers, never Indigenous people; and 4) engagement of cultural rhetorics aims towards epistemic disobedience, or delinking, from settler logics. A haole rhetoric attempts to rethink whiteness in a way that doesn't double down on whiteness's own understanding of itself, but instead a way of inhabiting whiteness that is obligated and ethical to Hawaiian thought and culture. Embracing this a decolonial haole rhetoric doesn't forgive haoles for colonial wrongs, but instead focuses on colonial wrongs as sites to build relationalites of settler allyship.

This study explores the benefits of multimodality in the drafting process and advocates for more disciplinary support of multimodality across first-year writing curriculums in the field of Rhetoric and Composition/Writing Studies. I explore my primary research question, how might multimodal drafting through the Literacy Sponsors Gallery Wall assignment support students' process of writing the Literacy Narrative, through an IRB-exempt study of the implementation of an original multimodal writing assignment, the Literacy Sponsors Gallery Wall, in my first-year writing classroom during the Fall 2021 semester. The results of this study illuminate valuable opportunities for multimodality to be integrated into the composition classroom during the writing process rather than it being utilized primarily as a transformative tool once the writing process has concluded. When multimodality is included as a natural aspect of the writing process, it allows more room for students to express and celebrate their multiliterate identities. / Master of Arts / Most of the time, writing assignments in academic contexts are limited to page-bound essays of block text. This is because of tradition and the cultural belief that this type of writing is the only mode, or format, worthy of value in a classroom. But that is not necessarily the case. Multimodal writing, i.e., more visually stimulating compositions that combine more than one mode of communication, are generative for student writers. However, multimodality is usually seen as a "last but not least" aspect of a draft's life cycle, meaning it is employed once the draft has been completed and is used to transform the draft into a more visual mode (infographic, poster, etc.). In this paper, I argue that multimodality should be taken up more in the field of Rhetoric and Composition/Writing Studies and embraced as a worthy aspect of the drafting process. I created a qualitative study in which I evaluated how multimodal drafting acts as a beneficial scaffolding tool for teachers.

What is ‘legitimacy’? Is legitimation possible through non-linguistic modes? These are the key theoretical questions with which this study is concerned. It explores them in conjunction with an analysis of American political spot advertisements. These ads are situated at the nexus between legitimation and multimodality, and their relevance to contemporary politics on the world stage is reflected in the immense financial and skilled resources which have been — and continue to be — devoted to them. A historical perspective into legitimation, multimodality and the attendant concepts of rationality and irrationality is given, followed by a discussion challenging the assumed rational role accorded to language. So challenged, the discussion moves to looking at the pairing of multimodality and politics; first from a historical viewpoint, and then from a more contemporary one. The role of myth, in the form of the American Dream, is investigated, leading to discussion of political appropriation, branding, tangibility, affordances and the (im)possibility of restricting interpretation. Spot ads are analysed with a specific focus: first on modal salience, and secondly on how the semiotic richness of the concept of nature is exploited for purposes of legitimation.

Studien syftar till att få en djupare förståelse för hur sångare använder och upplever olika verktyg för att forma och kommunicera musik, vilka verktyg de prioriterar samt vilken funktion dessa verktyg fyller. För att undersöka detta har jag utgått från olika teorier om musikalisk kommunikation samt ett multimodalt och socialsemiotiskt perspektiv med fokus på kommunikation som en social process av teckenskapande. Jag har genomfört åtta kvalitativa intervjuer med sångare som arbetar/arbetat professionellt med sång inom olika genrer. Resultaten visar att sångarnas musikaliska kommunikation formas och uttrycks genom ett användande av flera olika verktyg varav röstteknik, samspel, publikens förväntningar, personlighet och tradition är de viktigaste. Sångarnas användande och prioritering av dessa olika verktyg kan sammanfattas i olika övergripande förhållningssätt som sångarna har till sin musikaliska kommunikation, varav tradition och personlighet är de mest framträdande förhållningssätten. En slutsats är att en sångares musikaliska uttryck kan beskrivas som en väv av kommunikativa strukturer och att sångare strävar efter att gestalta helheter av dessa strukturer för att uppleva mening. En annan slutats är att sångare kan betona olika aspekter av den musikaliska kommunikationen vid olika tillfällen i musicerandet, att den sker på flera plan samtidigt samt att musikens sammanhang avgör hur den musikaliska gestaltningen formas. Jag vill slutligen tillägga att sångarens gestaltande av musikaliska helheter sker genom att skapa, tolka och imitera olika resurser i syfte att skapa en meningsfull kommunikation för både publik, medspelare samt med sångaren själv. / The study aims to demonstrate a deeper understanding in how singers can use different tools to shape and communicate music, and the order of priority and functions of these tools. To attain knowledge about this, I have applied different theories of musical communication, in addition to a multimodal and social semiotic perspective with focus on communication as a social process of sign making. This has been examined through qualitative interviews with eight active and former professional singers in various genres. The results show that the singers use many different tools to shape and communicate music, of which the most predominant are voice-technique, interplay, expectations from the audience, personality and tradition. The singers usage and priority of these tools can be summarised in different overall ways of musical communication, of which tradition and personality are the most conspicuous. One conclusion is that the musical expression of a singer can be described as a weave of communicative structures, and that the singers strive to achieve a wholeness of these structures to yield meaning. Another conclusion is that singers may emphasize different aspects of the musical communication at different stages in a performance, that this may occur at different levels simultaneously, and that the musical context decides how the musical communication is being shaped. Finally, I want to describe the singers’ function as achieving completeness in the musical expression by creating, interpreting and imitating different resources by using different tools, thereby creating a meaningful musical communication to the audience, in interplay with other musicians and the singer.

A suposta ubiquidade de sistemas decomponíveis foi interpretada por Holland (1975) como o principal motivo para o desempenho dos algoritmos genéticos (Genetic Algorithms (GAs)). A hipótese de Building Blocks (BBs) sugere que algoritmos genéticos mais eficientes poderiam ser implementados, contudo, apenas anos depois essas ideias puderam ser avaliadas experimentalmente no contexto de algoritmos de estimação de distribuição (Estimation of Distribution Algorithms (EDAs)). EDAs utilizam modelos probabilísticos, estimados a partir da população, para inferir características do espaço de busca que poderiam ser utilizadas para implementar operadores de reprodução mais eficazes. Tanto em problemas mono- quanto multi-objetivo, EDAs emergiram sob a premissa de que a eficácia dos operadores de reprodução seria proporcional à representatividade dos modelos probabilísticos utilizados. No entanto, estudos recentes tem demonstrado que a dificuldade em se construir modelos confiáveis pode tornar essa premissa inviável. Ou seja, para certos problemas de otimização os modelos probabilísticos utilizados seriam, em geral, de baixa qualidade e, portanto, não produziriam operadores eficazes. Esta tese trata das limitações encontradas na construção de modelos probabilísticos (linkage learning) sob a perspectiva da multimodalidade dos problemas em questão. A análise teórica considerou problemas aditivamente separáveis, enquanto a generalização das conclusões foi investigada em instâncias do modelo NK-landscapes e do problema da mochila multidimensional (Multidimensional Knapsack Problem (MKP)). Os resultados indicaram que a acurácia dos modelos probabilísticos é se relaciona inversamente ao grau de multimodalidade da função objetivo e que, em casos de extrema multimodalidade a construção de modelos probabilísticos confiáveis pode ser tornar infactível. Este resultado poderia inviabilizar o uso de EDAs no contexto multiobjetivo, devido a intrínseca multimodalidade de tais problemas. No entanto, observou-se que apesar da ausência de estatísticas confiáveis sobre cada uma das funções objetivo, a correlação entre elas se torna estatisticamente observável e útil aos operadores de reprodução na manutenção da diversidade e controle convergência da população. / The supposed ubiquity of nearly-decomposable systems was interpreted by Holland (1975) as the rationale for the performance of Genetic Algorithms (GAs), the Building Block (BB) hypothesis. His seminal studies suggest more efficient GAs as viable, but only later on his ideas have become practically tangible in the context of Estimation of Distribution Algorithms (EDAs). EDAs employ probabilistic modeling so as to infer properties of the search space (BBs) that could be useful for the effectiveness of reproduction operators. In both, single- and multi-objective contexts, EDAs have emerged on the assumption there is a correlation between how much information a model can conceive and how effective reproduction operators can be. However, more recent results suggest the difficulties in producing accurate linkage models can prevent such a relation to be true. In other words, for some optimization problems linkage learning might not be able to produce accurate linkage models, hence EDAs would not outperform GAs. This thesis addresses the limits of linkage learning in the context of single- and bi-objective problems, regarding the influence of multimodality on the accuracy of the linkage models and the efficiency of EDAs. A theoretical analysis was performed in terms of additively separable functions and general conclusions are assessed through experimentation with instances of the NK-model and the Multidimensional Knapsack Problem (MKP). The results indicated that the accuracy of the linkage models tends to decrease as a result of increasing multimodality, which weakens pairwise dependencies and might lead to pairwise independence in extreme cases. Since most EDAs rely on bivariate statistics to estimate multivariate distributions, their applicability is limited to optimization problems within a certain range of multimodality. In multi-objective problems, on the other hand, some EDAs have shown better performance than GAs, which seemed as a contradiction since multi-objective problems are inherently multimodal. Our results suggest that in such cases the correlation among the objective functions becomes statistically evident, as a consequence, linkage learning models such correlation instead of problems substructures, which is useful to obtain a better exploration of extreme regions of the objective space.