This thesis explores the role of multimodality in language learners’ comprehension, and more specifically, the effects on students’ audio-visual comprehension when different orchestrations of modes appear in the visualization of vodcasts. Firstly, I describe the state of the art of its three main areas of concern, namely the evolution of meaning-making, Information and Communication Technology (ICT), and audio-visual comprehension. One of the most important contributions in the theoretical overview is the suggested integrative model of audio-visual comprehension, which attempts to explain how students process information received from different inputs. Secondly, I present a study based on the following research questions: ‘Which modes are orchestrated throughout the vodcasts?’, ‘Are there any multimodal ensembles that are more beneficial for students’ audio-visual comprehension?’, and ‘What are the students’ attitudes towards audio-visual (e.g., vodcasts) compared to traditional audio (e.g., audio tracks) comprehension activities?’. Along with these research questions, I have formulated two hypotheses: Audio-visual comprehension improves when there is a greater number of orchestrated modes, and students have a more positive attitude towards vodcasts than traditional audios when carrying out comprehension activities. The study includes a multimodal discourse analysis, audio-visual comprehension tests, and students’ questionnaires. The multimodal discourse analysis of two British Council’s language learning vodcasts, entitled English is GREAT and Camden Fashion, using ELAN as the multimodal annotation tool, shows that there are a variety of multimodal ensembles of two, three and four modes. The audio-visual comprehension tests were given to 40 Spanish students, learning English as a foreign language, after the visualization of vodcasts. These comprehension tests contain questions related to specific orchestrations of modes appearing in the vodcasts. The statistical analysis of the test results, using repeated-measures ANOVA, reveal that students obtain better audio-visual comprehension results when the multimodal ensembles are constituted by a greater number of orchestrated modes. Finally, the data compiled from the questionnaires, conclude that students have a more positive attitude towards vodcasts in comparison to traditional audio listenings. Results from the audio-visual comprehension tests and questionnaires prove the two hypotheses of this study.

Tese de doutoramento. Engenharia Electrotécnica e de Computadores. Faculdade de Engenharia. Universidade do Porto. 2009

Glaucoma is the second leading cause of blindness worldwide. The clinical standard for monitoring the functional deficits in the retina that are caused by glaucoma is the visual field test. In addition to monitoring the functional loss, evaluating the disease-related structural changes in the human retina also helps with diagnosis and management of this progressive disease. The characteristic changes of retinal structures such as the optic nerve head (ONH) are monitored utilizing imaging modalities such as color (stereo) fundus photography and, more recently, spectral-domain optical coherence tomography (SD-OCT). With the inherent subjectivity and time required for manually segmenting retinal structures, there has been a great interest in automated approaches. Since both fundus and SD-OCT images are often acquired for the assessment of glaucoma, the automated segmentation approaches can benefit from combining the multimodal complementary information from both sources. The goal of the current work is to automatically segment the retinal structures and extract the proper parameters of the optic nerve head related to the diagnosis and management of glaucoma. The structural parameters include the cup-to-disc ratio (CDR) which is a 2D parameter and is obtainable from both fundus and SD-OCT modalities. Bruch's membrane opening-minimum rim width (BMO-MRW) is a recent 3D structural parameter that is obtainable from the SD-OCT modality only. We propose to use the complementary information from both fundus and SD-OCT modalities in order to enhance the segmentation of structures of interest. In order to enable combining information from different modalities, a feature-based registration method is proposed for aligning the fundus and OCT images. In addition, our goal is to incorporate the machine-learning techniques into the graph-theoretic approach that is used for segmenting the structures of interest. Thus, the major contributions of this work include: 1) use of complementary information from SD-OCT and fundus images for segmenting the optic disc and cup boundaries in both modalities, 2) identifying the extent that accounting for the presence of externally oblique border tissue and retinal vessels in rim-width-based parameters affects structure-structure correlations, 3) designing a feature-based registration approach for registering multimodal images of the retina, and 4) developing a multimodal graph-based approach to segment the optic nerve head (ONH) structures such as Internal Limiting Membrane (ILM) surface and Bruch's membrane surface's opening.

If not ignored completely, the body has been under theorized in literacy research. However, recent research in cultural studies, linguistics, philosophy, psychology, neuroscience, education, and the arts suggests that the body is implicated in thinking and knowing as well as doing. In this dissertation I examine high school technical theater. In this robustly embodied activity students build sets, rig lights, and paint backdrops in preparation for a theatrical production, as well as run sound and lighting and perform scene changes during the production. I use data from high school technical theater to explore the body in literacy, embodied learning, collaboration, composition processes, and experiential learning. I gathered data primarily during out of school work sessions over the multi-week production cycles of six plays produced at one high school over two school years. As an experienced theater technician, I used participant observation as the primary method of data collection, supplemented with semi-structured interviews with the technical director, artistic director, and four students. I collected data and analyzed data through iterative processes in which analysis began during data collection, emerging analyses influenced data collection, and constant comparisons to new data influenced emerging analyses. Observations of student work revealed that student theater technicians employed literacy skills including speaking, reading, writing, drawing, calculating, and interpreting the written text of plays as necessary elements of the normal course of technical theater work. Observations of teaching and learning showed that little explicit instruction was used but that mini-lessons, individual and collective problem solving, and multiple configurations of collaboration among more and less experienced technicians led to the development of critical thinking and physical skills, as well as proficiency in the creation of props through the evaluation and application of building techniques and materials. I used theories from art making and multimodal literacy to examine technical theater building projects as examples of composition. My findings show that the design of technical theater texts - e.g. props, scenery, lighting - emerges through a recursive process of creation and interpretation and is mediated by the technicians' knowledge of building techniques and materials. Situated learning and activity theory were used to analyze learning in the technical theater community. Results demonstrate that the structure of the community allows for learning through experience, apprenticeship, and collaboration as well as through the creation of texts that balance personal expression with collaborative enterprise.

Le domaine des applications de communication a connu dernièrement de grands bouleversements, ce qui a conduit les chercheurs à suivre un nouveau paradigme d’évaluation, basé sur l’expérience utilisateur. Un grand nombre de méthodes de mesure ont ainsi été mises au point afin de la mesurer, mais demeurent immatures, voire contradictoires entre elles. C’est pourquoi un travail mérite d’être mené afin d’en augmenter la validité et la fiabilité. Ce travail de thèse s’est attaché donc à utiliser plusieurs de ces méthodes, de les combiner et les articuler selon différentes techniques afin d’améliorer la qualité de la mesure. Nous nous sommes appuyés pour cela sur un large spectre d’indicateurs, d’ordre physiologiques, comportementaux et auto rapportés et de deux stratégies de triangulation en particulier : multi-facettes et multi-mesures. Enfin, ces méthodes ont testées dans des cas d’application réels et selon une complexification croissante des procédures et traitements statistiques. Cela a donné lieu à trois études distinctes. La première a consisté à évaluer la pertinence d’un algorithme de recommandation de films face à son concurrent en utilisant une stratégie d’évaluation multi-facettes. Une deuxième étude a été élaborée afin de tester la pertinence de modèles d’évaluation multi-mesures, en évaluant l’utilisabilité de sites universitaires grâce à un logiciel de test utilisateur à distance (Evalyzer) et la combinaison multimodale de divers indicateurs d’utilisabilité. Enfin, une dernière étude a été réalisée afin de valider un protocole de mesure d’immersion multi-mesure (questionnaire, expression faciale, conductance de la peau, rythme cardiaque, comportement oculaire). Ces trois études nous ont permis d’évaluer la pertinence d’un certain nombre de mesures (d'utilisabilité et d’expérience utilisateur), la valeur ajoutée de certaines de leurs combinaisons, ainsi qu’un retour critique sur la procédure de validation multi-facettes utilisée dans cette thèse / The communication application domain has recently experienced some great changes. This shift has led researchers to follow a new evaluation paradigm, based on the user experience. This trend differs fundamentally from its predecessors by its holistic, subjective and positive nature. Many measurement methods have been developed to assess the user experience, but remain immature and even contradictory. Moreover, the basic nature of the user experience has led to the loss of reliability in the exercise of its measurement. This is why studies should be conducted in order to increase the validity and reliability of this new approach. The main aim of this research is to use several methods and to combine and articulate them using various techniques to improve the measurement quality. These studies was based on a broad spectrum of indicators (physiological, behavioral and self-reported) and two triangulation strategies in particular: multi-faceted and multi-measures. Finally, these methods were tested in real applied cases and under an increasing procedure complexity and statistical processing. This research has resulted in three separate studies. The first aims to evaluate the relevance of a movie recommendation algorithm against its competitor using a multifaceted evaluation strategy. A second study was designed to test the relevance of a multi-measures evaluation model by assessing the usability of university sites based on a remote user testing software (Evalyzer) and a multimodal combination of various indicators of usability. A final study was conducted to validate multi-measures immersion protocol (questionnaire, facial expression, skin conductance, heart rate, eye behavior). These three studies have highlighted the relevance of numerous measures (of usability and user experience), the added value of some of their combinations, as well as a critical return to the multi-facet validation procedure used in this thesis

A partir d’une intervention auprès de médecins du travail, cette thèse explore les conditions de développement des sentiments au travail. Elle stabilise une conceptualisation de l’affectivité distinguant affect, émotion et sentiment qui pose des rapports entre l’affect lié aux conflits de l’activité (conflits liés à la conception de l’activité comme triade vivante sujet/objet/autrui et conflits liés aux rapports entre le déjà vécu et le vivant) et les sentiments et émotions qui en sont les instruments de réalisation. Le sentiment est défini comme l’instrument de réalisation de l’affect détaché de l’événement affectif et relié à l’activité de pensée. Une analyse multimodale de nos matériaux s’attache à repérer des indices de l’affect dans trois modalités étudiées (regard, voix, mot) et des indices de développement de la pensée (développement des significations des mots et des objets de discours). Nous concluons que l’intervention en clinique de l’activité par l’exposition de l’activité qu’elle autorise et la production de débats sur les critères du travail bien fait qu’elle organise peut provoquer des affects et contribuer au développement du sentiment du travail bien fait. / Based on an intervention with occupational health physicians, this thesis explores the developmental conditions of sentiments at work. The thesis seek to stabilize a conceptualisation of affectivity distinguishing affect, emotion and sentiment, and lays the relationships between affect, which is related to conflicts of activity (conflicts related to activity as a living triad subject/object/others and conflicts related to relationships between the « already lived » and the « living »), and sentiments and emotions, which constitute its instruments of realization. Sentiment is defined as instrument of affect realization, detached of affective event, and related to thinking activity.Multimodal analysis of research materials allows the identification of affect indices, based on three studied modalities (gaze, voice, word), and development of thinking indices (development of signification of word, and discourse objects).We conclude that intervention in clinic of activity, by exposing activity and producing disputations on quality of work criteria, can cause affects and contributes to develop the sentiment of « well-done-work ».

I dagens samhälle utgör skärmtiden en stor del av barn och ungas fritid (Statens medieråd, 2017) och en majoritet av barnen tillbringar sin fria tid på fritidshemmet (Skolverkets elevstatistik, 2018). Skärmtiden inom skola och fritidshem ökar (Olin-Scheller och Roos, 2015) därmed kan det vara av intresse att undersöka vad eleverna möter i den digitala världen utifrån ett genusperspektiv. Studiens syfte var att analysera digitala spel-appar som eleverna har tillgång till under sin vistelse på fritidshemmet. Spel-apparna och dess karaktärer analyserades ur ett genusperspektiv med inspiration av en multimodal textanalysmodell. Det är intressant att undersöka karaktärernas genusframställning för att detta kan ske som ett informellt lärande i spelen och kan påverka elevernas uppfattningar på vad som anses som kvinnligt och manligt. Resultat visar att de kvinnliga och manliga karaktärerna gestaltas utifrån de stereotypa normer och föreställningar som finns av vad som upplevs kvinnligt och manligt. Kvinnliga karaktärer avbildas som passiva och oskuldsfulla eller framställs som kloka gamla gummor. De manliga karaktärerna tilldelas attribut som framställer dem som handlingskraftiga och aktiva såsom vapen och deras kroppshållning. I resultatet framkom det att karaktärerna kunde framställas som mer eller mindre betydelsefulla beroende på form och färgspråk av deras attribut. / <p>Godkännande datum: 2019-06-07</p>

Syftet med denna studie var att studera vilka föreställningar som speglas i Fjällrävens produktkatalog och vilka diskurser samt myter som används i kommunikationen av dessa. Således var studiens syfte att skapa en ökad förståelse inom strategisk hållbarhetskommunikation. Studien tar avstamp i en multimodal kritisk diskursanalys som är genomförd på företaget Fjällrävens produktkatalog för vår/sommar 2019. Materialet består av artiklar relaterade till hållbar utveckling med tillhörande bilder. Studiens resultat visar på att de analyserade texterna och bilderna förstärker myten om att människan är ett med naturen samtidigt som bilderna visar på en oproblematisk natur. Detta stärker komplexiteten som finns kring hållbarhetsbegreppet trots att företaget visar på en tendens att vilja öppna en dialog med sina konsumenter om hur man ska hantera vissa hållbarhetsfrågor. Resultaten indikerar även på en överrepresentation av den miljömässiga dimensionen som är en av tre dimensioner inom hållbar utveckling.

Multimodal writing refers to texts that use more than one communicative mode to convey information. While there is much scholarship that examines the history of alphabetic writing instruction and the alphabetic composing processes of students, little research explores the historical origins of multimodal composition and the processes in which students engage as they compose multimodal texts. This two-part project takes a fresh approach to studying multimodal writing by exploring the multimodal pedagogies of ancient Greek and Roman rhetoric and writing teachers, analyzing the role of mental and physical images in modern writers’ composing practices, and investigating contemporary students’ processes for composing multimodal texts. In Part I, I re-imagine the history of multimodal writing by exploring the multimodal pedagogies that instructors of rhetoric and writing developed during Greek and Roman Antiquity, and I show how contemporary students use an array of multimodal composing processes that rely on both mental and physical images to write alphabetic text. In Part II, I share the results of a case study in which I investigate the processes students use to compose audio- and video-essays while enrolled in a multimodal writing course. This study explores what students know about multimodal writing before beginning the course, how they learn the software needed to compose these projects, the challenges students experience as they compose, and the similarities and differences students perceive between their own processes for composing alphabetic and multimodal texts. Ultimately, I argue that composition teachers must acknowledge our long history of teaching with multimodal pedagogies and our experience composing alphabetic text through multimodal processes. Recognizing this lengthy history will decrease the anxiety that many composition teachers experience when tasked with teaching multimodal writing because, while typically only time and experience can grow confidence, in this case, a recognition of how much we already know will allow us to teach with the self-assurance we have earned.

Le mélanome cutané est un cancer très invasif, dont l'évolution est rapidement fatale au stade métastatique du fait d'une absence de thérapies réellement efficaces. Face à ce constat, une stratégie de multimodalité a été évaluée au moyen d'un vecteur spécifique des cellules de mélanome et présentant la double potentialité diagnostique (imagerie TEP) et thérapeutique (radiothérapie interne vectorisée (RIV)) en fonction du radioélément introduit. Dans ce but, des analogues iodés et fluorés d'ICF01012, un vecteur ciblant la mélanine surexprimée dans un grand nombre de mélanomes, ont été synthétisés puis radiomarqués (iode-125 d'une part et fluor-18 d'autre part). Les profils pharmacocinétiques des radiotraceurs ont par la suite été étudiés in vivo sur modèle murin de mélanome, par imagerie scintigraphique γ (125I) et par imagerie TEP (18F). A l'issue de ces études, le traceur 4 a été sélectionné comme molécule leader de cette série, en raison d'un tropisme tumoral spécifique, élevé et durable, associé à élimination rapide des tissus non-cibles. Ce composé a ensuite été radiomarqué à l'iode-131 pour une évaluation en RIV sur le même modèle murin. Le traitement avec [131I]4 a induit une inhibition significative de la croissance tumorale et une augmentation significative de la médiane de survie. Au vu des résultats prometteurs obtenus avec le traceur 4, des études de métabolisme sur le même modèle murin ont été effectuées. En conclusion, en termes de chimie, radiochimie, stabilités in vitro et in vivo, TEP et RIV, le traceur 4 a validé notre concept de multimodalité. A terme, ce composé pourrait être transféré pour des études cliniques afin : de sélectionner les patients présentant des lésions de mélanome pigmentées (18F/TEP) et donc susceptibles de répondre au protocole de RIV ; de traiter ces patients (131I/RIV) ; de suivre la réponse au traitement (18F/TEP). En parallèle a été développé le premier groupement prosthétique iodé et fluoré multimodal, le 4-fluoro-3-iodobenzoate de tétrafluorophényle 108, pour le marquage de vecteurs d'intérêt (peptides, protéines, nanoparticules…). Le composé 108 et les précurseurs de radiomarquages correspondants ont été synthétisés en peu d'étapes, avec de bons rendements. Les radiomarquages ont ensuite été mis au point, et ont permis d'obtenir [125I]108 et [18F]108 en des temps relativement courts, avec d'excellents rendements et puretés radiochimiques. Le groupement prosthétique a ensuite été couplé avec succès à un ligand organique et deux peptides. En conclusion, nous avons démontré que le composé multimodal 108 peut être utilisé pour la radiofluoration et la radioiodation de vecteurs d'intérêt, autorisant des applications à la fois diagnostiques (TEP/18F) et thérapeutiques (RIV/131I) pour la prise en charge de nombreux cancers. / Melanoma is the most serious form of skin cancer with a poor prognosis for patients with metastatic disease. Our project deals with a multimodal approach, using a single fluorinated and iodinated melanintargeting compound, and offering both imaging (PET/18F) and therapeutic (targeted radionuclide therapy (TRT)/131I) applications, depending on the radioisotope introduced. Furthermore, using PET imaging, our strategy allows a selection of TRT-responded patients (i.e. bearing pigmented metastases) as well as a monitoring of treatment response. In previous works, the iodinated quinoxaline-carboxamide compound ICF01012 was evaluated in a TRT protocol, using melanoma-bearing mice models. It demonstrated efficacy, with significant inhibition of tumoural growth and improvement of the median survival. Based on these results, iodinated and fluorinated analogs of ICF01012 were synthesized, for multimodality purposes. All new compounds were then radiolabelled with iodine-125 and fluorine-18 (fully automated radiosyntheses), with good radiochemical yields and excellent radiochemical purities. For pharmacokinetic profile studies on melanoma-bearing mice, [125I]radiotracers were evaluated by γ-scintigraphy and [18F]radiotracers by PET. Compound 4 emerged as the lead tracer, with a specific and long-lasting tumoural uptake and a fast clearance from non-specific tissues, leading to highly contrasted images. The tracer 4 was then radiolabelled with iodine-131, with excellent radiochemical yield and purity, to perform a TRT assay on the same melanoma model. Treatment with [131I]4 significantly inhibited tumoural growth and lung metastasis occurrence. Moreover, it significantly improved the median survival. As tracer 4 demonstrated promising results in PET imaging and TRT of melanoma, its metabolism was investigated with [125I]4 and [18F]4: radiotracers were found unchanged in melanin-containing tissues (tumour and eyes), while a fast breakdown was observed in excretion organs and fluids (four metabolites were identified). In conclusion, in terms of chemistry, radiochemistry, in vitro and in vivo stability, PET imaging and TRT, compound 4 validated our multimodality concept. We also developed the first bimodal fluorinated and iodinated prosthetic group, the 2,3,5,6-tetrafluorophenyl 4-fluoro-3-iodobenzoate (108), as a suitable acylating agent for the labelling of a large variety of compounds. In this approach, this new compound allows applications in diagnosis (PET imaging/18F) and therapy (TRT/131I). Compound 108 and its corresponding precursors for radiolabelling were synthesized in very few steps, with good yields. [125I]108 was then prepared in one-step starting from a perfluorostannane precursor, and purified by F-SPE cartridge to avoid time-consuming HPLC. As for [18F]108, it was produced by a fully automated three steps, two-pots radiosynthesis process. [125I]108 and [18F]108 were both obtained in a short time, with excellent radiochemical yields and purities. These prosthetic groups were then successfully used to radiolabel small organic ligand N,N-diethylethylenediamine and peptides NDP-MSH and PEG3[c(RGDyK)]2, in mild conditions, with good yields. In conclusion, we demonstrated that compound 108 could be a promising acylating bimodal prosthetic group for radiofluorination and radioiodination of small organic molecules, peptides, proteins, antibodies as well as nanoparticles.