Vårt syfte är att synliggöra den didaktiska användningen av multimodala texter genom att undersöka hur fyra lärare arbetar med multimodala texter i undervisningen. Våra teoretiska utgångspunkter är sociokulturellt perspektiv på lärande, socialsemiotisk teoribildning och socialsemiotisk multimodal teoribildning. Vi analyserar våra resultat utifrån socialsemiotisk multimodal teoribildning. Studien utgår även från multimodal skolforskning som berör multimodala arbetssätt och hur dessa förekommer i undervisning. Studien sker genom observationer och halvstrukturerade intervjuer med lärare som undervisar i svenska på mellanstadiet. Utifrån våra observationer och halvstrukturerade intervjuer kan vi dra slutsatsen att lärare ser stora möjligheter att möta fler elever utifrån deras behov, förutsättningar och utvecklingszon genom att använda sig av olika medium, modaliteter och semiotiska resurser. Genom våra halvstrukturerade intervjuer och diskussioner upplever vi att lärare arbetar multimodalt men saknar ett metaspråk för multimodala arbetssätt. Avslutningsvis drar vi slutsatsen att elever måste utveckla grundläggande kompetens i hur de ska läsa, skriva och analysera multimodala texter för att kunna delta i samhället.

Examensarbetets syfte är att undersöka vilka insatser och vilket stöd som personer med ADHD respektive AST är i behov av, samt kan bistås med under sitt livslopp. Studien är en övergripande litteraturstudie som genom kvalitetsbedömda forskningsartiklar utvädrat dess effekt och långtidseffekt för psykisk och socialt liv. Resultat: samtliga studier visar att multimodal effekt, farmakoterapi i kombination med gruppbaserad behandling samt psykoedukation givit bäst effekt för att utveckla nedsatta förmågor och färdigheter användbara under för ett fungerande vardagsliv. Genom inlärning och psykoedukation för anhöriga har dessutom minskat stigma för denna målgrupp. Studiens slutsats är därmed att multimodal behandling både för individer med funktionsnedsättningar samt dess anhöriga är effektiva. / This degree projects aim was to examine the treatments and support that people with ADHD and ASD has proven need of during their lifetime. This study is a literary overview of current empirical studies examining treatments and interventions for people with ADHD and ASD. The result has shown no significant effect making any treatments effect generalizable on its own. However, this conclusion shows that combined psychosocial treatment alongside pharmacotherapy, as well as treatment including peers proven successful in establishing social skills, relationships and a functioning everyday life. Conclusion: multimodal treatments for both individuals with ADHD and ASD, alongside its peers is considered effective.

The biopharmaceutical industry is constantly developing biological drugs, resulting in increased levels of product related impurities having similar characteristics as the target. The aim of the ligand project was to address future challenging purifications by developing new ligands for future resins for the biopharmaceutical industry. The purpose of this study was to develop a high-throughput screening method and use it to compare 15 novel multimodal anionic exchange ligand analogues with two reference ligands, for future polishing steps in the downstream process. The protein binding behavior of the ligands were studied with alkaline phosphatase, human serum albumin, α-chymotrypsinogen A and a monoclonal antibody as model proteins, at various pH values and salt concentrations. The selection process of the model proteins was based on stability studies, a study of their adsorption to the 96 well plate, and their binding behavior on three of the ligand analogues and one reference ligand. The percent protein bound to the ligands at the various conditions was calculated and presented in plots in order to study their binding behaviors. The calculated values were also used in order to evaluate the results in principal component analysis, creating chromatographic diversity maps. The maps were used to get an overview of the differences and similarities of the ligand analogues compared to the reference resins, which can be used for selecting ligands for future research and biomanufacturing. Four analogues and one reference ligand were also studied in a column format where different gradients were used, which confirmed the obtained results in the plate experiments.

This thesis explores the use of multimodality to communicate retrieved results of Internet search engines. The investigation aimed to investigate suitable multimodal metaphors which would increase the level of usability of Internet search engine interfaces and enhance users` experience in the search activity. The study consisted of three experiments based on questionnaires and Internet search activities with a set of text-based and multimodal interfaces. These interfaces were implemented in two browsing platforms, named AVBRO and AVBRO II. In the first experiment, the efficiency of specific multimodal metaphors to communicate additional information of retrieved results was investigated. The experiment also sought to obtain users` views of these metaphors with a questionnaire. An experimental multimodal interface of the AVBRO platform, which communicated additional information with a combination of three 2D graphs and musical stimuli, was used as a basis for the experiment, together with the Google search engine. The results obtained led to the planning of a second experiment. The aim of this experiment was to obtain and compare the level of usability of four different experimental multimodal interfaces and one traditional text-based interface, all implemented in the AVBRO II platform. Effectiveness, efficiency and users` satisfaction were used as criteria to evaluate the usability of these interfaces. In the third and final experiment the usability analysis of a traditional text-based interface and the two most suitable experimental multimodal interfaces of the AVBRO II platform was further investigated. Learnability, errors rate, efficiency, memorability and users` satisfaction were used as criteria to evaluate the usability of these interfaces. The analysis of the results obtained from these experiments provided the basis for a set of design guidelines for the development of usable interfaces based on a multimodal approach.

Recently, there has been a lot of interest in explainable predictions, with new explainability approaches being created for specific data modalities like images and text. However, there is a dearth of understanding and minimal exploration in terms of explainability in the multimodal machine learning domain, where diverse data modalities are fused together in the model. In this thesis project, we look into two multimodal model architectures namely single-stream and dual-stream for the Visual Entailment (VE) task, which compromises of image and text modalities. The models considered in this project are UNiversal Image-TExt Representation Learning (UNITER), Visual-Linguistic BERT (VLBERT), Vision-and-Language BERT (ViLBERT) and Learning Cross-Modality Encoder Representations from Transformers (LXMERT). Furthermore, we conduct three different experiments for multimodal explainability by applying the Local Interpretable Model-agnostic Explanations (LIME) technique. Our results show that UNITER has the best accuracy among these models for the problem of VE. However, the explainability of all these models is similar. / Under den senaste tiden har intresset för förklarbara prediktioner (eng. explainable predictions) varit stort, med nya metoder skapade för specifika datamodaliteter som bilder och text. Samtidigt finns en brist på förståelse och lite utforskning har gjorts när det gäller förklarbarhet för multimodal maskininlärning, där olika datamodaliteter kombineras i modellen. I detta examensarbete undersöker vi två multimodala modellarkitekturer, så kallade en-ström och två-strömsarkitekturer (eng. single-steam och dual-stream) för en uppgift som kombinerar bilder och text, Visual Entailment (VE). Modellerna som studeras är UNiversal Image-TExt Representation Learning (UNITER), Visual-Linguistic BERT (VLBERT), Vision-and-Language BERT (ViLBERT) och Learning Cross-Modality Encoder Representations from Transformers (LXMERT). Dessutom genomför vi tre olika experiment för multimodal förklarbarhet genom att tillämpa en metod som heter Local Interpretable Model-agnostic Explanations (LIME). Våra resultat visar att UNITER har bäst prestanda av dessa modeller för VE-uppgiften. Å andra sidan är förklarbarheten för alla dessa modeller likvärdig.

With the development and renovation of digital information technology, images, sounds, animations, colors and other models are integrated into one system to convey information. Multimodality is also applied in language research. One example of multimodal discourse are tourism video commercials, which combine written, visual and auditory signals to construct the metaphorical meaning of advertisements. Therefore, this paper selects six pieces of German and Chinese tourism video commercials as objects, which are parsed, transcribed and annotated by ELAN 5.1. Both quantitative and qualitative approaches are adopted to analyze the selected corpus. The theoretical framework for this study is Systemic-Functional Grammar and Visual Grammar, and it aims to explore the similarities and differences in the fusion of multimodal features between German and Chinese tourism video advertisements by comparative analysis.

I denna uppsats undersöks om svenska kommuner uppfyller lagarna om tillgänglighetsanpassning på offentliga webbplatser. I dagens digitaliserade samhälle används internet som ett kommunikationsverktyg för att framföra samhällsinformation. På grund av olika varianter av funktionsnedsättningar kan information som publiceras på internet vara svår att ta del av. Genom att en webbplats fullföljer lagarna för tillgänglighetsanpassning på offentliga webbplatser kan information som publiceras nå ut till en bredare målgrupp. Eftersom kommunal information är viktig för alla invånare bosatt i en kommun är det av stor betydelse att invånare har möjlighet att ta del av information som publiceras på kommunala webbplatser. I denna uppsats undersöker vi 10 webbplatser från olika kommuner i olika delar av Sverige; sedan analyseras materialet och framställer om det finns skillnader mellan de olika kommunerna. Undersökningen utförs utifrån europeiska standarden EN 301 549 V3.2.1, baserad på WCAG 2.1. Materialet presenteras med hjälp av diagram samt med beskrivande text. / This essay examines how Swedish municipalities follow the laws on accessibility adaptation on public websites. In today's digitized society, the internet is used as a communication tool to convey social information. Due to different types of disabilities, information published on the Internet can be difficult to read. By complying with the laws for accessibility on public websites, published information can reach a wider target group. As municipal information is important for all residents of a municipality, it is important that residents can get the information that is published on municipal websites. In this essay, we examine 10 websites from different municipalities in different parts of Sweden; then we analyse the material and present whether there are differences between the different municipalities. The examination is carried out based on the European standard EN 301 549 V3.2.1, based on WCAG 2.1. The material is presented using diagrams and descriptive text.

La communication multimodale, qui est primordiale dans les relations interpersonnelles, reste encore très limitée dans les interfaces homme-machine actuelles. Parmi les différentes modalités qui ont été adoptées par les recherches en interaction homme-machine, la modalité posturale a été moins explorée que d’autres modalités comme la parole ou les expressions faciales. Les postures corporelles sont pourtant indispensables pour interpréter et situer l’interaction entre deux personnes, que ce soit en termes de contexte spatial ou de contexte social. Les briques de base que sont les dispositifs et les modalités d’interaction (par exemple la kinect ou les avatars) sont pourtant disponibles. Il manque cependant des modèles informatiques reliant ces médias et modalités aux fonctions de communication pertinentes dans les interactions interpersonnelles comme celles liées à l’espace ou aux émotions.L’objectif de cette thèse est de concevoir un premier modèle informatique permettant d’exploiter les postures dans les interactions homme-machine. Cela soulève plusieurs questions de recherche : comment représenter symboliquement des postures exprimées durant des interactions interpersonnelles ? Comment spécifier les comportements posturaux de personnages virtuels ? Quelles doivent être les caractéristiques d’un modèle d’interaction corporel permettant des interactions entre un personnage virtuel et un utilisateur dans différents contextes physiques ou sociaux ?L’approche proposée consiste dans un premier temps à prendre comme point de départ des corpus vidéo filmés dans différentes situations. Nous avons défini un schéma de codage pour annoter manuellement les informations posturales à différents niveaux d’abstraction et pour les différentes parties du corps. Ces représentations symboliques ont été exploitées pour effectuer des analyses des relations spatiales et temporelles entre les postures exprimées par deux interlocuteurs.Ces représentations symboliques de postures ont été utilisées dans un deuxième temps pour simuler des expressions corporelles de personnages virtuels. Nous nous sommes intéressés à un composant des émotions particulièrement pertinent pour les études sur les postures : les tendances à l’action. Des animations impliquant deux personnages virtuels ont été ainsi conçues puis évaluées perceptivement.Enfin, dans un troisième temps, les expressions corporelles d’un personnage virtuel ont été conçues dans une application mixte faisant intervenir un personnage virtuel et un utilisateur dans un cadre d’interaction ambiante. Les postures et gestes du personnage virtuel ont été utilisées pour aider l’utilisateur à localiser des objets du monde réel.Cette thèse ouvre des perspectives sur des études plus spécifiques de l’interaction corporelle nécessitant par exemple des annotations automatiques via des dispositifs de capture de mouvement ou la prise en compte des différences individuelles dans l’expression posturale. / Multimodal communication is key to smooth interactions between people. However, multimodality remains limited in current human-computer interfaces. For example, posture is less explored than other modalities, such as speech and facial expressions. The postural expressions of others have a huge impact on how we situate and interpret an interaction. Devices and interfaces for representing full-body interaction are available (e.g., Kinect and full-body avatars), but systems still lack computational models relating these modalities to spatial and emotional communicative functions.The goal of this thesis is to lay the foundation for computational models that enable better use of posture in human-computer interaction. This necessitates addressing several research questions: How can we symbolically represent postures used in interpersonal communication? How can these representations inform the design of virtual characters' postural expressions? What are the requirements of a model of postural interaction for application to interactive virtual characters? How can this model be applied in different spatial and social contexts?In our approach, we start with the manual annotation of video corpora featuring postural expressions. We define a coding scheme for the manual annotation of posture at several levels of abstraction and for different body parts. These representations were used for analyzing the spatial and temporal relations between postures displayed by two human interlocutors during spontaneous conversations.Next, representations were used to inform the design of postural expressions displayed by virtual characters. For studying postural expressions, we selected one promising, relevant component of emotions: the action tendency. Animations were designed featuring action tendencies in a female character. These animations were used as a social context in perception tests.Finally, postural expressions were designed for a virtual character used in an ambient interaction system. These postural and spatial behaviors were used to help users locate real objects in an intelligent room (iRoom). The impact of these bodily expressions on the user¡¯s performance, subjective perception and behavior was evaluated in a user studyFurther studies of bodily interaction are called for involving, for example, motion-capture techniques, integration with other spatial modalities such as gaze, and consideration of individual differences in bodily interaction.

Syftet med studien är att undersöka hur lärare beskriver möjligheter och hinder med traditionell skrivinlärning respektive digital skrivinlärning genom metoden ASL. Målet är att bidra med kunskap om hur metoderna kan användas i undervisningen. Fyra forskningsfrågor ställdes utifrån syftet: Hur beskriver lärare att de arbetar med traditionell skrivinlärning? Hur beskriver lärare att de arbetar med ASL? Vilka fördelar finns det med metoderna anser lärare? Vilka nackdelar finns det med metoderna anser lärare? Studien har sin utgångspunkt i det sociokulturella perspektivet. En kvalitativ metod har använts för att genom semistrukturerade intervjuer samla in data. Respondetnerna bestod av fem lärare, yrkesverksamma i grundskolans lägre åldrar. Studiens empiri analyserades genom en tematisk analysmetod där materialet färgkodades med utgångspunkt i hur studiens forskningsfrågor besvarades. Resultatet visar hur fem lärare undervisar traditionell skrivinlärning och digital skrivinlärning samt vad det finns för möjligheter och hinder med de båda metoderna. / The purpose of the study is to investigate how teachers describe opportunities and barriers with traditional writing acquisition through WTR. The goal is to provide knowledge about how the methods can be used in teaching. Four research questions were asked based on the aim: How do teachers describe that they work with traditional writing acquisition? How do teachers describe that they work with WTR? What are the advantages of the methodes according to teachers? What are the disadvantages of the methodes according to teachers? The study is based on the socio-cultural perspective. A qualitative method has been used to collect data through semi-structured interviews. The respondents were of five teachers who work in primary schools. The collected material was analyzed with the help of a thematic analysis method in which the collected material was color-coded according to how it respondse to the questions of the study. The findings shows how teachers work with traditional writing and digital writing in school, as well as what there are for opportunities and barriers there are in connection with the two methodes.

Les mécanismes de compréhension chez l'être humain sont par essence multimodaux. Comprendre le monde qui l'entoure revient chez l'être humain à fusionner l'information issue de l'ensemble de ses récepteurs sensoriels. La plupart des documents utilisés en traitement automatique de l'information sont multimodaux. Par exemple, du texte et des images dans des documents textuels ou des images et du son dans des documents vidéo. Cependant, les traitements qui leurs sont appliqués sont le plus souvent monomodaux. Le but de cette thèse est de proposer des traitements joints s'appliquant principalement au texte et à l'image pour le traitement de documents multimodaux à travers deux études : l'une portant sur la fusion multimodale pour la reconnaissance du rôle du locuteur dans des émissions télévisuelles, l'autre portant sur la complémentarité des modalités pour une tâche d'analyse linguistique sur des corpus d'images avec légendes. Pour la première étude nous nous intéressons à l'analyse de documents audiovisuels provenant de chaînes d'information télévisuelle. Nous proposons une approche utilisant des réseaux de neurones profonds pour la création d'une représentation jointe multimodale pour les représentations et la fusion des modalités. Dans la seconde partie de cette thèse nous nous intéressons aux approches permettant d'utiliser plusieurs sources d'informations multimodales pour une tâche monomodale de traitement automatique du langage, afin d'étudier leur complémentarité. Nous proposons un système complet de correction de rattachements prépositionnels utilisant de l'information visuelle, entraîné sur un corpus multimodal d'images avec légendes. / The human faculties of understanding are essentially multimodal. To understand the world around them, human beings fuse the information coming from all of their sensory receptors. Most of the documents used in automatic information processing contain multimodal information, for example text and image in textual documents or image and sound in video documents, however the processings used are most often monomodal. The aim of this thesis is to propose joint processes applying mainly to text and image for the processing of multimodal documents through two studies: one on multimodal fusion for the speaker role recognition in television broadcasts, the other on the complementarity of modalities for a task of linguistic analysis on corpora of images with captions. In the first part of this study, we interested in audiovisual documents analysis from news television channels. We propose an approach that uses in particular deep neural networks for representation and fusion of modalities. In the second part of this thesis, we are interested in approaches allowing to use several sources of multimodal information for a monomodal task of natural language processing in order to study their complementarity. We propose a complete system of correction of prepositional attachments using visual information, trained on a multimodal corpus of images with captions.