Although digital media has been exploited to improve digital libraries, social networking sites, and book promotion for adult and child stakeholders, but encouraging children who have the choice to either read from a book or on a screen remains limited worldwide, including Jordan. This interest has meant that data about children's reading habits were needed, and the present study was intended as a contribution towards this aim. Interviews were conducted with Jordanian writers, publishers, child specialists, and various children's cultural centres. The managers and personnel unanimously showed that Jordanian children are not good readers and that a limited number of books are published for children as there are actual boundaries preventing Jordanian writers from publishing books. In particular, subjecting the typical sorts of children's websites - 'Club Penguin', 'PBS Kids', 'A Story before Bed', 'Baraem', 'Storyline Online', and 'Raneen' - to evaluation showed that 'Club Penguin' got the highest rank among the other websites in terms of multimodal features, usability, and language, while 'PBS Kids' got the highest rank regarding interactivity, and 'A Story before Bed' got the highest rank in reading activities. Although it was realised that most children were satisfied with the aspects of usability and ease of use rather than the structure or the aesthetic of the website, and were more attracted to the websites that provide multimodal features such as special characters, narration, gesture, and interactivity. The targeted websites' parameters obtained from the survey were used as guidance in the design structure of the KITABAK website, as a virtual reading environment for children's reading practices. The evaluation results that were obtained showed that there is a significant correlation towards encouraging children's reading habits and reading from printed books accompanying the website; girls showed more interest in reading iv than boys; and there is an obvious willingness for the adaptation of the website as a part of the Jordanian school curriculum. In addition, the KITABAK website was accepted significantly more than 'Club Penguin', mainly because the KITABAK website has facilities, games and reading activities. Also, results showed that children who were subjected to testing the KITABAK website for a one-week period proved to accept the website significantly more than those who were subjected to testing it once.

Texts and images provide alternative, yet orthogonal views of the same underlying cognitive concept. By uncovering synergistic, semantic relationships that exist between words and images, I am working to develop novel techniques that can help improve tasks in natural language processing, as well as effective models for text-to-image synthesis, image retrieval, and automatic image annotation. Specifically, in my dissertation, I will explore the interoperability of features between language and vision tasks. In the first part, I will show how it is possible to apply features generated using evidence gathered from text corpora to solve the image annotation problem in computer vision, without the use of any visual information. In the second part, I will address research in the reverse direction, and show how visual cues can be used to improve tasks in natural language processing. Importantly, I propose a novel metric to estimate the similarity of words by comparing the visual similarity of concepts invoked by these words, and show that it can be used further to advance the state-of-the-art methods that employ corpus-based and knowledge-based semantic similarity measures. Finally, I attempt to construct a joint semantic space connecting words with images, and synthesize an evaluation framework to quantify cross-modal semantic relationships that exist between arbitrary pairs of words and images. I study the effectiveness of unsupervised, corpus-based approaches to automatically derive the semantic relatedness between words and images, and perform empirical evaluations by measuring its correlation with human annotators.

Background Musculoskeletal disorders (MSDs) are one of the most important causes of temporary and permanent disability, causing acute or persistent pain, resulting in reduced and/or lost ability to work. Return to work (RTW) is multidimensional problem including many different factors and aspects. Few recent studies have analyzed factors predicting RTW after multimodal rehabilitation (MMR). Identification of predictors for RTW may help to improve the planning and optimization of the RTW strategy. The REHSAM II project is a randomized controlled trial with the aim to evaluate if MMR together with a web Behavior Change Program for Activity could increase work ability among persons with persistent MSDs as compared to MMR. Therefore the aim of this study was to identify factors explaining RTW 12 month after baseline in the REHSAM II project. Methods The present study is a secondary assessment of the data from the randomized controlled trial REHSAM II. A total of 97 participants with persistent musculoskeletal pain were randomly allocated to MMR + web-based education or only MMR group. The subjects were followed from baseline to 12 months. Information on potential predictors was obtained from self-administered questionnaires. Data were analyzed with univariate and multiple logistic regression models. Results In the final multiple regression model RTW was predicted by the Örebro Musculoskeletal Pain Screening Questionnaire (ÖMPSQ score) (p=0.003, OR=0.961) and EuroQol (EQ-5D index) (p=0.017, OR=7.283). The univariate regression analyses showed that pain and disability level, the capacity to perform a task in relation to pain and other symptoms, hospital and psychiatric care, medication for insomnia, catastrophizing, self-assessed work ability compared with the lifetime best, satisfaction with life, ability for coping and controlling work situation, ability for coping with life outside work and sense of responsibility for managing health condition were significantly associated with RTW. Conclusion In conclusion, psychosocial pain-related variable and health-related quality of life predicted RTW in the final model. The result confirms the fact that RTW is a multidimensional problem involving a complex interaction of many factors.

Le travail de recherche présenté dans cette thèse considère la mémoire humaine comme un système unique et non abstractif qui reflète l’ensemble de nos expériences sous forme de traces épisodiques multimodales. Les objectifs de cette thèse sont multiples, mais le principal est de montrer qu’un effet robuste de la mémoire, l’effet de distinctivité, peut émerger aussi bien dans les tâches implicites que dans les tâches explicites et que cet effet s’expliquerait en termes de mécanismes spécifiques (activation et intégration multimodales) et non pas en termes de systèmes mnésiques sous-jacents.Trois séries d’expériences ont été élaborées. Dans une première série, nous avons manipulé l’information contextuelle extrinsèque associée à ces concepts. Une tâche de catégorisation nous a permis de démontrer que l’effet de distinctivité pouvait se manifester avec une tâche implicite de mémoire.Dans une deuxième série d’expériences, nous avons manipulé la distance entre des images à catégoriser dans une phase d’encodage. Ainsi, les images apparaissaient soit plus éloignées les unes des autres, soit plus proches. Suite à une récupération implicite, nous avons mis en évidence que notre hypothèse de distinctivité spatiale était validée, c’est-à-dire que les items spatialement plus distinct lors de l’encodage sont associés à des performances supérieures en phase test.Dans la troisième série d’expériences, nous avons mis en évidence que lesperformances liées à l’effet de distinctivité dans les tâches implicites et explicites de mémoire variaient selon différents niveaux d’intégration des dimensions sensorielles. Cette idée a été testée en rappel libre, en décision lexicale et en reconnaissance.Au final, nos résultats expérimentaux suggèrent que les performances issuesdes tâches implicites et explicites peuvent être expliquées au sein du même système mnésique unique. Ainsi, les processus d’intégration seraient à l’origine de ce phénomène. / The series of research presented in this thesis considers human memory as a single and non-abstractive system which reflects all of our experiments in an episodical multimodal traces form. Objectives of this thesis are multiple, but the main issue is to show that the distinctiveness effect, a well known phenomenon of memory, can emerge in implicit memory tasks as well as in explicit memory tasks and this effect can be explained in specific terms of mechanisms (activation and multimodal integration) and not in terms of subjacent memory systems.Three series of experiments were elaborated. In the first series, we used an extrinsic contextual information associated to concepts. A categorization task permitted us to show that distinctiveness effect can appear within an implicit memory task.In the second series of experiments, we manipulated the distance between images to be categorized in an encoding phase. Thus, images presented were either more distant to each other, or more closer. Following an implicit retrieval, we highlighted that spatial distinctiveness hypothesis was validated. This means that spatially more distinct items during the encoding phase are associated to higher retrieval performances in the test phase.In the third series of experiments, we showed that performances related to distinctiveness effect in implicit and explicit memory tasks varied according to various levels of sensory dimension integration. This postulate was tested with free recall, lexical decision and recognition tasks.Finally, our experimental results suggest that performances from implicit and explicit memory tasks can be explained within the same single memory system. Thus, the integration process would be at the origin of this phenomenon.

This thesis investigated the use of novel combinations of multimodal metaphors in the presentation of learning information to evaluate the effect of these combinations on the usability of e-learning interfaces and on the users' learning performance. The empirical research described in this thesis comprised three experimental phases. In the first phase, an initial experiment was carried out with 30 users to explore and compare the usability and learning performance of facially animated expressive avatars with earcons and speech, and text with graphics metaphors. The second experimental phase involved an experiment conducted with 48 users to investigate their perception of avatar's facial expressions and body gestures when presented in both the absence and presence of interactive e-learning context. In addition, the experiment aimed at evaluating the role that an avatar could play as virtual lecturer in e-learning interfaces by comparing the usability and learning performance of three different modes of interaction: speaking facially expressive virtual lecturer, speaking facially expressive full-body animated virtual lecturer, and two speaking facially expressive virtual lecturers. In the third phase, a total of 24 users experimentally examined a novel approach for the use of earcons and auditory icons in e-learning interfaces to support an animated facially expressive avatar with body gestures during the presentation of the learning material. The obtained results demonstrated the usefulness of the tested metaphors to enhance e-learning usability and to enable users to attain better learning performance. These results provided a set of empirically derived innovative guidelines for the design and use of these metaphors to generate more usable e-learning interfaces. For example, when designing avatars as animated virtual lecturers in e-learning interfaces, specific facial expression and body gestures should be incorporated due to its positive influence in enhancing learners' attitude towards the learning process.

In this thesis, I combined field observations and lab experiments to explore the ecological significance of floral signals in a North American wildflower, Penstemon digitalis. More specifically, to determine the potential mechanisms driving selection on floral scent, I studied how scent mediates interactions with pollinators and antagonists by (1) observing spatiotemporal variation in scent emission (2), floral volatile ability to suppress microbes (3) the honest advertisement of nectar, and (4) if scent could aid pollinator learning by reinforcing visual signals. Scent sampling of flower development, flower tissues, rewards and inflorescence day/night emission, revealed a complexity in floral scent composition and emission that could reflect several ecological functions. The floral bouquet of P. digitalis was strongest when flowers opened, primarily emitted from flower nectaries and was strongest during the day when pollinators are most active, suggesting a role in plant-pollinator interactions. Because linalool was one of the few floral compounds found in nectar where microbe growth can degrade the pollinator reward, I studied its role in plant-microbe interactions. Bacteria strains isolated from floral and vegetative tissues were exposed to varying concentrations of nectar volatiles: linalool and methyl nicotinate. Linalool inhibited bacteria growth rate from all tissue origins whereas methyl nicotinate had little effect, suggesting that microbes could drive selection on linalool emission strength.    To determine the extent that linalool could honestly signal nectar availability, linalool-nectar associations were measured for inflorescences and flowers. Linalool predicted inflorescence nectar availability but not flower, exposing a limit to its honesty. Pollinator Bombus impatiens could use linalool as a foraging signal at varying concentrations, suggesting linalool could be learned and used to choose the most rewarding plants.    Measurement and comparison of signal-reward associations for both olfactory and visual signals/cues of P. digitalis displays found display size and linalool honest indicators of nectar. Lab behaviour experiments showed multiple signals correlated with reward could increase bumblebee foraging efficiency and promote learning, providing an explanation for why floral displays are complex and consist of multiple signals.    Together my results show that an integrated approach is required to understand the mechanisms driving the evolution of the floral phenotype.

The present study explores informal English language teaching and learning (ELTL) on the social media platform of Facebook. In contrast to a formal second/foreign language (L2/FL) education setting which is institutionally sponsored and highly structured, the context of informal ELTL under investigation is "recreational" (Chik, 2015) in the sense that it is unconstrained by institutional structures and largely driven by personal interests and goals. With the unprecedented success of social media and social networking sites (SNSs), this new form of learning and teaching can be found in a variety of languages and discourses across diverse digital landscapes. This study pays particular attention to the context of Thai speakers in three ELTL communities formed on Facebook Pages (FPs). The aim is to describe the participants' SNS-mediated L2 literacy practices as informed by Thai culture, beliefs, and values. This research project is grounded in the concept of new literacies (Lankshear & Knobel, 2011) and the multimodal social semiotic approaches to analyzing digitally-mediated communication (Kress, 2010; Jewitt, 2009; Van Leeuwen, 2005). The analysis is also informed by the interpretative framework of positioning theory (Davies and Harré, 1990; Harré & van Langenhove, 1991) and sociolinguistic approaches to stances (Du Bois, 2007; Jaffe, 2009a). The integration of various theoretical and analytical models offers a holistic understanding of the participants' Facebook-based literacy practices from different perspectives. A mixed method approach that combines qualitative (e.g., multimodal analysis, online ethnographic observation) and quantitative (e.g., survey, user statistics) data analysis also helps describe the users’ semiotic productions and interactions from a diachronic point of view. While the purpose of the project is to examine contemporary L2 literacy engagement in an underexplored historical and cultural context, the analysis does not simply discuss the way Thai SNS users teach and/or learn English online. Rather, the findings also shed light on other important issues relating to digital literacies including multimodal production, identity construction, social relationship formation, stance-taking acts, and language ideology. These emerging literacy practices are presented in three separate but interrelated analysis chapters. They are comprised of: 1) the multimodal analysis of the interplay between Facebook affordances and the users' semiotic activities; 2) the investigation of the participants’ self- and other-positioning strategies; and 3) the discussion of the participants' stance-taking acts in various aspects relating to English language teaching, learning, and use in contemporary society. Altogether, the findings pinpoint the complex and interconnected relationships among digital media, self, community, and ideology as fundamental to meaningful learning experiences on SNSs. They also support the view of language learning as a social practice, which highlights the fact that meaning and knowledge are socially and culturally situated, shaped by contexts, and shared by members within particular communities (Gee, 2010; Lankshear & Knobel, 2011; Thorne, 2013). The dissertation, therefore, has implications for pedagogies and practice because it provides insights about how to design equitable learning materials and activities that address these contemporary social practices. Ultimately, the research suggests the use of a social media-enhanced ELTL site to develop meaningful interactions between learner, language, media, and community during the process of learning.

The aim of this study is to gain a greater understanding of how companies use the social media platform Instagram to represent and promote a place. More precisely, it is of interest to analyze the media production of luxury tourism and how communication tools are applied in order to affect the consumers’ perceptions and increase tourism. In this study we examine the city Dubai’s official tourism account Visit Dubai on Instagram. We focus our research on how the social actors are represented in both image and text, but also the environments in which they are located. In order to investigate how the marketing company Visit Dubai uses semiotic tools to portray its city as a luxury city, we apply a multimodal critical discourse analysis (MCDA). The theoretical framework is therefore based on research regarding social semiotics and place branding. To achieve the aim of the study, the following questions are formulated: In what ways are social actors and the environments in which they are visually represented? In what ways are social actors and the environments they are placed represented linguistically? In what ways are visual and linguistic resources combined to create performances of Dubai as a tourist destination in the material? What discourses are articulated in Visit Dubai´s material and which relevant discourses are excluded? The result of the analysis shows that all 18 images examined can be related to either luxury, power or status. It is clear that Visit Dubai via their Instagram account, encourages and reinforces the need to achieve high social status, financial status and also to promote luxury consumption.

Den här studien syftar till att granska vad Systembolaget förmedlar i sina reklamfilmer och hur det förmedlas. Genom en kvalitativ utgångspunkt, den multimodala kritiska diskursanalysen och semiotiska resurser urskiljs dolda budskap i reklamfilmerna. De teoretiska utgångspunkterna i studien är social marknadsföring och den socialsemiotiska teorin vilka stärker och lyfter resultaten. Analysen av reklamfilmerna visar att Systembolaget använder sig av humoristisk, informativ och berättande strategi, samt att deras samhällsuppdrag enbart förmedlas till en viss del. Andra budskap som analysen urskiljer i de tre reklamfilmerna är Systembolagets syn på kvalité och deras värdering om kapitalism.

This study is an exploration of the relationship between subjectivity and pedagogy in the secondary school English classroom in South Africa during a time that can be characterised as one of considerable social change. It examines the subject positions students take up in relation to a teaching intervention that invites them to historicise their identities. In so doing, it seeks to contribute to the growing body of education research on how to meaningfully engage young people in post-conflict societies with their recent past and their shifting present, with the primary aim being to understand how these students are positioning themselves in relation to the changing sociopolitical context. The research was conducted in two Grade 11 English classrooms, one a de(re)segregated former Model C school and the other an elite private school. The research design is a two-case case study, employing ethnographic tools to generate a multi-layered and multifaceted understanding of the students’ engagement in all its forms. Poststructuralist theories on discourse and subjectivity form the theoretical framework for this study, informing both the methodology and the data analysis. At the heart of this lies Foucault’s notion of the discursively constructed subject, extended through the work of Stuart Hall, Chris Weedon, Bronwyn Davies and others in ways that facilitate their application to individual subjectivity, particularly in relation to the classroom as a pedagogically structured discursive space. The data is subjected to poststructuralist discourse analysis, adjusted to suit the mode and type of data which includes, inter alia, the analysis of a multimodal artefact, analysis of performative classroom talk and moment-by-moment analysis of classroom interaction. The analysis shows that students’ subjectivities are not fixed but shift in ways that are contingent on the pedagogic context. Such shifts are particularly noticeable when there is a shift in the interactional situation; when students move between different semiotic modes; or when they are provided with the opportunity for extended conversational interaction around an issue. In addition, students’ participation in the section of work on South Africa’s Truth and Reconciliation Commission (TRC) shows that engaging with the past in post-apartheid South African classrooms can have unpredictable results. Students’ resistance to engaging with recent history seems to be related to discomfort with the ways in which the grand narrative of the past works to position them in racialised ways. While there is evidence of students seeking to ‘unfix’ racialised subject positions, it is also clear that past discourses linger. Despite their desire to be rid of the past, students’ subject positions are frequently tied to their historically constructed locations in the sociopolitical and economic landscape of South Africa. These ambiguities and contradictions are viewed in part as a function of the complexity of the relationship between subjectivity and pedagogy, where what students are able to say and who they are able to be is shaped by the discursive structure of the classroom space. Ultimately it would seem that more serious consideration needs to be given to ways of developing a pedagogy that is able to tolerate contingency and heterogeneity and that would have relevance not only in post-conflict contexts but also beyond. Keywords: subjectivity, pedagogy, poststructuralist discourse analysis, positioning, identity, English classroom, TRC, multimodal artefact, classroom talk, South Africa