This study was designed to examine the effect of Science Writing Heuristic Approach on Students’ Learning of Multimodal Representations across 4th Grade to 8the Grade Levels. Multimodal representations in the forms of figures, tables, pictures, and charts are part of scientific language. A quasi-experimental design with control and treatment group of classes was used. Students completed the summary writing task by including multimodal representations in the both control and treatment classes. The students’ writing samples were evaluated with four measures of multimodal categories, including sign, functional, conceptual and embeddedness structures. To examine the differences of treatment and control groups and the effect of age, the Hierarchical Linear Modeling (HLM) analysis was used in this study. The HLM provides an opportunity to use statistical models that account for nesting of the data. Analysis of quantitative data indicated that the treatment classes significantly outperformed than the control classes on four measures of categories. Age also was a significant contributor to students’ learning of multimodal representations. Three key points emerged from the results. Firstly, the SWH approach had positive effects on students’ understanding of the multimodal representations. Secondly, the impact of the age was different for each category. Thirdly, the categories were used in this study had significant potential when exploring the students learning of multimodal representations. The study indicated some practical benefits that the strategy of promoting argumentative scientific language effectively was resulted in better communication, understanding of the topic with multimodal representations, and some transferring impacts of all these with the summary writing activities.

Ce travail se concentre sur une tâche de micro-navigation en plein air où le but est de naviguer vers une adresse de rue spécifiée en utilisant plusieurs modalités (par exemple, images, texte de scène et GPS). La tâche de micro-navigation extérieure s’avère etre un défi important pour de nombreuses personnes malvoyantes, ce que nous démontrons à travers des entretiens et des études de marché, et nous limitons notre définition des problèmes à leurs besoins. Nous expérimentons d’abord avec un monde en grille partiellement observable (Grid-Street et Grid City) contenant des maisons, des numéros de rue et des régions navigables. Ensuite, nous introduisons le Environnement de Trottoir pour la Navigation Visuelle (ETNV), qui contient des images panoramiques avec des boîtes englobantes pour les numéros de maison, les portes et les panneaux de nom de rue, et des formulations pour plusieurs tâches de navigation. Dans SEVN, nous formons un modèle de politique pour fusionner des observations multimodales sous la forme d’images à résolution variable, de texte visible et de données GPS simulées afin de naviguer vers une porte d’objectif. Nous entraînons ce modèle en utilisant l’algorithme d’apprentissage par renforcement, Proximal Policy Optimization (PPO). Nous espérons que cette thèse fournira une base pour d’autres recherches sur la création d’agents pouvant aider les membres de la communauté des gens malvoyantes à naviguer le monde. / This work focuses on an Outdoor Micro-Navigation (OMN) task in which the goal is to navigate to a specified street address using multiple modalities including images, scene-text, and GPS. This task is a significant challenge to many Blind and Visually Impaired (BVI) people, which we demonstrate through interviews and market research. To investigate the feasibility of solving this task with Deep Reinforcement Learning (DRL), we first introduce two partially observable grid-worlds, Grid-Street and Grid City, containing houses, street numbers, and navigable regions. In these environments, we train an agent to find specific houses using local observations under a variety of training procedures. We parameterize our agent with a neural network and train using reinforcement learning methods. Next, we introduce the Sidewalk Environment for Visual Navigation (SEVN), which contains panoramic images with labels for house numbers, doors, and street name signs, and formulations for several navigation tasks. In SEVN, we train another neural network model using Proximal Policy Optimization (PPO) to fuse multi-modal observations in the form of variable resolution images, visible text, and simulated GPS data, and to use this representation to navigate to goal doors. Our best model used all available modalities and was able to navigate to over 100 goals with an 85% success rate. We found that models with access to only a subset of these modalities performed significantly worse, supporting the need for a multi-modal approach to the OMN task. We hope that this thesis provides a foundation for further research into the creation of agents to assist members of the BVI community to safely navigate.