Doctor of Philosophy / Department of Physics / N. Sanjay Rebello / Research has demonstrated that attentional cues overlaid on diagrams and animations can help students attend to the relevant areas and facilitate problem solving. In this study we investigate the influence of visual cues and outcome feedback on students’ problem solving, performance, reasoning, and visual attention as they solve conceptual physics problems containing a diagram. The participants (N=90) were enrolled in an algebra-based physics course and were individually interviewed. During each interview students solved four problem sets while their eye movements were recorded. The problem diagrams contained regions that were relevant to solving the problem correctly and separate regions related to common incorrect responses. Each problem set contained an initial problem, six isomorphic training problems, and a transfer problem. Those in the cued condition saw visual cues overlaid on the training problems. Those in the feedback conditions were told if their responses (answer and explanation) were correct or incorrect. Students’ verbal responses were used to determine their accuracy. The study produced two major findings. First, short duration visual cues coupled with correctness feedback can improve problem solving performance on a variety of insight physics problems, including transfer problems not sharing the surface features of the training problems, but instead sharing the underlying solution path. Thus, visual cues can facilitate re-representing a problem and overcoming impasse, enabling a correct solution. Importantly, these cueing effects on problem solving did not involve the solvers’ attention necessarily embodying the solution to the problem. Instead, the cueing effects were caused by solvers attending to and integrating relevant information in the problems into a solution path. Second, these short duration visual cues when administered repeatedly over multiple training problems resulted in participants becoming more efficient at extracting the relevant information on the transfer problem, showing that such cues can improve the automaticity with which solvers extract relevant information from a problem. Both of these results converge on the conclusion that lower-order visual processes driven by attentional cues can influence higher-order cognitive processes associated with problem solving.

Although language usually occurs in an interactive and world-situated context (Clark, 1996), most research on language use to date has studied comprehension and production in isolation. This thesis combines research on comprehension and production, and explores the links between them. Its main focus is on the coordination of visual attention between speakers and listeners, as well as the influence this has on the language they use and the ease with which they understand it. Experiment 1 compared participants’ eye movements during comprehension and production of similar sentences: in a syntactic priming task, they first heard a confederate describe an image using active or passive voice, and then described the same kind of picture themselves (cf. Branigan, Pickering, & Cleland, 2000). As expected, the primary influence on eye movements in both tasks was the unfolding sentence structure. In addition, eye movements during target production were affected by the structure of the prime sentence. Eye movements in comprehension were linked more loosely with speech, reflecting the ongoing integration of listeners’ interpretations with the visual context and other conceptual factors. Experiments 2-7 established a novel paradigm to explore how seeing where a speaker was looking during unscripted production would facilitate identification of the objects they were describing in a photographic scene. Visual coordination in these studies was created artificially through an on-screen cursor which reflected the speaker’s original eye movements (cf. Brennan, Chen, Dickinson, Neider, & Zelinsky, 2007). A series of spatial and temporal manipulations of the link between cursor and speech investigated the respective influences of linguistic and visual information at different points in the comprehension process. Implications and potential future applications are discussed, as well as the relevance of this kind of visual cueing to the processing of real gaze in face-to-face interaction.

L'analyse temps réel de la masse de données générée par les mécanismes de gestion de la vision dans les applications interactives est un problème toujours ouvert, promettant des avancées importantes dans des domaines aussi variés que la robotique, l’apprentissage à distance ou les nouvelles formes d’interactions avec l’utilisateur, sans clavier ni souris. Dans le cadre général de la vision, les algorithmes d’analyse de scène doivent trouver un compromis entre d'une part la qualité des résultats recherchés et d'autre part la quantité de ressources allouable aux différents tâches. Classiquement, ce choix est effectué à la conception du système (sous la forme de paramètres et d’algorithmes prédéfinis), mais cette solution limite le champ d’application de celui-ci. Une solution plus flexible consiste à utiliser un système de vision adaptatif qui pourra modifier sa stratégie d’analyse en fonction des informations disponibles concernant son contexte d’exécution. En conséquence, ce système doit posséder un mécanisme permettant de guider rapidement et efficacement l’exploration de la scène afin d’obtenir ces informations. Chez l’homme, les mécanismes de l’évolution ont mis en place le système d’attention visuelle. Ce système sélectionne les informations importantes afin de réduire la charge cognitive et les ambiguïtés d’interprétation de la scène. Nous proposons, dans cette thèse, un système d'attention visuelle, dont nous définissons l’architecture et les principes de fonctionnement. Ce dernier devra permettre l’interaction avec un système de vision afin qu’il adapte ses traitements en fonction de l’intérêt de chacun des éléments de la scène, i.e. ce que nous appelons saillance. A la croisée des chemins entre les modèles centralisés et hiérarchiques (ex : [Koch1985], puis [Itti1998]), et les modèles distribués et compétitifs (ex : [Desimone1995], puis [Deco2004, Rolls2006]), nous proposons un modèle hiérarchique, compétitif et non centralisé. Cette approche originale permet de générer un point de focalisation attentionnel à chaque pas de temps sans utiliser de carte de saillance ni de mécanisme explicite d’inhibition de retour. Ce nouveau modèle computationnel d'attention visuelle temps réel est basé sur un système d'équations proies / prédateurs, qui est bien adapté pour l'arbitrage entre un comportement attentionnel non déterministe et des propriétés de stabilité, reproductibilité, et réactivité. L'analyse des expérimentations menées est positive : malgré le comportement non-déterministe des équations proies / prédateurs, ce système possède des propriétés intéressantes de stabilité, reproductibilité, et réactivité, tout en permettant une exploration rapide et efficace de la scène. Ces propriétés ouvrent la possibilité d’aborder différents types d’applications allant de l’évaluation de la complexité d’images et de vidéos à la détection et au suivi d’objets. Enfin, bien qu’il soit destiné à la vision par ordinateur, nous comparons notre modèle au système attentionnel humain et montrons que celui-ci présente un comportement aussi plausible (voire plus en fonction du comportement défini) que les modèles classiques existants. / Providing real time analysis of the huge amount of data generated by computer vision algorithms in interactive applications is still an open problem. It promises great advances across a wide variety of fields : robotics, distance education, or new mouse-less and keyboard-less human computer interaction.When using scene analysis algorithms for computer vision, a trade-off must be found between the quality of the results expected, and the amount of computer resources allocated for each task. It is usually a design time decision, implemented through the choice of pre-defined algorithms and parameters. However, this way of doing limits the generality of the system. Using an adaptive vision system provides a more flexible solution as its analysis strategy can be changed according to the information available concerning the execution context. As a consequence, such a system requires some kind of guiding mechanism to explore the scene faster and more efficiently.In human, the mechanisms of evolution have generated the visual attention system which selects the most important information in order to reduce both cognitive load and scene understanding ambiguity.In this thesis, we propose a visual attention system tailored for interacting with a vision system (whose theoretical architecture is given) so that it adapts its processing according to the interest (or salience) of each element of the scene.Somewhere in between hierarchical salience based (ex: [Koch1985], then [Itti1998]) and competitive distributed (ex: [Desimone1995], then [Deco2004, Rolls2006]) models, we propose a hierarchical yet competitive and non salience based model. Our original approach allows the generation of attentional focus points without the need of neither saliency map nor explicit inhibition of return mechanism. This new real-time computational model is based on a preys / predators system. The use of this kind of dynamical system is justified by an adjustable trade-off between nondeterministic attentional behavior and properties of stability, reproducibility and reactiveness.Our experiments shows that despite the non deterministic behavior of preys / predators equations, the system exhibits interesting properties of stability, reproducibility and reactiveness while allowing a fast and efficient exploration of the scene. These properties are useful for addressing different kinds of applications, ranging from image complexity evaluation, to object detection and tracking. Finally, while it is designed for computer vision, we compare our model to human visual attention. We show that it is equally as plausible as existing models (or better, depending on its configuration).

ABSTRACT Autism Spectrum Disorder (ASD) is often accompanied by atypical attention to faces. Some previous studies have suggested that children with ASD demonstrate strengths when processing visual information from cartoons, whereas others have argued that photographic stimuli confer benefits. No previous studies have compared photograph and cartoon images of faces (i.e., Boardmaker [BM] images) in the context of a Social Story™ (Gray, 2010): a common intervention to support behavior and social cognition in children with ASD. In this study, we examined visual attention to static face stimuli in the context of Social Stories™. Participants were 19 typically developing (TD) children and 18 age-matched children with ASD. We addressed two questions: 1) Is there a difference between TD children and children with ASD in how they attend to cartoon and photographic stimuli in the context of a Social Story™? and 2) Do group differences in visual attention to BM and/or photographic stimuli correlate with age and indices of autism severity, executive function, intellectual functioning, and weak central coherence? With regard to question 1 and with one exception, we found no differences between groups when viewing images of faces. The exception involved our cartoon and photograph images that differed in content from the other face images in that they represented a person's full body as well as a range of objects (i.e., it was a more complex scene). For these images an interaction was observed such that the TD and ASD groups were no different in their looking patterns in the BoardMaker condition but they were different in the photograph condition. More specifically, we found that a shift toward more mouth-looking in the photograph condition among children with ASD was negatively associated with attention shifting and verbal IQ and that a shift toward more "other"-looking (i.e., looking that occurred outside the eye and mouth region of the face) was negatively associated with attention shifting, age, and central coherence. These findings suggest that children with ASD demonstrate typical visual attention patterns to both cartoon and photographic stimuli representing faces but that children with ASD employ an atypical scanning strategy when presented with photographic stimuli representing more complex social scenes. The theoretical and clinical implications of the findings are discussed.

As there is a rapid growth in the field of wireless communications, the demand for various multimedia services is also increasing. The data that is being transmitted suffers from distortions through source encoding and transmission over errorprone channels. Due to these errors, the quality of the content is degraded. There is a need for service providers to provide certain Quality of Experience (QoE) to the end user. Several methods are being developed by network providers for better QoE.The human tendency mainly focuses on distortions in the Region of Interest(ROI) which are perceived to be more annoying compared to the Background(BG). With this as a base, the main aim of this thesis is to get an accurate prediction quality metric to measure the quality of the image over ROI and the BG independently. Reduced Reference Image Quality Assessment (RRIQA), a reduced reference image quality assessment metric, is chosen for this purpose. In this method, only partial information about the reference image is available to assess the quality. The quality metric is measured independently over ROI and BG. Finally the metric estimated over ROI and BG are pooled together to get aROI aware metric to predict the Mean Opinion Score (MOS) of the image.In this thesis, an ROI aware quality metric is used to measure the quality of distorted images that are generated using a wireless channel. The MOS of distorted images are obtained. Finally, the obtained MOS are validated with the MOS obtained from a database [1].It is observed that the proposed image quality assessment method provides better results compared to the traditional approach. It also gives a better performance over a wide variety of distortions. The obtained results show that the impairments in ROI are perceived to be more annoying when compared to the BG.

Nowadays, there exist a huge number of end devices with different screen properties for watching television content, which is either broadcasted or transmitted over the internet. To allow best viewing conditions on each of these devices, different image formats have to be provided by the broadcaster. Producing content for every single format is, however, not applicable by the broadcaster as it is much too laborious and costly. The most obvious solution for providing multiple image formats is to produce one high resolution format and prepare formats of lower resolution from this. One possibility to do this is to simply scale video images to the resolution of the target image format. Two significant drawbacks are the loss of image details through ownscaling and possibly unused image areas due to letter- or pillarboxes. A preferable solution is to find the contextual most important region in the high-resolution format at first and crop this area with an aspect ratio of the target image format afterwards. On the other hand, defining the contextual most important region manually is very time consuming. Trying to apply that to live productions would be nearly impossible. Therefore, some approaches exist that automatically define cropping areas. To do so, they extract visual features, like moving reas in a video, and define regions of interest (ROIs) based on those. ROIs are finally used to define an enclosing cropping area. The extraction of features is done without any knowledge about the type of content. Hence, these approaches are not able to distinguish between features that might be important in a given context and those that are not. The work presented within this thesis tackles the problem of extracting visual features based on prior knowledge about the content. Such knowledge is fed into the system in form of metadata that is available from TV production environments. Based on the extracted features, ROIs are then defined and filtered dependent on the analysed content. As proof-of-concept, this application finally adapts SDTV (Standard Definition Television) sports productions automatically to image formats with lower resolution through intelligent cropping and scaling. If no content information is available, the system can still be applied on any type of content through a default mode. The presented approach is based on the principle of a plug-in system. Each plug-in represents a method for analysing video content information, either on a low level by extracting image features or on a higher level by processing extracted ROIs. The combination of plug-ins is determined by the incoming descriptive production metadata and hence can be adapted to each type of sport individually. The application has been comprehensively evaluated by comparing the results of the system against alternative cropping methods. This evaluation utilised videos which were manually cropped by a professional video editor, statically cropped videos and simply scaled, non-cropped videos. In addition to and apart from purely subjective evaluations, the gaze positions of subjects watching sports videos have been measured and compared to the regions of interest positions extracted by the system.

Doctor of Philosophy / Department of Physics / N. Sanjay Rebello / The work described here represents an effort to understand and influence visual attention while solving physics problems containing a diagram. Our visual system is guided by two types of processes -- top-down and bottom-up. The top-down processes are internal and determined by ones prior knowledge and goals. The bottom-up processes are external and determined by features of the visual stimuli such as color, and luminance contrast. When solving physics problems both top-down and bottom-up processes are active, but to varying degrees. The existence of two types of processes opens several interesting questions for physics education. For example, how do bottom-up processes influence problem solvers in physics? Can we leverage these processes to draw attention to relevant diagram areas and improve problem-solving? In this dissertation we discuss three studies that investigate these open questions and rely on eye movements as a primary data source. We assume that eye movements reflect a person’s moment-to-moment cognitive processes, providing a window into one’s thinking. In our first study, we compared the way correct and incorrect solvers viewed relevant and novice-like elements in a physics problem diagram. We found correct solvers spent more time attending to relevant areas while incorrect solvers spent more time looking at novice-like areas. In our second study, we overlaid these problems with dynamic visual cues to help students’ redirect their attention. We found that in some cases these visual cues improved problem-solving performance and influenced visual attention. To determine more precisely how the perceptual salience of diagram elements influenced solvers’ attention, we conducted a third study where we manipulated the perceptual salience of the diagram elements via changes in luminance contrast. These changes did not influence participants’ answers or visual attention. Instead, similar to our first study, the time spent looking in various areas of the diagram was related to the correctness of an answer. These results suggest that top-down processes dominate while solving physics problems. In sum, the study of visual attention and visual cueing in particular shows that attention is an important component of physics problem-solving and can potentially be leveraged to improve student performance.

Existe interesse crescente em informações científicas sobre o papel da percepção visual e da atenção visuo-espacial nos esportes. Comparamos os Tempos de Reação (TR) de atletas profissionais com os de não atletas em 5 condições de atenção visual voluntária: difusa, manifesta, encoberta (à esquerda e à direita do ponto de fixação) e dividida. Os atletas apresentaram TR médios menores que os não atletas em todos os experimentos. Diferenças em TRs a estímulos visuais dentro e fora dos focos atencionais não foram significativas (p>0,31), exceto na condição de atenção manifesta (p=0,0001), mas foram ligeiramente maiores em atletas. Tais diferenças foram significativas em experimentos com outros grupos de interesse do nosso laboratório, provavelmente devido a pequenas mudanças no protocolo experimental (detalhes nas instruções fornecidas aos voluntários). Importante salientar que na literatura existem resultados contraditórios na comparação entre atletas e não atletas em tarefas atencionais, provavelmente também por pequenas diferenças nos protocolos experimentais. / There is growing interest in scientific information on the role of visual perception and visuospatial attention in sports. We compared Reaction Times (RT) of professional athletes with those of non-athletes in 5 conditions of voluntary visual attention: diffuse, manifest, covert (to the right and to the left of gaze) and divided. Athletes showed shorter mean RT than non-athletes in all experiments. Differences in RT to visual stimuli inside and outside the attentional focus were not significant (p>0,31), except under manifest attention conditions (p=0,0001), but where slightly higher in athletes. These differences were significant in experiments on other populations of interest in our laboratory, probably due to small changes in the experimental protocol (details in the instructions provided to volunteers). It is important to note that the literature presents contradictory results in the comparison between athletes and non-athletes in attentional tasks, again, probably due to small differences in experimental protocol.

Avaliamos a orientação da atenção visual de pacientes com Doença de Parkinson (DP), idosos e jovens saudáveis. Foram feitos cinco experimentos de Tempo de Reação (TR) e de Julgamento de Ordem Temporal (JOT), nos quais se avaliaram a atenção automática e voluntária e a percepção de ordem temporal. Os pacientes com DP apresentaram déficits em relação aos idosos na atenção voluntária, na percepção de ordem temporal e na flexibilidade mental, mas tiveram resultados semelhantes aos idosos na avaliação da atenção automática. A correlação entre os valores do TR e do JOT foi significativa entre os experimentos de TR e JOT, o que evidencia que os déficits dos DP e idosos têm origem num mecanismo em comum: a atenção. / We investigated the orienting of visual attention in patients with Parkinson?s disease (PD), healthy elderly and young subjects. Five Reaction Time (RT) and Temporal Order Judgment (TOJ) experiments were performed in order to evaluate automatic and voluntary attention and perception of temporal order. The PD patients exhibited a deficit when compared to the healthy elderly subjects in voluntary attention, perception of temporal order and mental flexibility, but had similar results in the automatic attention task. The correlation between the values obtained in RT and TOJ experiments was significant, which points towards a common mechanism underlying deficits of PD patients and effects of aging: attention.

Cada vez mais observa-se a inserção de novas tecnologias em salas de aulas. Com o auxílio de políticas publicas, computadores ligados a internet tem estado presentes nas classes nos mais longinquos lugares do nosso país. Isto tem proporcionado que o conhecimento chegue de forma mais ampla e irrestrita a todas as crianças em fase de desenvolvimento. Na ultima decada, além de microcomputadores, tem-se observado a presença, em salas de aulas, de Ipads, celulares, cujos proprietários são os próprios alunos e até mesmo lousa eletrônica em escolas com poder aquisitivo maior. Aliado a isto, nota-se também a inserção de kits robóticos que tem motivado muito os alunos no aprendizado de raciocínio lógico e de programação, pois, eles experimentam o conceito: \"aprender por meio do fazer\". O uso de todas estas tecnologias tem como objetivo principal cativar a atenção dos alunos, incentivar a pesquisa e o aprendizado interativo, uma vez que o ensino antes expositivo dá lugar ao ensino interativo, isto é, que conta com a participação mais ativa do estudante. Nesta direção, esta dissertação de Mestrado traz uma inovação no sentido que está sendo proposto um sistema que permite que um robô humanoide seja inserido em salas de aulas. Trata-se de um protótipo que permite que o robô reconheça figuras geométricas planas, que pode ser estendido para outros tipos de conteúdos. O objetivo é a integração de um sistema de visão computacional em um ambiente de controle de um robô humanoide para torná-lo capaz de reconhecer figuras geométricas planas, para ser utilizada como uma ferramenta de ensino. Este sistema de visão é baseado em técnicas de Atenção Visual e utiliza uma rede neural LEGION para segmentar os objetos mais salientes da imagem e uma rede neural do tipo Multicamadas (MLP), para realizar a classificação desses objetos. Graças a este sistema de visão, o robô consegue discernir figuras sobrepostas independente do ambiente real no qual esteja inserido. Para avaliar o desempenho do sistema proposto, algumas aplicações foram desenvolvidas que envolveram a participação de crianças interagindo com o robô no reconhecimento de figuras geométricas. Embora sejam necessários uma maior numero de experimentos, os resultados obtidos indicam que o sistema proposto apresenta-se como uma ferramenta alternativa, promissora e interessante, tendo sida muito bem recebida por parte dos alunos e professores. / Increasingly there has been the introduction of new technologies in the classroom. With the help of public policies, computers connected to the internet has been used in the classes in the far reaches of our country. This has provided the knowledge reachs broader and unrestricted way to all children under development. In the last decade, besides computers, has been seen the presence in classrooms of Ipads, smart phones, owned by the students themselves and even electronic whiteboard in schools with higher purchasing power. Added to this, there is also the inclusion of robotic kits that has motivated much students in learning logical reasoning and programming, as they experience the concept: \"learning through doing\". The use of all these technologies aims to captivate the attention of students, encourage research and interactive learning, since the school, before exhibition, gives way to interactive teaching, that is, who has the most active student participation. In this direction, this Masters thesis brings an innovation in the sense that is being proposed a system that allows a robot humanoid is inserted into classrooms. It is a prototype that allows the robot to recognize planar geometric figures, which can be extended to other types of content. The goal is the integration of a computer vision system in a control a humanoid robot environment to make it able to recognize This has provided the knowledge gets broader and unrestricted way all children under development geometric figures, to be used as a teaching tool. This vision system is based on Visual Attention techniques and uses a neural network LEGION to target the salient objects image and a Multilayer (MLP) neural network, to perform the classification of these objects. Thanks to the vision system, the robot can distinguish independent of the actual environment in which overlapping figures is inserted. To evaluate the performance of the proposed system, some applications were developed that involved the participation of children interacting with the robot in the recognition of geometric figures. Although a larger number of experiments are needed, the results indicate that the proposed system is presented as an alternative tool, promising and interesting, and it was very well received by students and teachers.