This thesis is a systematic investigation of preparatory reorienting of task-relevant spatial attention. Task switching experiments typically report a performance overhead when the current task is different to that performed on the previous trial relative to when the task repeats. This ‘switch cost’ tends to reduce as participants are given more time to prepare (consistent with an active reconfiguration process) but a ‘residual’ switch cost usually remains even at very long preparation intervals (often interpreted as evidence of carryover of response selection parameters from the previous trial which are immune to preparation). Although attentional selection of perceptual attributes is often considered to be part of task-set and is included in some models of task-set control, little research has investigated the dynamics of this component in detail. Over a series of seven experiments in which tasks were consistently mapped to screen locations, eye-tracking was used to systematically investigate task-relevant spatial selection of perceptual attributes during the preparation interval and early after stimulus onset. Experiment 1 revealed a switch-induced delay in appropriate attention orientation and a measure of ‘attentional inertia’ which could not be explained by task-independent re-orienting to locations or low-level oculomotor phenomena but were markers of task-relevant spatial selection. Experiment 2 provided a sensitive measure of both of these attentional handicaps and demonstrated that they both contribute to the switch cost (including its residual component). Although attentional inertia reduced with preparation, both handicaps were present at the longest preparation intervals. The constancy of the delay in attending to the relevant attribute reflects the effort to re-allocate attention, rather than peculiarities of spatial orienting when the cue and stimulus are presented near-simultaneously on trials with short cue-stimulus intervals. The presence of attentional inertia in blocks with long preparation intervals suggested some component of inertia immune to preparation (though see Experiments 5 and 6 below). Experiments 3 and 4 investigated the extent to which attentional selection can be decoupled from other task-set components. Cues which explicitly provided location information reduced (or eliminated) the attentional effects found in Experiment 2 indicating that attentional selection can be decoupled from other task-set components. However, Experiment 3 found that the ‘natural’ state is for attentional selection to be coupled at least to a degree (and accessed via) task-set. Experiment 5 combined eye-tracking with ERPs to investigate the relative order of attentional selection and reconfiguration of other task-set components. A well-documented ERP marker of task-set preparation always followed onset of the first fixation on the currently relevant stimulus element indicating that (at least some) task-set components are reconfigured in a serial order with spatial selection preceding other components (e.g., loading of S-R rules or other parameters into working memory). Experiments 6 and 7 investigated the nature of attentional inertia. In Experiment 6 participants were given ultimate control over the duration of the preparation interval which eliminated attentional inertia (at least as indexed by preferential fixation of the previously relevant element on switch trials). In Experiment 7 the stimulus comprised three items which were from perceptually distinct classes (digits, letters, objects) to investigate whether the presence of task-specific features would elicit extra attentional inertia and whether early spatial selection was effective enough to block the processing of task-irrelevant features once the stimulus was presented. Although there was some evidence that the previously relevant stimulus element ‘captured’ attention, this tendency was modest in the fixations and absent in performance measures (response congruence effects).

This thesis investigates the mechanisms underlying the formation, maintenance, and sharing of reference in tasks in which language and vision interact. Previous research in psycholinguistics and visual cognition has provided insights into the formation of reference in cross-modal tasks. The conclusions reached are largely independent, with the focus on mechanisms pertaining to either linguistic or visual processing. In this thesis, we present a series of eye-tracking experiments that aim to unify these distinct strands of research by identifying and quantifying factors that underlie the cross-modal interaction between scene understanding and sentence processing. Our results show that both low-level (imagebased) and high-level (object-based) visual information interacts actively with linguistic information during situated language processing tasks. In particular, during language understanding (Chapter 3), image-based information, i.e., saliency, is used to predict the upcoming arguments of the sentence, when the linguistic material alone is not sufficient to make such predictions. During language production (Chapter 4), visual attention has the active role of sourcing referential information for sentence encoding. We show that two important factors influencing this process are the visual density of the scene, i.e., clutter, and the animacy of the objects described. Both factors influence the type of linguistic encoding observed and the associated visual responses. We uncover a close relationship between linguistic descriptions and visual responses, triggered by the cross-modal interaction of scene and object properties, which implies a general mechanism of cross-modal referential coordination. Further investigation (Chapter 5) shows that visual attention and sentence processing are closely coordinated during sentence production: similar sentences are associated with similar scan patterns. This finding holds across different scenes, which suggests that coordination goes beyond the well-known scene-based effects guiding visual attention, again supporting the existence of a general mechanism for the cross-modal coordination of referential information. The extent to which cross-modal mechanisms are activated depends on the nature of the task performed. We compare the three tasks of visual search, object naming, and scene description (Chapter 6) and explore how the modulation of cross-modal reference is reflected in the visual responses of participants. Our results show that the cross-modal coordination required in naming and description triggers longer visual processing and higher scan pattern similarity than in search. This difference is due to the coordination required to integrate and organize visual and linguistic referential processing. Overall, this thesis unifies explanations of distinct cognitive processes (visual and linguistic) based on the principle of cross-modal referentiality, and provides a new framework for unraveling the mechanisms that allow scene understanding and sentence processing to share and integrate information during cross-modal processing.

Ingeniero Civil Industrial / El presente Trabajo de Título tiene como objetivo principal conocer el impacto que significa la incorporación de una nueva fuente de información a la metodología de identificación de Website Keyobjects. Esta nueva fuente de información será la medida de la actividad bioeléctrica cerebral frente a los estímulos presentados en una página web. Específicamente, se busca diseñar e implementar un módulo con algoritmos de Data Mining para clasificar los objetos relevantes presentes en un sitio web según variables de actividad cerebral. El trabajo se enmarca en el proyecto Fondef Plataforma informática basada en web-intelligence y herramientas de análisis de exploración visual para la mejora de la estructura y contenido de sitios web (AKORI: Advanced Kernel for Ocular Research and Web Intelligence) . Este proyecto está siendo llevado a cabo en conjunto por el Laboratorio de Neurosistemas y el Departamento de Ingeniería Industrial de la Universidad de Chile. La hipótesis que se valida en este trabajo es: "Es posible realizar una clasificación de los objetos relevantes de un sitio web según variables que caractericen la actividad bioelétrica cerebral". Para la validación de esta hipótesis se realiza un experimento en que 20 usuarios navegan libremente por un sitio web de estudio, mientras que un dispositivo de Eye Tracking guarda el posicionamiento de los ojos en la pantalla y los cambios en el tamaño de las pupilas y un electroencefalograma graba los potenciales eléctricos en la corteza cerebral en tiempo real. El análisis de los datos obtenidos se realiza mediante el proceso KDD, en un principio se realiza para la dilatación pupilar con el fin de obtener una línea base de estudio. Luego se aplica a los datos del EEG para realizar una comparación a lo obtenido de las señales oculares. Para ambos tipos de data se seleccionan 19 sujetos, cuyos datos son preprocesados y limpiados, luego transformados según distintas características a los que se les aplican diversos algoritmos de clasificación. Con esto se determina como resultado una lista de objetos relevantes dentro del sitio de estudio. Los resultados obtenidos indican que utilizando variables obtenidas de las señales eléctricas producidas en la corteza cerebral es posible clasificar Website Keyobjects con un 90\% de precisión, mediante el algoritmo K-Means. Esta clasificación es en base a la línea base obtenida, donde de un total de 20 objetos, 18 fueron clasificados correctamente, y fue la mejor que se obtuvo dentro de todas las combinaciones de variables y clusterizaciones realizadas. Finalmente se concluye que el trabajo fue exitoso y se proponen diversos trabajos futuros que aportan al proyecto AKORI y la metodología. Además se entrega una reflexión final correspondiente al uso de metodologías, algoritmos y conocimientos similares en otras áreas de estudio que generen valor para la sociedad.

Ingeniero Civil Industrial / El proyecto AKORI está pensado como una plataforma informática basada en técnicas de Web Intelligence, y análisis de la exploración visual para mejorar la estructura y los contenidos de un sitio web. En su etapa de investigación, se han incorporado estudios relacionados con la dilatación pupilar, el posicionamiento ocular y análisis de la actividad cerebral de los usuarios web. Una de las líneas de investigación indagó en la relación entre la dilatación pupilar y la toma de decisión de los usuarios de sitios web. Para profundizar en este estudio, se pretende realizar una investigación exploratoria, relacionando la toma de decisión con un nuevo marcador fisiológico. El objetivo general es determinar si existe una relación entre el potencial eléctrico producido por la actividad cerebral y la toma de decisión de un usuario en un sitio web. La hipótesis de investigación se plantea de la siguiente manera: El potencial bioeléctrico producido por la actividad cerebral puede predecir la toma de decisión de un usuario web . La metodología de esta investigación considera cuatro etapas: Análisis del estado del arte en el área de la neurociencia y técnicas de Data Mining; Diseño e implementación de experimentos; Análisis y procesamiento de los datos para validar la hipótesis de investigación y la construcción del clasificador; y finalmente comparar los resultados de la metodología obtenidas anteriormente. Para obtener los datos, se utilizaron herramientas de Eye-Tracking y EEG, en experimentos que simulan una situación discriminatoria dentro de un sitio web, mostrando imágenes de valencia neutra y donde el usuario debía escoger una de ellas. Para poder realizar extrapolación, se experimentó con 20 sujetos, de los cuales 18 finalizaron con la totalidad de los registros. Para analizar los datos, se separaron las observaciones que coinciden con el objeto elegido y las que no. Aquellas que tenía más de 600ms. de duración, fueron promediadas y graficadas. Al graficar cada una de las curvas, no se observan diferencias significativas. Sin embargo, se observa un patrón de comportamiento común, que consiste en la disminución de la actividad cerebral en los primeros 100ms. y un alza abrupta en la actividad durante los siguientes 200ms. Dado que no hay diferencias significativas, se realizaron dos tipos de caracterizaciones descripción estadística y transformada de Wavelet, que finalmente no permitieron validar la hipótesis de este trabajo. Si bien, la hipótesis de investigación no fue validada con los parámetros descritos, se reconoce que existen variables fisiológicas que constituyen una fuente de información que puede ser aplicadas en diversos ámbitos, ya sea en web-intelligence, web-mining, u otra disciplina ajena a la medicina pura.

Multiple elements of the retail environment can have an impact on a consumer's behaviour and purchase decisions. Much of the influence that the environment has on behaviour often goes unnoticed, as it affects internal processes that happen below the level of conscious awareness. This research aims to explore and quantify the effect a retail environment has on consumers' affective (emotional) and cognitive responses towards products. Priming is the influence of external stimuli on one's behaviour or response towards target stimuli. This research designed an experiment to prime participants with a particular coloured stimulus (pink, blue, or red) in order to measure the influence of this prime on the participants' purchase decisions. The participants entered a real-world simulated retail shop, and within a guided format they shopped through the available dresses, eventually picking out their three ranked favourites. The participants' physiological responses were measured using an eye-tracker and a portable Electroencephalogram (EEG) recording unit. The eye-tracking data were analysed using the Gaze Cascade Theory, testing for an increase in gaze bias towards preferred and primed products. The EEG data provided information about the participants' brain activity, and were analysed in accordance with Davidson's model of emotion, indicating an approach or withdrawal tendency towards different products. The results showed that with both eye-tracking and EEG it is possible to measure a difference between the participants' cognitive and affective responses towards the products that they preferred and chose as their favourites, compared with the products they did not choose. The EEG data provided evidence of a difference in neural responses between the prime matching coloured products and the non-prime matching products. However, the eye-tracking responses did not demonstrate a significant difference in eye-movements between the primed and not primed products. Technical innovation was required to allow the recording of EEG data in the semi-controlled shop environment, to allow data free of motion artefacts to be analysed. These results demonstrate the ability to measure consumers' physiological, neural, and subconscious responses in a real-world retail environment, whilst allowing the participants to move freely and unhindered. A novel methodology for analysing motion artefact free EEG data is presented. The results demonstrate a significant difference in emotional responses, as detected by EEG, in preference towards the prime coloured products, suggesting that priming has an influence in decision making in fashion retail environments.

archives@tulane.edu / Despite the increasing use of keyboards in the classroom, handwriting is still considered a fundamental skill that young children need to master to succeed in most areas of the elementary school curriculum. Children’s school readiness is determined by an array of cognitive, perceptual, and motor abilities that provide the foundation for academic success. These foundational abilities must be integrated for efficient handwriting and a failure of this integration predicts later academic achievement. When children begin writing, they develop from producing incoherent lines to producing letters that are more common in their language or are found in their names. Children must then learn to produce the remaining letters in their alphabet to become proficient writers. The process that children use to master the letters of an alphabet is not well understood. Previous research has focused primarily on the production of handwriting as a single skill, yet the production of handwriting entails a process in which children must integrate visual, fine motor, and basic reading skills to produce letters and words. The process of handwriting also develops in an environment that is full of letters and words. To date, the literature on handwriting has failed to address the processes that children utilize during handwriting, how these processes change through children’s development, and how the environment children are developing in influences these processes. This dissertation provides a theoretically integrative account of children’s handwriting development. The objectives of this dissertation are to determine the letter frequencies in children’s picture books, determine the opportunities that children are presented to copy letters in handwriting workbooks, measure the influence of change in grade and growth of basic reading skills on children’s visual processing development during handwriting, and measure visual-motor integration during handwriting. The approach to test these objectives is to integrate methods from educational and developmental psychology literatures in a novel series of studies using content analyses of children’s educational resources, head-mounted eye-tracking and academic assessments. By understanding the interaction between cognitive, linguistic, visual, and motor processes, researchers may establish possible mechanisms for the process of children’s development of complex skills, such as handwriting. / 1 / Nicholas E. Fears

The present four studies are the first to examine the effect of presentation time on accurate facial expression discrimination while concurrently using eye movement monitoring to ensure fixation to specific features during the brief presentation of the entire face. Recent studies using backward masking and evaluating accuracy performance with signal detection methods (A’) have identified a happy-face advantage however differences between other facial expressions of emotion have not been reported. In each study, a specific exposure time before mask (150, 100, 50, or 16.67 ms) and eight different fixation locations were used during the presentation of neutral, disgusted, fearful, happy, and surprised expressions. An effect of emotion was found across all presentation times such that the greatest performance was seen for happiness, followed by neutral, disgust, surprise, and with the lowest performances seen for fear. Fixation to facial features specific to an emotion did not improve performance and did not account for the differences in accuracy performance between emotions. Rather, results suggest that accuracy performance depends on the integration of facial features, and that this varies across emotions and with presentation time.

The research involves the creation of an interactive installation showcasing the dynamic nature of human visual observation of a still photograph. Using an eye tracker as an input device, the data collected is used to create a photographic collage in stereoscopic 3D. The installation is artistically inspired by selected photographic works of artists David Hockney, Maurizio Galimberti, Joyce Neimanas and Cubist painters especially Picasso. One of the key factors in their work that is adapted in this research, is the representation of the way eyes search points of interest demonstrated in what they painted/photographed. The installation will demonstrate an expressive representation of the viewers' experience of looking at a photograph. This will be achieved by applying certain manipulations of the photograph based on the input obtained from the viewer using an eye tracker. The eye tracker collects information about the location and number of instances of where the viewer is when observing a photograph. This is fed into software that processes the data and determines the location and the size of the area of the photograph and amount of the manipulation to be applied to that area. These two constitute the artistic rules that are used to create the end product the photo collage. The individual pieces of the collage will be arranged in a virtual 3D model by the artist and will be projected in stereoscopic 3D. The development of this installation progressed through multiple case studies and optimization based on ease of use, cost and availability of resources. This process is intended to be a framework for artists working in interactive visual media.

It has been shown in experiments that a method of teaching where students are encouraged to create their own solution methods to mathematical problems (creative mathematically founded reasoning, CMR) results in better learning and proficiency than one where students are provided with solution methods for them to practice by repetition (algorithmic reasoning, AR). The present study investigated whether students in an AR practice condition pay less attention to information relevant for mathematical problem solving than students in a CMR condition. To test this, attentional behavior during practice was measured using eye-tracking equipment. These measurements were then associated with task proficiency in a follow-up test one week after the practice session. The findings support the theory and confirm previous studies in that CMR leads to better task performance in the follow-up test. The findings also suggest that students within the CMR condition whom focus less on extraneous information perform better. / Experiment har visat att en undervisningsmetod i vilken elever uppmuntras att själva komma på lösningsmetoder till matematiska problem (creative mathematically founded reasoning, CMR) resulterar i bättre inlärning och färdighet än en metod i vilken eleverna ges en färdig en lösningsmetod att öva på genom repetition (algorithmic reasoning, AR). Denna studie undersöker om elever under en AR-träningsbetingelse ägnar mindre uppmärksamhet åt information som är relevant för matematisk problemlösning än vad elever under en CMR-träningsbetingelse gör. För att testa detta mättes elevernas uppmärksamhetsbeteende under träning med hjälp av ögonrörelsekamera. Måtten ställdes sedan i relation till uppgiftsfärdighet i ett uppföljningstest en vecka efter träningssessionen. Resultaten stödjer teorin och bekräftar tidigare studier som visat att CMR leder till bättre prestation i uppföljningstestet. Resultaten tyder även på att de elever under CMR-betingelsen som fokuserar minst på ovidkommande information presterar bättre.

In this dissertation, we conducted a stereoscopic eye tracking experiment using naturalistic stereo images. We analyzed low level 2D and 3D scene features at binocular fixations and randomly selected places. The results reveal that humans tend to fixate on regions with higher luminance variations, but lower disparity variations. Because of the often observed co-occurrence of luminance and depth changes in natural environments, the dichotomy between luminance features and disparity features inspired us to study the accurate statistics of 2D and 3D scene properties. Using a range map database, we studied the distribution of disparity in natural scenes. The natural disparity distribution has a high peak at zero, and heavier tails that are similar to a Laplace distribution. The relevance of natural disparity distribution to other studies in neurobiology and visual psychophysics are discussed in detail. We also studied luminance, range and disparity statistics in natural scenes using a co-registered luminance-range database. The distributions of bandpass 2D and 3D scene features can be well modeled by generalized Gaussian models. There are positive correlations between bandpass luminance and depth, which can be captured by varying shape parameters in the probability density functions of the generalized Gaussians. In another study on suprathreshold luminance and depth discontinuities, we show that observing a significant luminance edge at a significant depth edge is much more likely than at homogeneous depth surfaces. It is also true that a significant depth edge happens at a significant luminance edge with a greater probability than at homogeneous luminance regions. Again, the dependency between luminance and depth discontinuities can be modeled successfully by generalized Gaussians. We applied our statistical models in 3D natural scenes to stereo correspondence. A Bayesian framework is proposed to incorporate the bandpass disparity prior, and the luminance-disparity dependency in the likelihood function. We compared our algorithm with a classical simulated annealing method based on heuristically defined energy functions. The computed disparity maps show great improvements both perceptually and objectively. / text