Abstract Humans are capable of narrowing their focus on the highlights of visual information in a fraction of time in order to handle enormous mass of data. Akin to human, computers should deal with a tremendous amount of visual information. To replicate such a focusing mechanism, computer vision relies on techniques that filter out redundant information. Consequently, saliency has recently been a popular subject of discussion in the computer vision community, though it is an old subject matter in the disciplines of cognitive sciences rather than computer science. The reputation of saliency techniques – particularly in the computer vision domain – is greatly due to their inexpensive and fast computation which facilitates their use in many computer vision applications, e.g., image/video compression, object recognition, tracking, etc. This study investigates visual saliency modeling, which is the transformation of an image into a salience map such that the identified conspicuousness agrees with the statistics of human eye movements. It explores the extent of image and video processing to develop saliency techniques suitable for computer vision, e.g., it adopts sparse sampling scheme and kernel density estimation to introduce a saliency measure for images. Also, it studies the role of eye movement in salience modeling. To this end, it introduces a particle filter based framework of saccade generation incorporated into a salience model. Moreover, eye movements and salience are exploited in several applications. The contributions of this study lie on the proposal of a number of salience models for image and video stimuli, a framework to incorporate a model of eye movement generation in salience modeling, and the investigation of the application of salience models and eye movements in tracking, background subtraction, scene recognition, and valence recognition. / Tiivistelmä Ihmiset kykenevät kohdistamaan katseensa hetkessä näkymän keskeisiin asioihin, mikä vaatii näköjärjestelmältä valtavan suurten tietomäärien käsittelyä. Kuten ihmisen myös tietokoneen pitäisi pystyä käsittelemään vastaavasti suurta määrää visuaalista informaatiota. Tällaisen mekanismin toteuttaminen tietokonenäöllä edellyttää menetelmiä, joilla redundanttista tietoa voidaan suodattaa. Tämän vuoksi salienssista eli silmiinpistävyydestä on muodostunut viime aikoina suosittu tutkimusaihe tietotekniikassa ja erityisesti tietokonenäön tutkimusyhteisössä, vaikka sitä sinänsä on jo pitkään tutkittu kognitiivisissa tieteissä. Salienssimenetelmien tunnettavuus erityisesti tietokonenäössä johtuu pääasiassa niiden laskennallisesta tehokkuudesta, mikä taas mahdollistaa menetelmien käytön monissa tietokonenäön sovelluksissa kuten kuvan ja videon pakkaamisessa, objektin tunnistuksessa, seurannassa, etc. Tässä väitöskirjassa tutkitaan visuaalisen salienssin mallintamista, millä tarkoitetaan muunnosta kuvasta salienssikartaksi siten, että laskennallinen silmiinpistävyys vastaa ihmisen silmänliikkeistä muodostettavaa statistiikkaa. Työssä tarkastellaan keinoja, miten kuvan- ja videonkäsittelyä voidaan käyttää kehittämään salienssimenetelmiä tietokonenäön tarpeisiin. Työssä esitellään esimerkiksi harvaa näytteistystä ja ydinestimointia hyödyntävä kuvien salienssimitta. Työssä tutkitaan myös silmänliikkeiden merkitystä salienssin mallintamisen kannalta. Tätä varten esitellään partikkelisuodatusta hyödyntävä lähestymistapa sakkadien generointiin, joka voidaan liittää salienssimalliin. Lisäksi silmänliikkeitä ja salienssia hyödynnetään useissa sovelluksissa. Suoritetun tutkimuksen tieteellisiin kontribuutioihin sisältyvät useat esitetyt salienssimallit kuvasta ja videosta saatavalle herätteelle, lähestymistapa silmänliikkeiden laskennalliseen mallintamiseen ja generointiin osana salienssimallia sekä salienssimallien ja silmänliikkeiden sovellettavuuden tutkiminen visuaalisessa seurannassa, taustanvähennyksessä, näkymäanalyysissa ja valenssin tunnistuksessa.

Purpose - The purpose of this research was to study millennial consumers' attention to online advertisements published by streetwear companies on social media. The aim was to investigate differences between consumers' attention to familiar products and unfamiliar products. Three hypotheses were constructed to test whether product familiarity affects consumers´ attention to advertisements: H1: Consumers who are familiar with the products are more likely to look at them in advertisements than consumers who are unfamiliar with the products. H2: Consumers who are familiar with the products will spend more time looking at them in advertisements than consumers who are unfamiliar with the products. H3: Consumers who are familiar with the products will find them faster from advertisements than consumers who are unfamiliar with the products. Method - The study applied between-subjects design by dividing 40 participants in two matched groups based on their gender. Both groups had 10 male and 10 female participants. During the eye-tracking experiment, participants in Group 1 were shown three product photos for two seconds and three advertisements containing the same products for four seconds. Participants in Group 2 were only presented with three advertisements for four seconds. The results of the study are presented in two parts. Firstly, heatmaps give an illustrative overview of the findings and Statistical analysis presents the results of statistical tests. Chi-square test was performed to test the first hypothesis and independent-samples t-tests were performed to test the second and third hypotheses. Results - The results of the study show that people who were familiar with the products were more likely to look at them in advertisements than consumers who were unfamiliar with the products. Also, people who were familiar with the products spent more time looking at them compared to people who were unfamiliar with the products. However, previous exposure to products did not result in faster identification of the products from advertisements.

京都大学 / 新制・課程博士 / 博士(情報学) / 甲第23398号 / 情博第767号 / 新制||情||131(附属図書館) / 京都大学大学院情報学研究科知能情報学専攻 / (主査)教授 熊田 孝恒, 教授 西田 眞也, 教授 齋木 潤, 准教授 延原 章平 / 学位規則第4条第1項該当 / Doctor of Informatics / Kyoto University / DFAM

This bachelor thesis researches the question of “How do Swedish interior design companies tailor their e-commerce marketing strategies to cope with the phenomenon of advertising blindness?” through a qualitative case study. In order to conduct this research, relevant aspects of the phenomenon have been analysed in relation to actual marketing strategies. By following the qualitative methods with a semi-structured interview and content analysis, a deeper understanding of the phenomenon and how Swedish interior design companies cope with it can be gathered. The thesis follows a deductive approach, due to that the research topic in the specified industry was considered rather unexplored, thus a better preunderstanding of the topic was deemed relevant. The existing knowledge and research have been compiled and resulted in a conceptual framework. The pre-existing studies on advertising blindness have been conducted in other industries; this thesis will thus be founding its analysis on previous studies but in a new industry with different types of consumers and marketing. The analysis of the empirical findings encompassed a discussion about how theoretical models and practical marketing methods are used by a Swedish interior design company. The conclusions drawn from the analysis present both implications, recommendations, and limitations. The empirical findings showed that advertising blindness is not a phenomenon actively planned against in the interior design case company that was researched for this thesis. The main theoretical implications derived from this thesis, was giving marketing students and marketing managers insight into the importance of understanding the advertising blindness phenomenon. The main practical implication was found to be that although certain KPI are used to monitor consumer behaviour and purchases, advertising blindness has not been prioritized in the measurements. Thus, as omni channels such as the case company have to increase their focus on e-commerce, the attention to advertising blindness has to be more present from managers. Furthermore, a continuation of this research would find value in conducting research from the consumer perspective in order to give marketing managers a second perspective that could show the importance of studying advertising blindness.

El texto completo de este trabajo no está disponible en el Repositorio Académico UPC por restricciones de la casa editorial donde ha sido publicado. / This paper seeks to study the correspondence between the results of the test of variable of attention (TOVA) and the signals acquired by the Muse electroencephalogram (EEG) in the positions AF7 and AF8 of the cerebral cortex. There are a variety of research papers that estimates an index of attention in which the different characteristics in discrete signals of the brain activity were used. However, many of these results were obtained without contrasting them with standardized tests. Due to this fact, in the present work, the results will be compared with the score of the TOVA, which aims to identify an attention disorder in a person. The indicators obtained from the test are the response time variability, the average response time, and the d′ prime score. During the test, the characteristics of the EEG signals in the alpha, beta, theta, and gamma subbands such as the energy, average power, and standard deviation were extracted. For this purpose, the acquired signals are filtered to reduce the effect of the movement of the muscles near the cerebral cortex and then went through a subband decomposition process by applying transformed wavelet packets. The results show a well-marked correspondence between the parameters of the EEG signal of the indicated subbands and the visual attention indicators provided by TOVA. This correspondence was measured through Pearson’s correlation coefficient which had an average result of 0.8. / Revisión por pares / Revisión por pares

Studying the field of visual attention in the context of games can help level designers to make the players feel more immersed and increase the general enjoyment of a game. With the help of visual cues, the designers are able to lead the players through the levels without breaking the aesthetics of the game, thus preventing to break the immersion. This study is presenting a method for analyzing visual cues in a 3D adventure platform game. Gathering data with the help of today's eye-tracking technology. The results can ultimately be used as a guideline for future work in the field.

The real-world sensorimotor paradigm is based on the premise that sufficient ecological complexity is a prerequisite for inducing naturally relevant sensorimotor relations in the experimental context. The aim of this thesis is to embed visual attention research within the real-world sensorimotor paradigm using an innovative mobile gaze-tracking system (EyeSeeCam, Schneider et al., 2009). Common laboratory set-ups in the field of attention research fail to create natural two-way interaction between observer and situation because they deliver pre-selected stimuli and human observer is essentially neutral or passive. EyeSeeCam, by contrast, permits an experimental design whereby the observer freely and spontaneously engages in real-world situations. By aligning a video camera in real time to the movements of the eyes, the system directly measures the observer’s perspective in a video recording and thus allows us to study vision in the context of authentic human behaviour, namely as resulting from past actions and as originating future actions. The results of this thesis demonstrate that (1) humans, when freely exploring natural environments, prefer directing their attention to local structural features of the world, (2) eyes, head and body perform distinct functions throughout this process, and (3) coordinated eye and head movements do not fully stabilize but rather continuously adjust the retinal image also during periods of quasi-stable “fixation”. These findings validate and extend the common laboratory concept of feature salience within whole-body sensorimotor actions outside the laboratory. Head and body movements roughly orient gaze, potentially driven by early stages of processing. The eyes then fine-tune the direction of gaze, potentially during higher-level stages of visual-spatial behaviour (Studies 1 and 2). Additional head-centred recordings reveal distinctive spatial biases both in the visual stimulation and the spatial allocation of gaze generated in a particular real-world situation. These spatial structures may result both from the environment and form the idiosyncrasies of the natural behaviour afforded by the situation. By contrast, when the head-centred videos are re-played as stimuli in the laboratory, gaze directions reveal a bias towards the centre of the screen. This “central bias” is likely a consequence of the laboratory set-up with its limitation to eye-in-head movements and its restricted screen (Study 3). Temporal analysis of natural visual behaviour reveals frequent synergistic interactions of eye and head that direct rather than stabilize gaze in the quasi-stable eye movement periods following saccades, leading to rich temporal dynamics of real-world retinal input (Study 4) typically not addressed in laboratory studies. Direct comparison to earlier data with respect to the visual system of cats (CatCam), frequently taken as proxy for human vision, shows that stabilizing eye movements play an even less dominant role in the natural behaviour of cats. This highlights the importance of realistic temporal dynamics of vision for models and experiments (Study 5). The approach and findings presented in this thesis demonstrate the need for and feasibility of real- world research on visual attention. Real-world paradigms permit the identification of relevant features triggered in the natural interplay between internal-physiological and external-situational sensorimotor factors. Realistic spatial and temporal characteristics of eye, head and body interactions are essential qualitative properties of reliable sensorimotor models of attention but difficult to obtain under laboratory conditions. Taken together, the data and theory presented in this thesis suggest that visual attention does not represent a pre-processing stage of object recognition but rather is an integral component of embodied action in the real world.

This thesis deals with biologically-inspired interactive neural networks for the task of object recognition. Such networks offer an interesting alternative approach to traditional image processing techniques. Although the networks are very powerful classification tools, they are difficult to handle due to their bidirectional interactivity. It is one of the main reasons why these networks do not perform the task of generalization to novel objects well. Generalization is a very important property for any object recognition system, as it is impractical for a system to learn all instances of an object class before classifying. In this thesis, we have investigated the working of an interactive neural network by fine tuning different structural and algorithmic parameters.  The performance of the networks was evaluated by analyzing the generalization ability of the trained network to novel objects. Furthermore, the interactivity of the network was utilized to simulate focus of attention during object classification. Selective attention is an important visual mechanism for object recognition and provides an efficient way of using the limited computational resources of the human visual system. Unlike most previous work in the field of image processing, in this thesis attention is considered as an integral part of object processing. Attention focus, in this work, is computed within the same network and in parallel with object recognition. As a first step, a study into the efficacy of Hebbian learning as a feature extraction method was conducted. In a second study, the receptive field size in the network, which controls the size of the extracted features as well as the number of layers in the network, was varied and analyzed to find its effect on generalization. In a continuation study, a comparison was made between learnt (Hebbian learning) and hard coded feature detectors. In the last study, attention focus was computed using interaction between bottom-up and top-down activation flow with the aim to handle multiple objects in the visual scene. On the basis of the results and analysis of our simulations we have found that the generalization performance of the bidirectional hierarchical network improves with the addition of a small amount of Hebbian learning to an otherwise error-driven learning. We also conclude that the optimal size of the receptive fields in our network depends on the object of interest in the image. Moreover, each receptive field must contain some part of the object in the input image. We have also found that networks using hard coded feature extraction perform better than the networks that use Hebbian learning for developing feature detectors. In the last study, we have successfully demonstrated the emergence of visual attention within an interactive network that handles more than one object in the input field. Our simulations demonstrate how bidirectional interactivity directs attention focus towards the required object by using both bottom-up and top-down effects. In general, the findings of this thesis will increase understanding about the working of biologically-inspired interactive networks. Specifically, the studied effects of the structural and algorithmic parameters that are critical for the generalization property will help develop these and similar networks and lead to improved performance on object recognition tasks. The results from the attention simulations can be used to increase the ability of networks to deal with multiple objects in an efficient and effective manner.

Le système visuel a évolué de manière à prendre en compte les conséquences de nos mouvements sur notre perception. L’évolution nous a particulièrement doté de la capacité à percevoir notre environnement visuel comme stable et continu malgré les importants déplacements de ses projections sur nos rétines à chaque fois que nous déplaçons nos yeux, notre tête ou notre corps. Des études chez l’animal ont récemment montré que dans certaines aires corticales et sous-corticales, impliquées dans le contrôle attentionnel et dans l’élaboration des mouvements oculaires, des neurones sont capables d’anticiper les conséquences des futurs mouvements volontaires des yeux sur leurs entrées visuelles. Ces neurones prédisent ce à quoi ressemblera notre environnement visuel en re-cartographiant la position des objets d’importance à l’endroit qu’ils occuperont après l’exécution d’une saccade. Dans une série d’études, nous avons tout d’abord démontré que cette re- cartographie pouvait être évaluée de manière non invasive chez l’Homme avec de simples cibles en mouvement apparent. En utilisant l’enregistrement des mouvements des yeux combinés à des méthodes psychophysiques, nous avons déterminé la distribution des erreurs de re-cartographie à travers le champ visuel et ainsi découvert que la compensation des saccades oculomotrices se faisait de manière relativement précise. D’autre part, les patterns d’erreurs observés soutiennent un modèle de la constance spatiale basé sur la re-cartographie de pointeurs attentionnels et excluent d’autres modèles issus de la littérature. Par la suite, en utilisant des objets en mouvement continu et l’exécution de saccades au travers de leurs trajectoires, nous avons mis à jour une visualisation directe des processus de re-cartographie. Avec ce nouveau procédé nous avons à nouveau démontré l’existence d’erreurs systématiques de correction pour les saccades, qui s’expliquent par une re-cartographie imprécise de la position attendue des objets en mouvement. Nous avons par la suite étendu notre modèle à d’autres types de mouvements du corps et notamment étudié les contributions de récepteurs sous-corticaux (otoliths et canaux semi-circulaires) dans le maintien de la constance spatiale à travers des mouvements de la tête. Contrairement à des études décrivant une compensation presque parfaite des mouvements de la tête, nous avons observé une rupture de la constance spatiale pour des mouvements de roulis et de translation de la tête. Enfin, nous avons testé cette re-cartographie de la position des objets compensant un déplacement oculaire avec des cibles présentées à la limite du champ visuel, une re-cartographie censée placer la position attendue de l’objet à l’extérieur du champ visuel. Nos résultats suggèrent que les aires visuelles cérébrales impliquées dans ce processus de re-cartographie construisent une représentation globale de l’espace allant au-delà du traditionnel champ visuel. Pour finir, nous avons conduit deux expériences pour déterminer le déploiement de l’attention à travers l’exécution de saccades. Nous avons alors démontré que l’attention capturée par la présentation brève d’un stimuli est re-cartographiée à sa position spatiale correcte après l’exécution d’une saccade, et que cet effet peut être observé avant même l’initiation d’une saccade. L’ensemble de ces résultats démontre le rôle des pointeurs attentionnels dans la gestion du rétablissement des positions d’un objet dans l’espace ainsi que l’apport des mesures comportementales à un champ de recherche initialement restreint à l’électrophysiologie / The visual system has evolved to deal with the consequences of our own movements onour perception. In particular, evolution has given us the ability to perceive our visual world as stableand continuous despite large shift of the image on our retinas when we move our eyes, head orbody. Animal studies have recently shown that in some cortical and sub-cortical areas involved inattention and saccade control, neurons are able to anticipate the consequences of voluntary eyemovements on their visual input. These neurons predict how the world will look like after a saccadeby remapping the location of each attended object to the place it will occupy following a saccade.In a series of studies, we first showed that remapping could be evaluated in a non-invasive fashion in human with simple apparent motion targets. Using eye movement recordingsand psychophysical methods, we evaluated the distribution of remapping errors across the visualfield and found that saccade compensation was fairly accurate. The pattern of errors observedsupport a model of space constancy based on a remapping of attention pointers and excluded otherknown models. Then using targets that moved continuously while a saccade was made across themotion path, we were able to directly visualize the remapping processes. With this novel method wedemonstrated again the existence of systematic errors of correction for the saccade, best explainedby an inaccurate remapping of expected moving target locations. We then extended our model toother body movements, and studied the contribution of sub-cortical receptors (otoliths and semi-circular canals) in the maintenance of space constancy across head movements. Contrary tostudies reporting almost perfect compensations for head movements, we observed breakdowns ofspace constancy for head tilt as well as for head translation. Then, we tested remapping of targetlocations to correct for saccades at the very edge of the visual field, remapping that would place theexpected target location outside the visual field. Our results suggest that visual areas involved inremapping construct a global representation of space extending out beyond the traditional visualfield. Finally, we conducted experiments to determine the allocation of attention across saccades.We demonstrated that the attention captured by a brief transient was remapped to the correctspatial location after the eye movement and that this shift can be observed even before thesaccade.Taken together these results demonstrate the management of attention pointers to therecovery of target locations in space as well as the ability of behavioral measurements to address atopic pioneered by eletrophysiologists.

Quand un stimulus visuel de haut contraste précède et puis succède à la présentation d'une cible de faible contraste à la même position, la cible peut devenir non-reconnaissable ou même non-détectable. Cet effet de masquage a été étudié en profondeur et beaucoup de ses paramètres sont bien définis. Mais en utilisant une nouvelle méthode avec une séquence de masques et de cibles en mouvement, il est maintenant possible de séparer de manière perceptuelle et attentionnelle la cible des masques tout en conservant l'ordre d'apparition des masques et de la cible à chaque endroit. Cette thèse utilise cette technique de stimuli en mouvement pour répondre à trois questions qui ne pourraient pas être résolu en utilisant une méthode de masquage conventionnelle. Dans la première série d'expériences on a testé si l'attention portée au masque influence l'impact de celle-ci sur la cible. Nous avons utilisé un écran où une séquence masque-cible-masque apparaît à une position puis à chaque 'frame' suivant elle apparaît à la position adjacente. Cette méthode permet à l'observateur de suivre attentionellement la cible sur son trajet, tout en évitant les masques distracteurs qui paraissent être à côté de la cible spatialement et non temporellement. Quand les masques sont efficaces, la position de la cible paraît être vide. Nous comparons l'efficacité du masquage quand l'attention n'est pas dirigée vers les masques, à l'efficacité du masquage quand l'attention ne peut pas éviter les masques, comme dans le masquage conventionnel. Nous n'avons trouvé aucune différence significative entre les deux conditions attentionnelles, ce qui indique que l'attention portée sur les masques ne module pas leur efficacité. Ensuite nous avons testé l'importance de l'espacement dans la séquence masque-cible-masque. Il n'y avait pas plus de masquage quand les espacements étaient réduits, ce qui suggère que le masquage n'est pas le résultat de masquage latéral ou de 'crowding' (encombrement spatial) aux distances testées. Enfin nous avons comparé la contribution de chaque masque: celui qui précède la cible (le pré-masque) et celui qui suit la cible (post-masque). Nous avons trouvé que le pré-masque génère la majorité de l'effet de masquage tandis que le post-masque a peu d'influence. Dans la seconde série d'expériences nous avons étudié ce qu'il advient du stimulus masqué. En baissant le contraste de la cible juste en dessous de son seuil de détection et en lui donnant une couleur saillante, nous avons découvert un stade intermédiaire où les caractéristiques ('features') de la cible sont reportées à la position d'un des masques. Avec une série d'expérience nous avons montré que la cible est intégrée avec le masque qui la suit directement et temporellement à la même position. Finalement nous avons étudié si un stimulus qui est masqué à un niveau invisible peut quand même exercer une influence sur une cible proche dans un arrangement de crowding. Nous avons comparé la détection d'une cible avec deux flankers (distracteurs adjacent) dans la zone de crowding, à la détection d'une cible quand les deux flankers sont masqué et que leur position paraît être vide. Pour conclure, la méthode de séquence en mouvement a révélé de nouvelles caractéristiques du masquage qui ne pouvaient pas être examinées en utilisant des techniques de masquage conventionnelles. Premièrement nous avons découvert que l'efficacité du masque est comparable que l'attention soit dirigée sur le masque ou non. Deuxièmement nous avons montré que les caractéristiques d'une cible qui est supprimée à sa position physique peuvent apparaître à la position du masque qui la suit temporellement. Et finalement, nous avons examiné l'effet de flankers masqués sur une cible dans un arrangement de crowding. / When high contrast visual stimuli precede and follow a low contrast target at the same location, the target may become unrecognizable and even undetectable. This masking effect has been extensively studied and many of its parameters are well characterized. However, using a new technique with a moving sequence of masks and targets it is now possible to perceptually and attentionally separate the target from the masks while retaining the same temporal order of masks and target at each location. This thesis exploits this moving stimulus technique to ask three questions that could not be addressed in standard masking paradigms. In the first series of experiments we looked at whether attention to the mask affected its impact on the target. We used a display where the mask-target-mask sequence appears at one location and on each subsequent frame it appears in the adjacent location. This allows observers to attentively track the target across locations without attending to the masks which appear perceptually adjacent in space rather than in time. When the masks are effective, the observer sees a blank space at the target location. We compare the effectiveness of this masking when the masks are unattended to the effectiveness when the masks are attended as in standard masking. We found no significant difference between the two attentional conditions suggesting that attention to the masks does not modulate their effectiveness. We next examined the importance of the spacing within the mask-target-mask sequences. There was no increase in the masking effect as spacing was reduced, suggesting that the masking was not a consequence of lateral masking or crowding at the spacings we tested. Finally, we compared the contributions of each mask individually: the one that preceded the target (pre-mask) and the one that followed (post-mask). We found that the pre-mask generated the majority of the masking effect while the post-mask was of little influence. In the second series of studies we investigated the fate of a masked stimulus. By lowering the target contrast just below its detection threshold and giving it a salient color, we found an intermediate, "lurking" stage where the target's features migrate to one of the mask locations. Through a succession of experiments we found that the target is integrated with the mask directly following it in time at the same location. Finally we looked at whether a stimulus that is masked to invisibility can still exert influence over nearby targets in a crowding array. We compared the detection of a crowded target with two flankers, compared to the detection of a target when the two adjacent flankers are masked so that their locations appear empty. In conclusion, the moving sequence technique revealed new characteristics of masking that could not be examined in standard masking paradigms. First, we found that the effectiveness of a mask was the same whether it is attended or not. Second, we showed that once a masked target has been suppressed from its physical location, its features can be found "lurking" at the location of the mask that follows the target in time. Finally, we examined the effect of masked flankers on a target in a crowding paradigm.