ABSTRACTAASHISH NEUPANE, for the Master of Science degree in BIOMEDICAL ENGINEERING, presented on July 35, 2020, at Southern Illinois University Carbondale. TITLE: VISUAL SALIENCY ANALYSIS ON FASHION IMAGES USING IMAGE PROCESSING AND DEEP LEARNING APPROACHES.MAJOR PROFESSOR: Dr. Jun QinState-of-art computer vision technologies have been applied in fashion in multiple ways, and saliency modeling is one of those applications. In computer vision, a saliency map is a 2D topological map which indicates the probabilistic distribution of visual attention priorities. This study is focusing on analysis of the visual saliency on fashion images using multiple saliency models, evaluated by several evaluation metrics. A human subject study has been conducted to collect people’s visual attention on 75 fashion images. Binary ground-truth fixation maps for these images have been created based on the experimentally collected visual attention data using Gaussian blurring function. Saliency maps for these 75 fashion images were generated using multiple conventional saliency models as well as deep feature-based state-of-art models. DeepFeat has been studied extensively, with 44 sets of saliency maps, exploiting the features extracted from GoogLeNet and ResNet50. Seven other saliency models have also been utilized to predict saliency maps on these images. The results were compared over 5 evaluation metrics – AUC, CC, KL Divergence, NSS and SIM. The performance of all 8 saliency models on prediction of visual attention on fashion images over all five metrics were comparable to the benchmarked scores. Furthermore, the models perform well consistently over multiple evaluation metrics, thus indicating that saliency models could in fact be applied to effectively predict salient regions in random fashion advertisement images.

Since the turn of the millennium, a large number of computational models of visual salience have been put forward. How best to evaluate a given model's ability to predict where human observers fixate in images of real-world scenes remains an open research question. Assessing the role of spatial biases is a challenging issue; this is particularly true when we consider the tendency for high-salience items to appear in the image center, combined with a tendency to look straight ahead (“central bias”). This problem is further exacerbated in the context of model comparisons, because some—but not all—models implicitly or explicitly incorporate a center preference to improve performance. To address this and other issues, we propose to combine a-priori parcellation of scenes with generalized linear mixed models (GLMM), building upon previous work. With this method, we can explicitly model the central bias of fixation by including a central-bias predictor in the GLMM. A second predictor captures how well the saliency model predicts human fixations, above and beyond the central bias. By-subject and by-item random effects account for individual differences and differences across scene items, respectively. Moreover, we can directly assess whether a given saliency model performs significantly better than others. In this article, we describe the data processing steps required by our analysis approach. In addition, we demonstrate the GLMM analyses by evaluating the performance of different saliency models on a new eye-tracking corpus. To facilitate the application of our method, we make the open-source Python toolbox “GridFix” available.

Cette thèse est motivée par le diagnostic et l’évaluation des maladies neuro-dégénératives et dans le but de diagnostique sur la base de l’attention visuelle.Néanmoins, le dépistage à grande échelle de la population n’est possible que si des modèles de prédiction automatique suffisamment robustes peuvent être construits. Dans ce contexte nous nous intéressons `a la conception et le développement des modèles de prédiction automatique pour un contenu visuel spécifique à utiliser dans l’expérience psycho-visuelle impliquant des patients atteints des maladies neuro-dégénératives. La difficulté d’une telle prédiction réside dans une très faible quantité de données d’entraînement. Les modèles de saillance visuelle ne peuvent pas être fondés sur les caractérisitiques “bottom-up” uniquement, comme le suggère la théorie de l’intégration des caractéristiques. La composante “top-down” de l’attention visuelle humaine devient prépondérante au fur et à mesure d’observation de la scène visuelle. L’attention visuelle peut-être prédite en se basant sur les scènes déjà observées. Les réseaux de convolution profonds (CNN) se sont révèlés être un outil puissant pour prédire les zones saillantes dans les images statiques.Dans le but de construire un modèle de prédiction automatique pour les zones saillantes dans les vidéos naturels et intentionnellement dégradées, nous avons conçu une architecture spécifique de CNN profond. Pour surmonter le manque de données d’apprentissage,nous avons conçu un système d’apprentissage par transfert dérivé de la méthode de Bengio.Nous mesurons ses performances lors de la prédiction de régions saillantes. Les r´esultatsobtenus sont int´eressants concernant la r´eaction des sujets t´emoins normaux contre leszones d´egrad´ees dans les vid´eos. La comparaison de la carte de saillance pr´edite des vid´eosintentionnellement d´egrad´ees avec des cartes de densit´e de fixation du regard et d’autresmod`eles de r´ef´erence montre l’int´erˆet du mod`ele d´evelopp´e. / This thesis is motivated by the diagnosis and the evaluation of the dementia diseasesand with the aim of predicting if a new recorded gaze presents a complaint of thesediseases. Nevertheless, large-scale population screening is only possible if robust predictionmodels can be constructed. In this context, we are interested in the design and thedevelopment of automatic prediction models for specific visual content to be used in thepsycho-visual experience involving patients with dementia (PwD). The difficulty of sucha prediction lies in a very small amount of training data.Visual saliency models cannot be founded only on bottom-up features, as suggested byfeature integration theory. The top-down component of human visual attention becomesprevalent as human observers explore the visual scene. Visual saliency can be predictedon the basis of seen data. Deep Convolutional Neural Networks (CNN) have proven tobe a powerful tool for prediction of salient areas in static images. In order to constructan automatic prediction model for the salient areas in natural and intentionally degradedvideos, we have designed a specific CNN architecture. To overcome the lack of learningdata we designed a transfer learning scheme derived from bengio’s method. We measureits performances when predicting salient regions. The obtained results are interestingregarding the reaction of normal control subjects against degraded areas in videos. Thepredicted saliency map of intentionally degraded videos gives an interesting results comparedto gaze fixation density maps and other reference models.

Since the turn of the millennium, a large number of computational models of visual salience have been put forward. How best to evaluate a given model's ability to predict where human observers fixate in images of real-world scenes remains an open research question. Assessing the role of spatial biases is a challenging issue; this is particularly true when we consider the tendency for high-salience items to appear in the image center, combined with a tendency to look straight ahead (“central bias”). This problem is further exacerbated in the context of model comparisons, because some—but not all—models implicitly or explicitly incorporate a center preference to improve performance. To address this and other issues, we propose to combine a-priori parcellation of scenes with generalized linear mixed models (GLMM), building upon previous work. With this method, we can explicitly model the central bias of fixation by including a central-bias predictor in the GLMM. A second predictor captures how well the saliency model predicts human fixations, above and beyond the central bias. By-subject and by-item random effects account for individual differences and differences across scene items, respectively. Moreover, we can directly assess whether a given saliency model performs significantly better than others. In this article, we describe the data processing steps required by our analysis approach. In addition, we demonstrate the GLMM analyses by evaluating the performance of different saliency models on a new eye-tracking corpus. To facilitate the application of our method, we make the open-source Python toolbox “GridFix” available.