This dissertation proposes a new methodology for building and using structured knowledge models for automatic image annotation. Specifically, our first proposals deal with the automatic building of explicit and structured knowledge models, such as semantic hierarchies and multimedia ontologies, dedicated to image annotation. Thereby, we propose a new approach for building semantic hierarchies faithful to image semantics. Our approach is based on a new image-semantic similarity measure between concepts and on a set of rules that allow connecting the concepts with higher relatedness till the building of the final hierarchy. Afterwards, we propose to go further in the modeling of image semantics through the building of explicit knowledge models that incorporate richer semantic relationships between image concepts. Therefore, we propose a new approach for automatically building multimedia ontologies consisting of subsumption relationships between concepts, and also other semantic relationships such as contextual and spatial relations. Fuzzy description logics are used as a formalism to represent our ontology and to deal with the uncertainty and the imprecision of concept relationships. In order to assess the effectiveness of the built structured knowledge models, we propose subsequently to use them in a framework for image annotation. We propose therefore an approach, based on the structure of semantic hierarchies, to effectively perform hierarchical image classification. Furthermore, we propose a generic approach for image annotation combining machine learning techniques, such as hierarchical image classification, and fuzzy ontological-reasoning in order to achieve a semantically relevant image annotation. Empirical evaluations of our approaches have shown significant improvement in the image annotation accuracy.

Nos domínios visuais, interpretações são iniciadas pela identificação de feições em imagens que irão, posteriormente, dar suporte aos processos mais abstratos de inferência. Para desenvolver sistemas de conhecimento neste tipo de domínio é necessário buscar a melhor representação do conhecimento visual para ser utilizado pelos métodos de inferência. A representação em formato simbólico deste conhecimento auxilia na captura do conhecimento implícito presente em imagens, permitindo seu uso nos processos de raciocínio, mesmo aceitando que parte desse conhecimento não é externalizado e, em conseqüência, não poderá ser representado adequadamente. Estudos recentes têm utilizado anotação de imagens como uma maneira capaz de auxiliar na explicitação do conhecimento, ampliando a expressividade dos formalismos de representação e permitindo o registro das informações associadas às imagens. Embora anotações de imagens flexibilizem a captura do conhecimento, ontologias são associadas às anotações para garantir a formalização do conhecimento nas imagens, suprindo os termos de domínio que podem ser usados para anotar e auxiliar a uniformização da linguagem nas consultas. O objetivo desse trabalho é capturar e documentar o conhecimento visual que dá suporte à inferência nas tarefas de interpretações. Nesse trabalho é elaborada uma maneira de identificar objetos em imagens que contenham feições diagnósticas através da utilização de uma ontologia de domínio pré-existente. Essa identificação de objetos é explorada para permitir a localização física de uma determinada feição em um objeto real. O resultado disso é a identificação de feições em uma imagem tendo-se um referencial de posição segundo um sistema de coordenadas espacial, possibilitando o seu re-posicionamento. O trabalho ainda explora recursos para a melhor utilização de informações relacionadas a uma imagem. Dessa forma, serve de documentação auxiliar para prover diferencial em interpretações. O domínio de aplicação deste trabalho é a Petrografia Sedimentar que tem como uma das tarefas a descrição quantitativa e qualitativa dos minerais que compõem uma amostra de rocha, visando a análise do potencial de um reservatório de petróleo. As informações são visualizadas em uma amostra de rocha e, assim, um especialista toma decisões quanto à viabilidade de exploração do reservatório. Esse modelo foi validado através da implementação de um módulo de anotação de imagem e de referenciamento de objetos, juntamente com um hardware. O processo de descrição de amostra de rocha é detalhado para se explorar o conhecimento em imagens com a precisão da localização dos objetos nela identificados. Essa abordagem complementa os modelos de representação simbólica, que normalmente são insuficientes para capturar a semântica das feições que dão suporte a inferência em domínios visuais. / Problem solvers in visual domains apply image interpretation to extract evidences to support the reasoning process in the search of solution. In order to develop knowledge systems in this kind of domain, it is necessary to represent the knowledge that is extracted from the scenes in the domain in such way that can be used by the inference methods. The symbolic representation formalisms help in shading light on the implicit knowledge embodied in images, allowing its use in reasoning processes, even accepting that part of this knowledge can not be properly expressed by the experts and, in consequence, it is never going to be adequately represented in knowledge models. Some studies use image annotation to extend the semantic capture of the visual knowledge beyond the expressiveness of representation formalisms, allowing that additional content could be described and stored with the images. Although the annotation is a more flexible way of capturing knowledge, ontologies are used along with it to guarantee the formalization of the knowledge annotated in the images, supplying the domain terms that can be used in the annotation process and helping the uniformization of the language and further consultation. The goal of this work is capturing and documenting the visual knowledge that supports inference on interpretation tasks. The work offer a way of identifying objects in images that correspond to diagnostic features and describing them with the support of a pre-existent domain ontology. The object identification is associated to the physical location of the feature in the real object, with the help of a hardware appliance. The result is the feature identification in the image with the reference of its position in a system of spatial coordinates. The work also explores multimedia resources for the best use of information associated to the image, providing an extra-support for knowledge documentation for interpretation tasks. The application domain of this work is the sedimentary petrography, the formal description of rocks in the scope of Petrological studies. The task in focus is the compositional description of minerals in a rock sample, both in qualitative and quantitative way, in order to evaluate the potential of a petroleum reservoir. Several aspects of the rock are studied by the geologist to suggest better techniques for exploration and production of oil fields. The approach defined in this project was validated through the implementation of an image annotation and referencing modules associated to microscope stage appliance that allows the computer to control the movement of the stage during the description process. The whole process of rock sample analysis using the software and the hardware is detailed in order to expose the visual knowledge and its physical position in the rock sample. This approach complements the symbolic representation models that are usually insufficient to capture the semantic of diagnostic features that support reasoning in visual domains.

Nos domínios visuais, interpretações são iniciadas pela identificação de feições em imagens que irão, posteriormente, dar suporte aos processos mais abstratos de inferência. Para desenvolver sistemas de conhecimento neste tipo de domínio é necessário buscar a melhor representação do conhecimento visual para ser utilizado pelos métodos de inferência. A representação em formato simbólico deste conhecimento auxilia na captura do conhecimento implícito presente em imagens, permitindo seu uso nos processos de raciocínio, mesmo aceitando que parte desse conhecimento não é externalizado e, em conseqüência, não poderá ser representado adequadamente. Estudos recentes têm utilizado anotação de imagens como uma maneira capaz de auxiliar na explicitação do conhecimento, ampliando a expressividade dos formalismos de representação e permitindo o registro das informações associadas às imagens. Embora anotações de imagens flexibilizem a captura do conhecimento, ontologias são associadas às anotações para garantir a formalização do conhecimento nas imagens, suprindo os termos de domínio que podem ser usados para anotar e auxiliar a uniformização da linguagem nas consultas. O objetivo desse trabalho é capturar e documentar o conhecimento visual que dá suporte à inferência nas tarefas de interpretações. Nesse trabalho é elaborada uma maneira de identificar objetos em imagens que contenham feições diagnósticas através da utilização de uma ontologia de domínio pré-existente. Essa identificação de objetos é explorada para permitir a localização física de uma determinada feição em um objeto real. O resultado disso é a identificação de feições em uma imagem tendo-se um referencial de posição segundo um sistema de coordenadas espacial, possibilitando o seu re-posicionamento. O trabalho ainda explora recursos para a melhor utilização de informações relacionadas a uma imagem. Dessa forma, serve de documentação auxiliar para prover diferencial em interpretações. O domínio de aplicação deste trabalho é a Petrografia Sedimentar que tem como uma das tarefas a descrição quantitativa e qualitativa dos minerais que compõem uma amostra de rocha, visando a análise do potencial de um reservatório de petróleo. As informações são visualizadas em uma amostra de rocha e, assim, um especialista toma decisões quanto à viabilidade de exploração do reservatório. Esse modelo foi validado através da implementação de um módulo de anotação de imagem e de referenciamento de objetos, juntamente com um hardware. O processo de descrição de amostra de rocha é detalhado para se explorar o conhecimento em imagens com a precisão da localização dos objetos nela identificados. Essa abordagem complementa os modelos de representação simbólica, que normalmente são insuficientes para capturar a semântica das feições que dão suporte a inferência em domínios visuais. / Problem solvers in visual domains apply image interpretation to extract evidences to support the reasoning process in the search of solution. In order to develop knowledge systems in this kind of domain, it is necessary to represent the knowledge that is extracted from the scenes in the domain in such way that can be used by the inference methods. The symbolic representation formalisms help in shading light on the implicit knowledge embodied in images, allowing its use in reasoning processes, even accepting that part of this knowledge can not be properly expressed by the experts and, in consequence, it is never going to be adequately represented in knowledge models. Some studies use image annotation to extend the semantic capture of the visual knowledge beyond the expressiveness of representation formalisms, allowing that additional content could be described and stored with the images. Although the annotation is a more flexible way of capturing knowledge, ontologies are used along with it to guarantee the formalization of the knowledge annotated in the images, supplying the domain terms that can be used in the annotation process and helping the uniformization of the language and further consultation. The goal of this work is capturing and documenting the visual knowledge that supports inference on interpretation tasks. The work offer a way of identifying objects in images that correspond to diagnostic features and describing them with the support of a pre-existent domain ontology. The object identification is associated to the physical location of the feature in the real object, with the help of a hardware appliance. The result is the feature identification in the image with the reference of its position in a system of spatial coordinates. The work also explores multimedia resources for the best use of information associated to the image, providing an extra-support for knowledge documentation for interpretation tasks. The application domain of this work is the sedimentary petrography, the formal description of rocks in the scope of Petrological studies. The task in focus is the compositional description of minerals in a rock sample, both in qualitative and quantitative way, in order to evaluate the potential of a petroleum reservoir. Several aspects of the rock are studied by the geologist to suggest better techniques for exploration and production of oil fields. The approach defined in this project was validated through the implementation of an image annotation and referencing modules associated to microscope stage appliance that allows the computer to control the movement of the stage during the description process. The whole process of rock sample analysis using the software and the hardware is detailed in order to expose the visual knowledge and its physical position in the rock sample. This approach complements the symbolic representation models that are usually insufficient to capture the semantic of diagnostic features that support reasoning in visual domains.

Cette thèse propose une nouvelle méthodologie pour la construction et l’utilisation de modèles à base de connaissances pour l'annotation automatique d'images. Plus précisément, nous proposons dans un premier lieu des approches pour la construction automatique de modèles de connaissances explicites et structurés, à savoir des hiérarchies sémantiques et des ontologies multimédia adaptées pour l'annotation d'images. Ainsi, nous proposons une approche pour la construction automatique de hiérarchies sémantiques. Notre approche est basée sur une nouvelle mesure « sémantico-visuelle » entre concepts et un ensemble de règles qui permettent de relier les concepts les plus apparentés jusqu'à l'aboutissement à la hiérarchie finale. Ensuite, nous proposons de construire des modèles de connaissances plus riches en terme de sémantique et qui modélisent donc d'autres types de relations entre les concepts de l’image. Par conséquent, nous proposons une nouvelle approche pour la construction automatique d'une ontologie multimédia qui modélise non seulement les relations de subsomption, mais aussi les relations spatiales et contextuelles entre les concepts de l'image. L'ontologie proposée est adaptée pour raisonner sur la cohérence de l’annotation d'images. Afin d'évaluer l'efficacité des modèles de connaissances construits, nous proposons de les utiliser par la suite dans un cadre d'annotation d'images. Nous proposons donc une approche, basée sur la structure des hiérarchies sémantiques, pour la classification hiérarchique d'images. Puis, nous proposons une approche générique, combinant des techniques d'apprentissage automatique et le raisonnement ontologique flou, afin de produire des annotations d’images sémantiquement pertinentes. Des évaluations empiriques de nos approches ont montré une amélioration significative de la précision des annotations d'images. / This dissertation proposes a new methodology for building and using structured knowledge models for automatic image annotation. Specifically, our first proposals deal with the automatic building of explicit and structured knowledge models, such as semantic hierarchies and multimedia ontologies, dedicated to image annotation. Thereby, we propose a new approach for building semantic hierarchies faithful to image semantics. Our approach is based on a new image-semantic similarity measure between concepts and on a set of rules that allow connecting the concepts with higher relatedness till the building of the final hierarchy. Afterwards, we propose to go further in the modeling of image semantics through the building of explicit knowledge models that incorporate richer semantic relationships between image concepts. Therefore, we propose a new approach for automatically building multimedia ontologies consisting of subsumption relationships between concepts, and also other semantic relationships such as contextual and spatial relations. Fuzzy description logics are used as a formalism to represent our ontology and to deal with the uncertainty and the imprecision of concept relationships. In order to assess the effectiveness of the built structured knowledge models, we propose subsequently to use them in a framework for image annotation. We propose therefore an approach, based on the structure of semantic hierarchies, to effectively perform hierarchical image classification. Furthermore, we propose a generic approach for image annotation combining machine learning techniques, such as hierarchical image classification, and fuzzy ontological-reasoning in order to achieve a semantically relevant image annotation. Empirical evaluations of our approaches have shown significant improvement in the image annotation accuracy.

No description available.

Nos domínios visuais, interpretações são iniciadas pela identificação de feições em imagens que irão, posteriormente, dar suporte aos processos mais abstratos de inferência. Para desenvolver sistemas de conhecimento neste tipo de domínio é necessário buscar a melhor representação do conhecimento visual para ser utilizado pelos métodos de inferência. A representação em formato simbólico deste conhecimento auxilia na captura do conhecimento implícito presente em imagens, permitindo seu uso nos processos de raciocínio, mesmo aceitando que parte desse conhecimento não é externalizado e, em conseqüência, não poderá ser representado adequadamente. Estudos recentes têm utilizado anotação de imagens como uma maneira capaz de auxiliar na explicitação do conhecimento, ampliando a expressividade dos formalismos de representação e permitindo o registro das informações associadas às imagens. Embora anotações de imagens flexibilizem a captura do conhecimento, ontologias são associadas às anotações para garantir a formalização do conhecimento nas imagens, suprindo os termos de domínio que podem ser usados para anotar e auxiliar a uniformização da linguagem nas consultas. O objetivo desse trabalho é capturar e documentar o conhecimento visual que dá suporte à inferência nas tarefas de interpretações. Nesse trabalho é elaborada uma maneira de identificar objetos em imagens que contenham feições diagnósticas através da utilização de uma ontologia de domínio pré-existente. Essa identificação de objetos é explorada para permitir a localização física de uma determinada feição em um objeto real. O resultado disso é a identificação de feições em uma imagem tendo-se um referencial de posição segundo um sistema de coordenadas espacial, possibilitando o seu re-posicionamento. O trabalho ainda explora recursos para a melhor utilização de informações relacionadas a uma imagem. Dessa forma, serve de documentação auxiliar para prover diferencial em interpretações. O domínio de aplicação deste trabalho é a Petrografia Sedimentar que tem como uma das tarefas a descrição quantitativa e qualitativa dos minerais que compõem uma amostra de rocha, visando a análise do potencial de um reservatório de petróleo. As informações são visualizadas em uma amostra de rocha e, assim, um especialista toma decisões quanto à viabilidade de exploração do reservatório. Esse modelo foi validado através da implementação de um módulo de anotação de imagem e de referenciamento de objetos, juntamente com um hardware. O processo de descrição de amostra de rocha é detalhado para se explorar o conhecimento em imagens com a precisão da localização dos objetos nela identificados. Essa abordagem complementa os modelos de representação simbólica, que normalmente são insuficientes para capturar a semântica das feições que dão suporte a inferência em domínios visuais. / Problem solvers in visual domains apply image interpretation to extract evidences to support the reasoning process in the search of solution. In order to develop knowledge systems in this kind of domain, it is necessary to represent the knowledge that is extracted from the scenes in the domain in such way that can be used by the inference methods. The symbolic representation formalisms help in shading light on the implicit knowledge embodied in images, allowing its use in reasoning processes, even accepting that part of this knowledge can not be properly expressed by the experts and, in consequence, it is never going to be adequately represented in knowledge models. Some studies use image annotation to extend the semantic capture of the visual knowledge beyond the expressiveness of representation formalisms, allowing that additional content could be described and stored with the images. Although the annotation is a more flexible way of capturing knowledge, ontologies are used along with it to guarantee the formalization of the knowledge annotated in the images, supplying the domain terms that can be used in the annotation process and helping the uniformization of the language and further consultation. The goal of this work is capturing and documenting the visual knowledge that supports inference on interpretation tasks. The work offer a way of identifying objects in images that correspond to diagnostic features and describing them with the support of a pre-existent domain ontology. The object identification is associated to the physical location of the feature in the real object, with the help of a hardware appliance. The result is the feature identification in the image with the reference of its position in a system of spatial coordinates. The work also explores multimedia resources for the best use of information associated to the image, providing an extra-support for knowledge documentation for interpretation tasks. The application domain of this work is the sedimentary petrography, the formal description of rocks in the scope of Petrological studies. The task in focus is the compositional description of minerals in a rock sample, both in qualitative and quantitative way, in order to evaluate the potential of a petroleum reservoir. Several aspects of the rock are studied by the geologist to suggest better techniques for exploration and production of oil fields. The approach defined in this project was validated through the implementation of an image annotation and referencing modules associated to microscope stage appliance that allows the computer to control the movement of the stage during the description process. The whole process of rock sample analysis using the software and the hardware is detailed in order to expose the visual knowledge and its physical position in the rock sample. This approach complements the symbolic representation models that are usually insufficient to capture the semantic of diagnostic features that support reasoning in visual domains.

La présente thèse s’intéresse à l’annotation multimodale d’images dans le contexte des médias sociaux. Notre objectif est de combiner les modalités visuelles et textuelles (tags) afin d’améliorer les performances d’annotation d’images. Cependant, ces tags sont généralement issus d’une indexation personnelle, fournissant une information imparfaite et partiellement pertinente pour un objectif de description du contenu sémantique de l’image. En outre, en combinant les scores de prédiction de différents classifieurs appris sur les différentes modalités, l’annotation multimodale d’image fait face à leurs imperfections: l’incertitude, l’imprécision et l’incomplétude. Dans cette thèse, nous considérons que l’annotation multimodale d’image est soumise à ces imperfections à deux niveaux : niveau représentation et niveau décision. Inspiré de la théorie de fusion de l’information, nous concentrons nos efforts dans cette thèse sur la définition, l’identification et la prise en compte de ces aspects d’imperfections afin d’améliorer l’annotation d’images. / This thesis deals with multimodal image annotation in the context of social media. We seek to take advantage of textual (tags) and visual information in order to enhance the image annotation performances. However, these tags are often noisy, overly personalized and only a few of them are related to the semantic visual content of the image. In addition, when combining prediction scores from different classifiers learned on different modalities, multimodal image annotation faces their imperfections (uncertainty, imprecision and incompleteness). Consequently, we consider that multimodal image annotation is subject to imperfections at two levels: the representation and the decision. Inspired from the information fusion theory, we focus in this thesis on defining, identifying and handling imperfection aspects in order to improve image annotation.

This master thesis is oriented on a problem of creating video and image annotations. This problem is solved by crowdsourcing approach. Crowdsourcing games were designed and implemented to make solution of this problem . It was proven by testing that these games are capable of creating high quality annotations. Launching these games on a larger scale could create large database of annotated videos and images.

Computational creativity is a new and developing field of artificial intelligence concerned with computational systems that either autonomously produce original and functional products, or that augment the ability of humans to do so. As the role of computers in our daily lives is continuing to expand, the need for such systems is becoming increasingly important. We introduce and document the development of a new “creative” system, called DARCI (Digital ARtist Communicating Intention), that is designed to autonomously create novel artistic images that convey linguistic concepts to the viewer. Within the scope of this work, the system becomes capable of creating non-photorealistic renderings of existing image compositions so that they convey the semantics of given adjectives. Ultimately, we show that DARCI is capable of producing surprising artifacts that are competitive, in some ways, with those produced by human artists. As with the development of any “creative” system, we are faced with the challenges of incorporating the philosophies of creativity into the design of the system, assessing the system's creativity, overcoming technical shortcomings of extant modern algorithms, and justifying the system within its creative domain (in this case, visual art). In meeting these challenges with DARCI, we demonstrate three broad contributions of the system: 1) the contribution to the field of computational creativity in the form of an original system, new approaches to achieving autonomy in creative systems, and new practical assessment methods; 2) the contribution to the field of computer vision in the form of new image features for affective image annotation and a new dataset; and 3) the contribution to the domain of visual art in the form of mutually beneficial collaborations and participation in several art galleries and exhibits.

Annotation is essential in machine learning. Building an accurate object detec-tion model requires a large, diverse dataset, which poses challenges due to thetime-consuming nature of manual annotation. This thesis was made in collabora-tion with Project Ngulia, which aims at developing technical solutions to protectand monitor wild animals. A contribution of this work was to integrate an effi-cient semi-automatic image annotation tool within the Ngulia system, with theaim of streamlining the annotation process and improving the employed objectdetection models. Through research into available annotation tools, a custom toolwas deemed the most cost-effective and flexible option. It utilizes object detec-tion model predictions as annotation suggestions, improving the efficiency of theannotation process. The efficiency was evaluated through a user test, with partic-ipants achieving an average reduction of approximately 2 seconds in annotationspeed when utilizing suggestions. This reduction was supported as statisticallysignificant through a one-way ANOVA test. Additionally, it was investigated which images should be prioritized for an-notation in order to obtain the the most accurate predictions. Different samplingmethods were investigated and compared. The performance of the obtained mod-els remained relatively consistent, although with the even distribution methodat top. This indicate that the choice of sampling method may not substantiallyimpact the accuracy of the model, as the performance of the methods was rela-tively comparable. Moreover, different methods of selecting training data in there-training process was compared. The difference in performance was consider-ately small, likely due to the limited and balanced data pool. The experimentsdid however indicate that incorporating previously seen data with unseen datacould be beneficial, and that a reduced dataset can be sufficient. However, furtherinvestigation is required to fully understand the extent of these benefits.