Right now you are reading a sentence. Earlier, you might have been looking at a realistic picture, such as a photograph, or an outline drawing in a set of instructions. If you are a programmer, you work with sentence-like structures, such as code, or a system diagram. These are all graphic representations. To varying degrees, the effectiveness of every graphic representation relies on its ability to convey the designer’s intended meaning and elicit the intended reaction from its audience. However, the design of graphic representations, even in technical domains such as visual programming language design or interactive information visualization, currently relies heavily on general principles based solely on practice, intuition, and informal measures of effectiveness from the applied art and craft of design (as opposed to scientific analysis or theory). There is an increasing demand for a scientific understanding of design and its evaluation from stakeholders (who seek evidence for effectiveness) and designers (who seek to advance their field). Because both the creation of graphic displays and their perception are literally embodied experiences, a model was developed with an embodiment orientation, specifically based on how graphics are perceptually and cognitively processed. In my research, I found that graphic representations are constituted of two properties, pictorial and symbolic information, that emerge through two interrelated aspects of perception. In sighted individuals, for example, every graphic representation makes use of biological capabilities to process visual sensation (i.e., light hitting the retina), which are processed in relation to culturally-learned capabilities (i.e., writing). I observed how graphic representations – such as pictures, diagrams, and sentences – are “naturally selected” (i.e., during different phases of design or problem solving). From these observations, I developed a model that distinguishes and predicts the effectiveness of pictures, diagrams, and sentences, in terms of how object relations and attributes are pictorially or symbolically represented, relative to the functional roles of those representations, contexts, and in some cases, individual perceptual-cognitive differences among perceivers. This model is a step toward a science of graphics that could lead to evaluation techniques for information systems, theories for inclusive design, and ergonomically designed software programming tools.

Right now you are reading a sentence. Earlier, you might have been looking at a realistic picture, such as a photograph, or an outline drawing in a set of instructions. If you are a programmer, you work with sentence-like structures, such as code, or a system diagram. These are all graphic representations. To varying degrees, the effectiveness of every graphic representation relies on its ability to convey the designer’s intended meaning and elicit the intended reaction from its audience. However, the design of graphic representations, even in technical domains such as visual programming language design or interactive information visualization, currently relies heavily on general principles based solely on practice, intuition, and informal measures of effectiveness from the applied art and craft of design (as opposed to scientific analysis or theory). There is an increasing demand for a scientific understanding of design and its evaluation from stakeholders (who seek evidence for effectiveness) and designers (who seek to advance their field). Because both the creation of graphic displays and their perception are literally embodied experiences, a model was developed with an embodiment orientation, specifically based on how graphics are perceptually and cognitively processed. In my research, I found that graphic representations are constituted of two properties, pictorial and symbolic information, that emerge through two interrelated aspects of perception. In sighted individuals, for example, every graphic representation makes use of biological capabilities to process visual sensation (i.e., light hitting the retina), which are processed in relation to culturally-learned capabilities (i.e., writing). I observed how graphic representations – such as pictures, diagrams, and sentences – are “naturally selected” (i.e., during different phases of design or problem solving). From these observations, I developed a model that distinguishes and predicts the effectiveness of pictures, diagrams, and sentences, in terms of how object relations and attributes are pictorially or symbolically represented, relative to the functional roles of those representations, contexts, and in some cases, individual perceptual-cognitive differences among perceivers. This model is a step toward a science of graphics that could lead to evaluation techniques for information systems, theories for inclusive design, and ergonomically designed software programming tools.

Capes / Essa pesquisa pretende refletir sobre as interações socioculturais no uso técnico-estético da fotografia. Para tanto, seguiu-se uma metodologia de análises de imagens, e observações de práticas de oficinas de fotografia com jovens e crianças de Curitiba e Região Metropolitana, realizadas pela autora anteriormente a esse estudo. As análises basearam-se em estudos da imagem e semiótica visual. Contudo, tornou-se essencial estimular a compreensão dos sistemas de representações como construção social e possibilidades de expressão através da fotografia, tema desafiador em um tempo em que as representações visuais são naturalizadas, sem uma percepção crítica do que nos é colocado diante dos olhos. Partindo de um breve histórico da fotografia, seus modos de utilização na transformação social, buscou-se articular a prática e a apropriação fotográfica na construção do olhar de si, do outro e do entorno. Considerando a percepção um processo interativo e criador, atualizada nos usos da fotografia, tanto como linguagem quanto como aparato tecnológico, faz-se necessário despertar novos olhares e dar sentido a interpretações além das próprias imagens. / This research aims to reflect on the socio-cultural interactions in the techno-aesthetic use of photography. To do so, it was served as a method data analysis and observations of practices in photography workshops with youth and children in the metropolitan region of Curitiba, applied by the author before this research. The analyzes were based on studies of image and visual semiotics. Hence, it became essential stimulate the understanding of the systems of representations as a social construction, and the possibilities of expression through photography, challenging theme in a time when visual representations are naturalized, without having a critical awareness of what is placed in front of our eyes. We start with a brief historical analysis of photography, its modes of using it in social transformations, seeking to articulate the practice and photographic appropriation, the look construction the of yourself, of the others, and of the environment. Considering the perception as an interactive and creative process, and updated in different uses of photography, and it is used both as language and as a technological device, it became necessary to arouse new looks ans give more sensibility to other interpretations beyond the images themselves.

Capes / Essa pesquisa pretende refletir sobre as interações socioculturais no uso técnico-estético da fotografia. Para tanto, seguiu-se uma metodologia de análises de imagens, e observações de práticas de oficinas de fotografia com jovens e crianças de Curitiba e Região Metropolitana, realizadas pela autora anteriormente a esse estudo. As análises basearam-se em estudos da imagem e semiótica visual. Contudo, tornou-se essencial estimular a compreensão dos sistemas de representações como construção social e possibilidades de expressão através da fotografia, tema desafiador em um tempo em que as representações visuais são naturalizadas, sem uma percepção crítica do que nos é colocado diante dos olhos. Partindo de um breve histórico da fotografia, seus modos de utilização na transformação social, buscou-se articular a prática e a apropriação fotográfica na construção do olhar de si, do outro e do entorno. Considerando a percepção um processo interativo e criador, atualizada nos usos da fotografia, tanto como linguagem quanto como aparato tecnológico, faz-se necessário despertar novos olhares e dar sentido a interpretações além das próprias imagens. / This research aims to reflect on the socio-cultural interactions in the techno-aesthetic use of photography. To do so, it was served as a method data analysis and observations of practices in photography workshops with youth and children in the metropolitan region of Curitiba, applied by the author before this research. The analyzes were based on studies of image and visual semiotics. Hence, it became essential stimulate the understanding of the systems of representations as a social construction, and the possibilities of expression through photography, challenging theme in a time when visual representations are naturalized, without having a critical awareness of what is placed in front of our eyes. We start with a brief historical analysis of photography, its modes of using it in social transformations, seeking to articulate the practice and photographic appropriation, the look construction the of yourself, of the others, and of the environment. Considering the perception as an interactive and creative process, and updated in different uses of photography, and it is used both as language and as a technological device, it became necessary to arouse new looks ans give more sensibility to other interpretations beyond the images themselves.

Der Einsatz von Bildern in den unterschiedlichsten Bereichen ist enorm gestiegen, da Bilder die Fähigkeit haben Erlebnisse, sowie Emotionen zu erzeugen und die Phantasie anzuregen. Zudem lässt die rasante Entwicklung im Multimediabereich die Anzahl der fotografierten und gespeicherten Bilder steigen. Die Suche nach dem „besten Bild“ für z.B. eine Kampagne gestaltet sich schwierig, da die Inhalte mehrerer Bilder zu einem Thema nicht selten eine hohe Ähnlichkeit aufweisen. Die Bilder können sich allerdings sehr deutlich in ihren Low-Level Features, wie Farbton, Sättigung und Helligkeit, unterscheiden. Jedoch ist der Emotional Gap zwischen diesen Low-Level Features und der dahinter steckenden High-Level-Semantik im inhaltsbasierten Image Retrieval nur marginal untersucht. Im Fokus dieser Arbeit steht die Analyse des Einflusses der emotionalen Wirkung eines Bildes auf die Qualität der Image Retrieval Ergebnisse. Dies umfasst zum einen die Untersuchung der von Farbeigenschaften eines Bildes ausgelösten Emotionen, sowie die Evaluation der Ergebnisse einer emotionalen Bildsuche. Durch verschiedene Experimente kann gezeigt werden, dass die Helligkeit und der Farbton die emotionale Wahrnehmung eines Bildes maßgeblich beeinflussen. Anhand der Ergebnisse konnte eine emotionale Annotation von Bildern und somit die Einbindung von Emotionen in den Suchprozess ermöglicht werden. Die anschließende Evaluierung der Suchergebnisse zeigt, dass die Qualität der Ergebnisse des Image Retrievals verbessert werden konnte. / The use of pictures in a variety of areas has increased tremendously in recent years, as they stimulate a person’s imagination and help to create first experiences and emotions. Furthermore, the rapid developments in multimedia have led to an escalation of the number of digitally stored pictures and photographs. Consequently, finding the ‘best picture’ for a convincing advertising campaign has been becoming increasingly difficult due to the abundance of available pictures. To further complicate this search process, a lot of pictures related to a specific topic are very similar with regard to their content. However, their low-level features, such as hue, saturation, and luminance, might differ considerably. Therefore, this work focusses on the influence of emotional characteristics on the image retrieval process. This includes the study of emotions caused by the color properties of a picture, as well as the evaluation of the results of an emotional image retrieval processes. Results of different experiments show that a picture’s luminance and color have the power to influence emotion. The subsequent evaluation of the results shows an improvement of emotional image retrieval processes. Consequently, one can conclude that the consideration of emotions for ranking affects the quality of the results of the Image Retrieval positively.

Der Einsatz von Bildern in den unterschiedlichsten Bereichen ist enorm gestiegen, da Bilder die Fähigkeit haben Erlebnisse, sowie Emotionen zu erzeugen und die Phantasie anzuregen. Zudem lässt die rasante Entwicklung im Multimediabereich die Anzahl der fotografierten und gespeicherten Bilder steigen. Die Suche nach dem „besten Bild“ für z.B. eine Kampagne gestaltet sich schwierig, da die Inhalte mehrerer Bilder zu einem Thema nicht selten eine hohe Ähnlichkeit aufweisen. Die Bilder können sich allerdings sehr deutlich in ihren Low-Level Features, wie Farbton, Sättigung und Helligkeit, unterscheiden. Jedoch ist der Emotional Gap zwischen diesen Low-Level Features und der dahinter steckenden High-Level-Semantik im inhaltsbasierten Image Retrieval nur marginal untersucht. Im Fokus dieser Arbeit steht die Analyse des Einflusses der emotionalen Wirkung eines Bildes auf die Qualität der Image Retrieval Ergebnisse. Dies umfasst zum einen die Untersuchung der von Farbeigenschaften eines Bildes ausgelösten Emotionen, sowie die Evaluation der Ergebnisse einer emotionalen Bildsuche. Durch verschiedene Experimente kann gezeigt werden, dass die Helligkeit und der Farbton die emotionale Wahrnehmung eines Bildes maßgeblich beeinflussen. Anhand der Ergebnisse konnte eine emotionale Annotation von Bildern und somit die Einbindung von Emotionen in den Suchprozess ermöglicht werden. Die anschließende Evaluierung der Suchergebnisse zeigt, dass die Qualität der Ergebnisse des Image Retrievals verbessert werden konnte. / The use of pictures in a variety of areas has increased tremendously in recent years, as they stimulate a person’s imagination and help to create first experiences and emotions. Furthermore, the rapid developments in multimedia have led to an escalation of the number of digitally stored pictures and photographs. Consequently, finding the ‘best picture’ for a convincing advertising campaign has been becoming increasingly difficult due to the abundance of available pictures. To further complicate this search process, a lot of pictures related to a specific topic are very similar with regard to their content. However, their low-level features, such as hue, saturation, and luminance, might differ considerably. Therefore, this work focusses on the influence of emotional characteristics on the image retrieval process. This includes the study of emotions caused by the color properties of a picture, as well as the evaluation of the results of an emotional image retrieval processes. Results of different experiments show that a picture’s luminance and color have the power to influence emotion. The subsequent evaluation of the results shows an improvement of emotional image retrieval processes. Consequently, one can conclude that the consideration of emotions for ranking affects the quality of the results of the Image Retrieval positively.