Der Einsatz von Bildern in den unterschiedlichsten Bereichen ist enorm gestiegen, da Bilder die Fähigkeit haben Erlebnisse, sowie Emotionen zu erzeugen und die Phantasie anzuregen. Zudem lässt die rasante Entwicklung im Multimediabereich die Anzahl der fotografierten und gespeicherten Bilder steigen. Die Suche nach dem „besten Bild“ für z.B. eine Kampagne gestaltet sich schwierig, da die Inhalte mehrerer Bilder zu einem Thema nicht selten eine hohe Ähnlichkeit aufweisen. Die Bilder können sich allerdings sehr deutlich in ihren Low-Level Features, wie Farbton, Sättigung und Helligkeit, unterscheiden. Jedoch ist der Emotional Gap zwischen diesen Low-Level Features und der dahinter steckenden High-Level-Semantik im inhaltsbasierten Image Retrieval nur marginal untersucht. Im Fokus dieser Arbeit steht die Analyse des Einflusses der emotionalen Wirkung eines Bildes auf die Qualität der Image Retrieval Ergebnisse. Dies umfasst zum einen die Untersuchung der von Farbeigenschaften eines Bildes ausgelösten Emotionen, sowie die Evaluation der Ergebnisse einer emotionalen Bildsuche. Durch verschiedene Experimente kann gezeigt werden, dass die Helligkeit und der Farbton die emotionale Wahrnehmung eines Bildes maßgeblich beeinflussen. Anhand der Ergebnisse konnte eine emotionale Annotation von Bildern und somit die Einbindung von Emotionen in den Suchprozess ermöglicht werden. Die anschließende Evaluierung der Suchergebnisse zeigt, dass die Qualität der Ergebnisse des Image Retrievals verbessert werden konnte. / The use of pictures in a variety of areas has increased tremendously in recent years, as they stimulate a person’s imagination and help to create first experiences and emotions. Furthermore, the rapid developments in multimedia have led to an escalation of the number of digitally stored pictures and photographs. Consequently, finding the ‘best picture’ for a convincing advertising campaign has been becoming increasingly difficult due to the abundance of available pictures. To further complicate this search process, a lot of pictures related to a specific topic are very similar with regard to their content. However, their low-level features, such as hue, saturation, and luminance, might differ considerably. Therefore, this work focusses on the influence of emotional characteristics on the image retrieval process. This includes the study of emotions caused by the color properties of a picture, as well as the evaluation of the results of an emotional image retrieval processes. Results of different experiments show that a picture’s luminance and color have the power to influence emotion. The subsequent evaluation of the results shows an improvement of emotional image retrieval processes. Consequently, one can conclude that the consideration of emotions for ranking affects the quality of the results of the Image Retrieval positively.

Search engines play a very important role in daily life. As multimedia product becomes more and more popular, people have developed search engines for images and videos. In the first part of this thesis, I propose a prototype of a book image search engine. I discuss tag representation for the book images, as well as the way to apply the probabilistic model to generate image tags. Then I propose the random walk refinement method using tag similarity graph. The image search system is built on the Galago search engine developed in UMASS CIIR lab. Consider the large amount of data the search engines need to process, I bring in cloud environment for the large-scale distributed computing in the second part of this thesis. I discuss two models, one is the MapReduce model, which is currently one of the most popular technologies in the IT industry, and the other one is the Maiter model. The asynchronous accumulative update mechanism of Maiter model is a great fit for the random walk refinement process, which takes up 84% of the entire run time, and it accelerates the refinement process by 46 times.

The National Library of Medicine (NLM), an institute in the National Institutes of Health (NIH), maintains a collection of 17,000 digitized spine X-ray images obtained from the second National Health and Nutrition Examination Survey (NHANES II). Research effort has been devoted to develop a web-accessible retrieval system that allows retrieval of images from the NHANES II database on relevant and frequently found pathologies. A comprehensive and successful image retrieval system requires effective image representation and matching methods, relevance feedback algorithms to incorporate user opinions, and efficient indexing schemes for fast access to image databases. This dissertation studies and develops approaches for all of the above areas within the context of content-Based Image Retrieval (CBIR) of spine X-ray images from the NHANES II collection. Shape is an important characteristic for describing pertinent pathologies in various types of medical images, including spine X-ray images. Retrieving images with shapes similar to a specific user query can be useful for finding pathologies exhibited in images in large survey collections. In this work, vertebral outlines are extracted for image retrieval using shape matching methods to detect the presence of anterior osteophytes. The Multiple Open Triangle (MOT) shape representation method is proposed for partial shape matching (PSM), and a Corner-Guided Dynamic Programming (DP) strategy is developed to search partial intervals for matching comparison based on a 9-point model marked by a board-certified radiologist. The MOT method demonstrates higher retrieval accuracy compared to other approaches and the retrieval speed is improved significantly through the use of Corner-Guided DP. Computer-calculated low-level image features fall short when imitating high-level human visual perception. Relevance Feedback (RF) attempts to bridge the gap between the two by analyzing and employing user feedback. The need for overcoming this gap is more evident in medical image retrieval. Existing RF approaches are analyzed and a weight-updating formula for RF is developed. A hybrid retrieval approach is proposed that utilizes both CBIR with RF and RF history. This hybrid approach uses short-term memory to store the feedback history, which contributes to the retrieval results and helps select images for user feedback. An approximate 20% average increase in retrieval recall percentage is achieved within two RF iterations. Efficient indexing methods are desired for fast database access. An agglomerative clustering algorithm is adopted to pre-index the database based on pre-calculated pair-wise distances between indexed parts. Retrieval with this pre-indexing procedure is shown to offer faster retrieval and maintain a comparable recall percentage.

An image mosaic is an artistic work that uses a number of smaller images creatively combined together to form another larger image. Each building block image, or tessera, has its own distinctive and meaningful content, but when viewed from a distance the tesserae come together to form an aesthetically pleasing montage. This work presents the design and implementation of MosaiX, a computer software system that generates these image mosaics automatically. To control the image mosaic creation process, several parameters are used within the system. Each parameter affects the overall mosaic quality, as well as required processing time, in its own unique way. A detailed analysis is performed to evaluate each parameter individually. Additionally, this work proposes two novel ways by which to evaluate the quality of an image mosaic in a quantitative way. One method focuses on the perceptual color accuracy of the mosaic reproduction, while the other concentrates on edge replication. Both measures include preprocessing to take into account the unique visual features present in an image mosaic. Doing so minimizes quality penalization due the inherent properties of an image mosaic that make them visually appealing.

Satellite imagery research used to be an expensive research topic for companies and organizations due to the limited data and compute resources. As the computing power and storage capacity grows exponentially, a large amount of aerial and satellite images are generated and analyzed everyday for various applications. Current technological advancement and extensive data collection by numerous Internet of Things (IOT) devices and platforms have amplified labeled natural images. Such data availability catalyzed the development and performance of current state-of-the-art image classification and cross-modal models. Despite the abundance of publicly available remote sensing images, very few remote sensing (RS) images are labeled and even fewer are multi-captioned.These scarcities limit the scope of fine tuned state of the art models to at most 38 classes, based on the PatternNet data, one of the largest publicly available labeled RS data. Recent state-of-the art image-to-image retrieval and detection models in RS have shown great results. Because the text-to-image retrieval of RS images is still emerging, it still faces some challenges in the retrieval of those images.These challenges are based on the inaccurate retrieval of image categories that were not present in the training dataset and the retrieval of images from descriptive input. Motivated by those shortcomings in current cross-modal remote sensing image retrieval, we proposed CLIP-RS, a cross-modal remote sensing image retrieval platform. Our proposed framework CLIP-RS is a framework that combines a fine-tuned implementation of a recent state of the art cross-modal and text-based image retrieval model, Contrastive Language Image Pre-training (CLIP) and FAISS (Facebook AI similarity search), a library for efficient similarity search. Our implementation is deployed on a Web App for inference task on text-to-image and image-to-image retrieval of RS images collected via the Mapbox GL JS API. We used the free tier option of the Mapbox GL JS API and took advantage of its raster tiles option to locate the retrieved results on a local map, a combination of the downloaded raster tiles. Other options offered on our platform are: image similarity search, locating an image in the map, view images' geocoordinates and addresses.In this work we also proposed two remote sensing fine-tuned models and conducted a comparative analysis of our proposed models with a different fine-tuned model as well as the zeroshot CLIP model on remote sensing data. / Master of Science / Satellite imagery research used to be an expensive research topic for companies and organizations due to the limited data and compute resources. As the computing power and storage capacity grows exponentially, a large amount of aerial and satellite images are generated and analyzed everyday for various applications. Current technological advancement and extensive data collection by numerous Internet of Things (IOT) devices and platforms have amplified labeled natural images. Such data availability catalyzed the devel- opment and performance of current state-of-the-art image classification and cross-modal models. Despite the abundance of publicly available remote sens- ing images, very few remote sensing (RS) images are labeled and even fewer are multi-captioned.These scarcities limit the scope of fine tuned state of the art models to at most 38 classes, based on the PatternNet data,one of the largest publicly avail- able labeled RS data.Recent state-of-the art image-to-image retrieval and detection models in RS have shown great results. Because the text-to-image retrieval of RS images is still emerging, it still faces some challenges in the re- trieval of those images.These challenges are based on the inaccurate retrieval of image categories that were not present in the training dataset and the re- trieval of images from descriptive input. Motivated by those shortcomings in current cross-modal remote sensing image retrieval, we proposed CLIP-RS, a cross-modal remote sensing image retrieval platform.Cross-modal retrieval focuses on data retrieval across different modalities and in the context of this work, we focus on textual and imagery modalities.Our proposed frame- work CLIP-RS is a framework that combines a fine-tuned implementation of a recent state of the art cross-modal and text-based image retrieval model, Contrastive Language Image Pre-training (CLIP) and FAISS (Facebook AI similarity search), a library for efficient similarity search. In deep learning, the concept of fine tuning consists of using weights from a different model or algorithm into a similar model with different domain specific application. Our implementation is deployed on a Web Application for inference tasks on text-to-image and image-to-image retrieval of RS images collected via the Mapbox GL JS API. We used the free tier option of the Mapbox GL JS API and took advantage of its raster tiles option to locate the retrieved results on a local map, a combination of the downloaded raster tiles. Other options offered on our platform are: image similarity search, locating an image in the map, view images' geocoordinates and addresses.In this work we also pro- posed two remote sensing fine-tuned models and conducted a comparative analysis of our proposed models with a different fine-tuned model as well as the zeroshot CLIP model on remote sensing data. Detection models in RS have shown great results. Because the text-to-image retrieval of RS images is still emerging, it still faces some challenges in the re- trieval of those images.These challenges are based on the inaccurate retrieval of image categories that were not present in the training dataset and the re- trieval of images from descriptive input. Motivated by those shortcomings in current cross-modal remote sensing image retrieval, we proposed CLIP-RS, a cross-modal remote sensing image retrieval platform.Cross-modal retrieval focuses on data retrieval across different modalities and in the context of this work, we focus on textual and imagery modalities.Our proposed frame- work CLIP-RS is a framework that combines a fine-tuned implementation of a recent state of the art cross-modal and text-based image retrieval model, Contrastive Language Image Pre-training (CLIP) and FAISS (Facebook AI similarity search), a library for efficient similarity search. In deep learning, the concept of fine tuning consists of using weights from a different model or algorithm into a similar model with different domain specific application. Our implementation is deployed on a Web Application for inference tasks on text-to-image and image-to-image retrieval of RS images collected via the Mapbox GL JS API. We used the free tier option of the Mapbox GL JS API and took advantage of its raster tiles option to locate the retrieved results on a local map, a combination of the downloaded raster tiles. Other options offered on our platform are: image similarity search, locating an image in the map, view images' geocoordinates and addresses.In this work we also pro- posed two remote sensing fine-tuned models and conducted a comparative analysis of our proposed models with a different fine-tuned model as well as the zeroshot CLIP model on remote sensing data.

No description available.

No description available.

Yes / Yes / The proliferation of digital media has led to a huge interest in classifying and indexing media objects for generic search and usage. In particular, we are witnessing a colossal growth in digital image repositories that are difficult to navigate using free-text search mechanisms, which often return inaccurate matches as they in principle rely on statistical analysis of query keyword recurrence in the image annotation or surrounding text. In this paper we present a semantically-enabled image annotation and retrieval engine that is designed to satisfy the requirements of the commercial image collections market in terms of both accuracy and efficiency of the retrieval process. Our search engine relies on methodically structured ontologies for image annotation, thus allowing for more intelligent reasoning about the image content and subsequently obtaining a more accurate set of results and a richer set of alternatives matchmaking the original query. We also show how our well-analysed and designed domain ontology contributes to the implicit expansion of user queries as well as the exploitation of lexical databases for explicit semantic-based query expansion.

The thesis explores the application of Convolutional Neural Networks (CNNs) in the fashion industry, focusing on fashion style classification and image retrieval. Employing transfer learning, the study investigates the effectiveness of fine-tuning pre-trained CNN models to adapt them for a specific fashion recognition task by initially performing an extensive hyperparameter optimisation, utilising the Optuna framework.  The impact of dataset size on model performance was examined by comparing the accuracy of models trained on datasets containing 2000 and 8000 images. Results indicate that larger datasets significantly improve model performance, particularly for more complex models like EfficientNetV2S, which showed the best overall performance with an accuracy of 85.38% on the larger dataset after fine-tuning. The best-performing and fine-tuned model was subsequently used for image retrieval as features were extracted from the last convolutional layer. These features were used in a cosine similarity measure to rank images by their similarity to a query image. This technique achieved a mean average precision (mAP) of 0.4525, indicating that CNNs hold promise for enhancing fashion retrieval systems, although further improvements and validations are necessary. Overall, this research highlights the versatility of CNNs in interpreting and categorizing complex visual data. The importance of well-prepared, targeted data and refined model training strategies is highlighted to enhance the accuracy and applicability of AI in diverse fields.

Increasing size and prevalence of real time information have become important characteristics of databases found on the internet. Due to changing information, the relevancy ranking of the search results also changes. Current methods in information retrieval, which are based on offline indexing, are not efficient in such dynamic search spaces and cannot quickly provide the most current results. Due to the explosive growth of the internet, stove-piped approaches for dealing with dynamism by simply employing large computational resources are ultimately not scalable. A new processing methodology that incorporates intelligent resource allocation strategies is required. Also, modeling the dynamism in the search space in real time is essential for effective resource allocation. In order to support multi-grained dynamic resource allocation, we propose to use a partial processing approach that uses anytime algorithms to process the documents in multiple steps. At each successive step, a more accurate approximation of the final similarity values of the documents is produced. Resource allocation algorithm use these partial results to select documents for processing, decide on the number of processing steps and the computation time allocated for each step. We validate the processing paradigm by demonstrating its viability with image documents. We design an anytime image algorithm that uses a combination of wavelet transforms and machine learning techniques to map low level visual features to higher level concepts. Experimental validation is done by implementing the image algorithm within an established multiagent information retrieval framework called I-FGM. We also formulate a multiagent resource allocation framework for design and performance analysis of resource allocation with partial processing. A key aspect of the framework is modeling changes in the search space as external and internal dynamism using a grid-based search space model. The search space model divides the documents or candidates into groups based on its partial-value and portion processed. Hence the changes in the search space can be effectively represented in the search space model as flow of agents and candidates between the grids. Using comparative experimental studies and detailed statistical analysis we validate the search space model and demonstrate the effectiveness of the resource allocation framework. / Ph. D.