With the proliferation of sensor-enabled smartphones, significant attention has been attracted to develop sensing-driven mobile systems. Current research on sensing-driven mobile systems can be classified into two categories, based on the purpose of sensing. In the first category, smartphones are used to sense personal context information, such as locations, activities, and daily habits to enable applications such as location-aware systems and virtual reality systems. In the second category, smartphones are exploited to collect sensing data of the physical world and enable applications including traffic monitoring, environmental monitoring, and others. As smartphones become blossomed in popularity and ubiquity, new problems have emerged in both categories of mobile sensing systems. In this thesis, we investigate three core challenges by answering the following fundamental questions: first, how can we utilize user context to improve the operating system performance? Second, how can we process sensing data, especially images, with high accuracy? Third, how can we enable distributed sensing while satisfy resource constraints of smartphones? The first part of this thesis studies how to exploit user context to improve the responsiveness of mobile operating systems. We propose a context-aware application-preloading engine named FALCON. The core of FALCON is a decision engine that learns application usage patterns of mobile users and preloads applications ahead of time to improve the responsiveness of mobile OS. Compared with other approaches such as caching schemes like Least Recently Used (LRU), Falcon improves the application responsiveness by two times. The second part of this thesis focuses on image search for mobile phones. We first explore how to improve image search accuracy on centralized servers, and propose an image search engine named CrowdSearch. The core idea of CrowdSearch is to incorporate crowdsourced human validation into the system for removing erroneous results from automated image search engines, while still provide realtime response for mobile users. Compared with existing automated image search engines, CrowdSearch achieves over 95% accuracy consistently across multiple categories of images with response time in a minute. We then extend image search to distributed mobile phones, and emphasis resource constraint problems, especially on energy and bandwidth. We propose a distributed image search system named SenSearch, which turns smartphones into micro image search engines. Images are collected, indexed, and transmitted using compact features that are two magnitudes smaller than their raw format. SenSearch improves the energy and bandwidth cost by five times compared with centralized image search engines.

The considerable growth of digital images online in recent years has shifted users’ concern from whether or not an image is available to how to find a specific image in a sea of online imagery. Image Search Engines cannot satisfy every user, especially users that require specific images with more details. Furthermore, the variety and quantity of available images do not add value for users if they cannot find what they require in an appropriate timeframe. Therefore, an Image Retrieval is required that lets users define detailed search perimeters and find images that match their requirements.This thesis focuses on providing better communication and interaction between users and Image Search Engines. The work presented here aims to let users describe their requirements visually and make approximations of the images that they require by setting perimeters like color, scale and position. This approximation can help in retrieving more appropriate images which more closely match users’ needs. This thesis also proposes to involve users first in improving the Image Search Engine database by uploading their photographs and images, and second in helping other users that are not satisfied with search results, by sending an image as response to their request.To achieve this goal, the thesis applied two methodologies, Research through Design and User Centered Design. These methodologies allowed considering future possibilities and users’ requirements. The communication with users provided by low-fidelity and high-fidelity prototypes as sketches, that were used in workshops and helped in framing the concept and improving different aspects of it.

With a multitude of images available on the Internet, how do we find what we are looking for? This project tries to determine how much the precision and recall of search queries is improved by using a word taxonomy on traditional Text-Based Image Search and Content-Based Image Search. By applying a word taxonomy to different data sources, a strong keyword filter and a keyword extender were implemented and tested. The results show that depending on the implementation, the precision or the recall can be increased. By using a similar approach on real life implementations, it is possible to force images with higher precisions to the front while keeping a high recall value, thus increasing the experienced relevance of image search. / Med den mängd bilder som nu finns tillgänglig på Internet, hur kan vi fortfarande hitta det vi letar efter? Denna uppsats försöker avgöra hur mycket bildprecision och bildåterkallning kan öka med hjälp av appliceringen av en ordtaxonomi på traditionell Text-Based Image Search och Content-Based Image Search. Genom att applicera en ordtaxonomi på olika datakällor kan ett starkt ordfilter samt en modul som förlänger ordlistor skapas och testas. Resultaten pekar på att beroende på implementationen så kan antingen precisionen eller återkallningen förbättras. Genom att använda en liknande metod i ett verkligt scenario är det därför möjligt att flytta bilder med hög precision längre fram i resultatlistan och samtidigt behålla hög återkallning, och därmed öka den upplevda relevansen i bildsök.

The objective of this work is object retrieval in large scale image datasets, where the object is specified by an image query and retrieval should be immediate at run time. Such a system has a wide variety of applications including object or location recognition, video search, near duplicate detection and 3D reconstruction. The task is very challenging because of large variations in the imaged object appearance due to changes in lighting conditions, scale and viewpoint, as well as partial occlusions. A starting point of established systems which tackle the same task is detection of viewpoint invariant features, which are then quantized into visual words and efficient retrieval is performed using an inverted index. We make the following three improvements to the standard framework: (i) a new method to compare SIFT descriptors (RootSIFT) which yields superior performance without increasing processing or storage requirements; (ii) a novel discriminative method for query expansion; (iii) a new feature augmentation method. Scaling up to searching millions of images involves either distributing storage and computation across many computers, or employing very compact image representations on a single computer combined with memory-efficient approximate nearest neighbour search (ANN). We take the latter approach and improve VLAD, a popular compact image descriptor, using: (i) a new normalization method to alleviate the burstiness effect; (ii) vocabulary adaptation to reduce influence of using a bad visual vocabulary; (iii) extraction of multiple VLADs for retrieval and localization of small objects. We also propose a method, SCT, for extremely low bit-rate compression of descriptor sets in order to reduce the memory footprint of ANN. The problem of finding images of an object in an unannotated image corpus starting from a textual query is also considered. Our approach is to first obtain multiple images of the queried object using textual Google image search, and then use these images to visually query the target database. We show that issuing multiple queries significantly improves recall and enables the system to find quite challenging occurrences of the queried object. Current retrieval techniques work only for objects which have a light coating of texture, while failing completely for smooth (fairly textureless) objects best described by shape. We present a scalable approach to smooth object retrieval and illustrate it on sculptures. A smooth object is represented by its imaged shape using a set of quantized semi-local boundary descriptors (a bag-of-boundaries); the representation is suited to the standard visual word based object retrieval. Furthermore, we describe a method for automatically determining the title and sculptor of an imaged sculpture using the proposed smooth object retrieval system.

The thesis deals with the detection and classification of flying objects. The work can be divided into three parts. The first part describes the creation of dataset of flying objects. The reverse image search is used to create the dataset. The next part is a research of algorithms for detection, tracking and classification. Subsequently, the individual algorithms are applied and evaluated. In the last part, the design of hardware components is performed.

No description available.

No description available.

Cette thèse aborde le problème de la recherche d'images à grande échelle. Pour aborder la recherche d'images à grande échelle, il est nécessaire de coder des images avec des représentations compactes qui peuvent être efficacement utilisées pour comparer des images de manière significative. L'obtention d'une telle représentation compacte peut se faire soit en comprimant des représentations efficaces de grande dimension, soit en apprenant des représentations compactes de bout en bout. Le travail de cette thèse explore et avance dans ces deux directions. Dans notre première contribution, nous étendons les approches de quantification vectorielle structurée telles que la quantification de produit en proposant une représentation somme pondérée de codewords. Nous testons et vérifions les avantages de notre approche pour la recherche approximative du plus proche voisin sur les caractéristiques d'image locales et globales, ce qui est un moyen important d'aborder la recherche d'images à grande échelle. L'apprentissage de la représentation compacte pour la recherche d'images a récemment attiré beaucoup d'attention avec diverses approches basées sur le hachage profond proposées. Dans de telles approches, les réseaux de neurones convolutifs profonds apprennent à coder des images en codes binaires compacts. Dans cette thèse, nous proposons une approche d'apprentissage supervisé profond pour la représentation binaire structurée qui rappelle une approche de quantification vectorielle structurée telle que PQ. Notre approche bénéficie de la recherche asymétrique par rapport aux approches de hachage profond et apporte une nette amélioration de la précision de la recherche au même débit binaire. L'index inversé est une autre partie importante du système de recherche à grande échelle en dehors de la représentation compacte. À cette fin, nous étendons nos idées pour l'apprentissage de la représentation compacte supervisée pour la construction d'index inversés. Dans ce travail, nous abordons l'indexation inversée avec un apprentissage approfondi supervisé et essayons d'unifier l'apprentissage de l'indice inversé et de la représentation compacte. Nous évaluons minutieusement toutes les méthodes proposées sur divers ensembles de données accessibles au public. Nos méthodes surpassent ou sont compétitives avec l'état de l'art. / This thesis addresses the problem of large-scale image search. To tackle image search at large scale, it is required to encode images with compact representations which can be efficiently employed to compare images meaningfully. Obtaining such compact representation can be done either by compressing effective high dimensional representations or by learning compact representations in an end-to-end manner. The work in this thesis explores and advances in both of these directions. In our first contribution, we extend structured vector quantization approaches such as Product Quantization by proposing a weighted codeword sum representation. We test and verify the benefits of our approach for approximate nearest neighbor search on local and global image features which is an important way to approach large scale image search. Learning compact representation for image search recently got a lot of attention with various deep hashing based approaches being proposed. In such approaches, deep convolutional neural networks are learned to encode images into compact binary codes. In this thesis we propose a deep supervised learning approach for structured binary representation which is a reminiscent of structured vector quantization approaches such as PQ. Our approach benefits from asymmetric search over deep hashing approaches and gives a clear improvement for search accuracy at the same bit-rate. Inverted index is another important part of large scale search system apart from the compact representation. To this end, we extend our ideas for supervised compact representation learning for building inverted indexes. In this work we approach inverted indexing with supervised deep learning and make an attempt to unify the learning of inverted index and compact representation. We thoroughly evaluate all the proposed methods on various publicly available datasets. Our methods either outperform, or are competitive with the state-of-the-art.