As Machine Learning (ML) methods have gained traction in recent years, someproblems regarding the construction of such methods have arisen. One such problem isthe collection and labeling of data sets. Specifically when it comes to many applicationsof Computer Vision (CV), one needs a set of images, labeled as either being of someclass or not. Creating such data sets can be very time consuming. This project setsout to tackle this problem by constructing an end-to-end system for searching forobjects in images (i.e. an Object Based Image Retrieval (OBIR) method) using an objectdetection framework (You Only Look Once (YOLO) [16]). The goal of the project wasto create a method that; given an image of an object of interest q, search for that sameor similar objects in a set of other images S. The core concept of the idea is to passthe image q through an object detection model (in this case YOLOv5 [16]), create a”fingerprint” (can be seen as a sort of identity for an object) from a set of feature mapsextracted from the YOLOv5 [16] model and look for corresponding similar parts of aset of feature maps extracted from other images. An investigation regarding whichvalues to select for a few different parameters was conducted, including a comparisonof performance for a couple of different similarity metrics. In the table below,the parameter combination which resulted in the highest F_Top_300-score (a measureindicating the amount of relevant images retrieved among the top 300 recommendedimages) in the parameter selection phase is presented. Layer: 23Pool Methd: maxSim. Mtrc: eucFP Kern. Sz: 4 Evaluation of the method resulted in F_Top_300-scores as can be seen in the table below. Mouse: 0.820Duck: 0.640Coin: 0.770Jet ski: 0.443Handgun: 0.807Average: 0.696 / Medan ML-metoder har blivit mer populära under senare år har det uppstått endel problem gällande konstruktionen av sådana metoder. Ett sådant problem ärinsamling och annotering av data. Mer specifikt när det kommer till många metoderför datorseende behövs ett set av bilder, annoterande att antingen vara eller inte varaav en särskild klass. Att skapa sådana dataset kan vara väldigt tidskonsumerande.Metoden som konstruerades för detta projekt avser att bekämpa detta problem genomatt konstruera ett end-to-end-system för att söka efter objekt i bilder (alltså en OBIR-metod) med hjälp av en objektdetekteringsalgoritm (YOLO). Målet med projektet varatt skapa en metod som; givet en bild q av ett objekt, söka efter samma eller liknandeobjekt i ett bibliotek av bilder S. Huvudkonceptet bakom idén är att köra bilden qgenom objektdetekteringsmodellen (i detta fall YOLOv5 [16]), skapa ett ”fingerprint”(kan ses som en sorts identitet för ett objekt) från en samling feature maps extraheradefrån YOLOv5-modellen [16] och leta efter liknande delar av samlingar feature maps iandra bilder. En utredning angående vilka värden som skulle användas för ett antalolika parametrar utfördes, inklusive en jämförelse av prestandan som resultat av olikalikhetsmått. I tabellen nedan visas den parameterkombination som gav högst F_Top_300(ett mått som indikerar andelen relevanta bilder bland de 300 högst rekommenderadebilderna). Layer: 23Pool Methd: maxSim. Mtrc: eucFP Kern. Sz: 4 Evaluering av metoden med parameterval enligt tabellen ovan resulterade i F_Top_300enligt tabellen nedan. Mouse: 0.820Duck: 0.640Coin: 0.770Jet ski: 0.443Handgun: 0.807Average: 0.696

This thesis focuses on four main research themes namely shot boundary detection, fast frame alignment, activity-driven video summarisation, and highlights based video annotation and retrieval. A number of novel algorithms have been proposed to address these issues, which can be highlighted as follows. Firstly, accurate and robust shot boundary detection is achieved through modelling of cuts into sub-categories and appearance based modelling of several gradual transitions, along with some novel features extracted from compressed video. Secondly, fast and robust frame alignment is achieved via the proposed subspace phase correlation (SPC) and an improved sub-pixel strategy. The SPC is proved to be insensitive to zero-mean-noise, and its gradient-based extension is even robust to non-zero-mean noise and can be used to deal with non-overlapped regions for robust image registration. Thirdly, hierarchical modelling of rush videos using formal language techniques is proposed, which can guide the modelling and removal of several kinds of junk frames as well as adaptive clustering of retakes. With an extracted activity level measurement, shot and sub-shot are detected for content-adaptive video summarisation. Fourthly, highlights based video annotation and retrieval is achieved, in which statistical modelling of skin pixel colours, knowledge-based shot detection, and improved determination of camera motion patterns are employed. Within these proposed techniques, one important principle is to integrate various kinds of feature evidence and to incorporate prior knowledge in modelling the given problems. High-level hierarchical representation is extracted from the original linear structure for effective management and content-based retrieval of video data. As most of the work is implemented in the compressed domain, one additional benefit is the achieved high efficiency, which will be useful for many online applications.

隨著數位多媒體技術的進步，越來越多的音樂以數位化的方式來儲存，數位音樂的檢索成為重要的研究領域之一。以內容為主的音樂檢索(Content-Based Music Retrieval, CBMR)能讓使用者直接利用音樂的內容做檢索，而非傳統以音樂的metadata查詢的方法。目前有關CBMR的研究，常見的查詢方式包括哼歌、唱歌或打拍子等。但是，這些方法都會因為查詢者缺乏音樂訓練而無法正確表達出想查詢的音樂，影響查詢效果。 人們常常會根據曲風將音樂分類，音樂曲風的探勘將有助於CBMR的研究。本篇論文主要目的在結合多媒體與資料探勘的技術，從大量MIDI音樂中，作音樂曲風的探勘及分類，並將曲風探勘的技術應用在個人化音樂推薦、音樂風格檢索及音樂風格瀏覽上。 在本論文的第一部份，音樂曲風探勘分類的研究，包括了三個研究議題：音樂特徵的粹取、頻繁樣式的探勘及曲風的分類。我們利用和絃作為音樂的特徵，根據和聲學的原理，從MIDI音樂中找出主旋律搭配的和絃。粹取出和絃後，我們研究不同的和絃特徵表示法與其頻繁樣式探勘演算法。針對所探勘出的頻繁樣式，我們修改associated classification演算法，以應用在音樂曲風的分類上。此外，不同的曲風，其風格的多樣性也不同。因此，為了提高分類的效果，我們提出Single-Type Variant-Support (STVS) 與Multi-Type Variant-Support (MTVS) classification演算法，使得分類規則中允許多種特徵表示與不同的最小支持度。 在本篇論文的第二部分，我們應用曲風探勘的技術，提出了個人化音樂推薦的機制。針對使用者對音樂風格的喜好，將新的音樂推薦給使用者。系統根據使用者對資料庫中音樂的存取行為，學習使用者在音樂曲風上的偏好，產生個人化的2-way preference classifier，以推薦符合使用者喜好的音樂。 第三部分為音樂曲風的檢索。目前大部分的CBMR系統中，使用者僅能尋找已經聽過的音樂。然而，使用者想查詢的很可能是沒聽過，但曲風感覺類似的音樂。針對上述的問題，我們提出了以音樂曲風作檢索的新方法。同時，我們提出四種曲風查詢的描述方式，並且利用音樂風格探勘與分類的技術產生的分類規則計算曲風的相似度，最後依照曲風的相似程度產生檢索結果。 本篇論文的最後一部分為音樂風格的分群。音樂風格的分群有助於瀏覽大量的音樂資料。我們利用和絃為特徵，針對不同的特徵表示方法，提出相似度的計算方式。我們將數種分群演算法應用於音樂風格的分群上，並比較各種分類演算法與不同的音樂特徵與表示法的分群效果。 / With the development of multimedia technology, digital music is now in widespread use. Content-based music retrieval (CBMR) has attracted much interest in recent years. CBMR allows users query by music content rather than metadata. However, even with the capability of query by humming, the effectiveness of CBMR system suffers from the ability of query content expression for people without music training. Music style is one of the features that people used to classify music. Discovery of music style is helpful for the design of content-based music retrieval systems. In this thesis, we investigate the mining techniques of music style by melody from a collection of MIDI music and apply the mining techniques to three applications, personalized music filtering, music retrieval by melody style and music style browsing. In the first part, the design issues of melody style mining and classification consist of the feature extraction, frequent pattern mining and melody style classification. We extracted the chord from the melody based on the harmony and investigated the representation of extracted features. For each extracted feature, the corresponding frequent pattern mining techniques are developed. For the melody style classification algorithm, we propose the Single-Type Uniform-Support classification (STUS) algorithm which is modified from the associated classification algorithm. To improve the performance of classification, we propose two new classification algorithms - Single-Type Variant-Support Classification (STVS) and Multi-Type Variant- Support classification (MTVS) algorithm. STVS learns the appropriate minimum supports of every category’s frequent patterns automatically. MTVS algorithm considers all types of frequent patterns for every category further and can decide the appropriate combination of frequent patterns and the corresponding minimum supports. In the second part, we present a personalized content-based music filtering system to support music recommendation based on user’s preference of melody style. The system learns the user preference by mining the melody patterns from the music access behavior of the user. A two-way melody preference classifier is therefore constructed for each user. Music recommendation is made through this melody preference classifier. Performance evaluation showed that the filtering effect of the proposed approach meets user’s preference. A new approach for CBMR by the semantic property of music – melody style is proposed in the third part of this thesis. Most CBMR systems provide users the capability to look for music that has been heard. However, sometimes, listeners are looking, not for something they already know, but for something new. Moreover, people sometimes wish to retrieve music that “feels like” another music object or a music style. We propose four types of query specification for melody style query. The output of the melody style query is a music list ranked by the degree of relevance to the query. We adopted melody style mining and classification rule learning algorithm to obtain style classification rules. The style ranking is determined by the style classification rules. In this thesis, we also investigate music clustering techniques which are useful to browse large music archives. We propose the similarity measures for the representation of the extracted chord-sets and compared the performance of different clustering algorithms with various extracted features.

數位化的今天，電影逐漸成為人們日常生活的一部份，電影資料的內涵式分析也成為目前重要的研究主題。透過電影拍攝手法，我們知道電影視聽覺特徵與情緒之間有密不可分的關係。因此，在本研究中，我們希望利用探勘電影視聽覺特徵與情緒的關聯來達到自動判斷電影場景的情緒。 首先，先由人工標記訓練場景的情緒，之後，我們對所有的場景擷取定義的六類特徵值。特徵值包括電影場景的顏色、燈光、影片速度、特寫鏡頭、聲音和字幕六類。最後，我們利用Mixed Media Graph演算法來探勘場景情緒與特徵值之間的關聯，達到自動判斷電影場景情緒的功能。實驗結果顯示，準確率最高可達到70%。 / Movies play an important role in our life nowadays. How to analyze the emotional content of movies becomes one of the major issues. Based on film grammar, there are many audiovisual cues in movies helpful for detecting the emotions of scenes. In this research, we investigate the discovery of the relationship between audiovisual cues and emotions of scenes and the automatic emotion annotation of scenes is achieved. First, the training scenes are labeled with the emotions manually. Second, six classes of audiovisual features are extracted from all scenes. These classes of features consist of color, light, tempo, close-up, audio, and textual. Finally, the graph-based approach, Mixed Media Graph is modified to mine the association between audiovisual features and emotions of the scenes. The experiments show that the accuracy achieves 70%.

Depuis 2002 en Asie, 2005 en Europe et aux États-Unis, des éditeurs de presse écrite et des annonceurs proposent à leur lectorat équipé en téléphone portable ou en ordiphone d’accéder à des contenus et services numériques via ceux imprimés dans le journal. Ces nouvelles formes de communications brouillent les relations traditionnelles entre éditeurs de presse, annonceurs et lectorat, également mobinaute. Les enjeux techniques et politiques de ces services de communication s’étudient simultanément à trois niveaux. Le premier, physique, se situe dans la combinaison entre le papier et le terminal électronique de l’individu relié au réseau. Le deuxième, logique, se situe dans la forme (privée ou publique) de ce qui est imprimé sur le papier (langage humain ou machine) et la méthode logicielle mise en oeuvre (lecture locale ou à distance) de manière synchrone (presse augmentée) ou asynchrone (code graphique, recherche ou identification par l’image). Le troisième niveau, des contenus, consiste à s’interroger sur le sens de la surimpression de contenus et de services numériques à l’écran de l’ordiphone suivant ce qui est imprimé dans le journal. Notre étude aura par ailleurs consisté à mettre en lumière les régimes juridiques des méthodes logicielles basées d’une part sur le contenu et régies par le droit de la propriété intellectuelle, et basées d’autre part sur un langage informatique et régies par le droit des liens hypertextes. Avec un ordiphone, un titre de presse évolue d’un support d’information à un canal de communications électroniques où le contexte et l’identité de l’individu jouent un rôle innovant dans la transmission et l’accès à l’information. / Since 2002 in Asia, and 2005 in Europe and the United States, press editors and advertisers have offered access to digital content and services to readers that possess a smartphone. These new forms of communication blur the traditional relationships between press editors, advertisers and readers. To study the technical and political elements concerning these new communication services, we rely on the three-layer network approach: the physical, the logical and the content infrastructure layer. The physical layer is located between the sheet of paper and the electronic device that links the user to a network. The logical layer corresponds to the form (private or public) of what is printed on the page (i.e., human readable or machine language) and the software employed (i.e., local or distant reading), be it synchronously (augmented press) or asynchronously (graphic codes, image search or image identification). The third layer concerns the way the content from the printed page are presented on the screen of the smartphone. Our study equally aims to shed light on the current legal translations of software methods: when based on content, the methods are seen to concern intellectual property law; whereas when based on computer language they concern the laws on hyperlinks. With smartphones, a printed edition evolves from an information medium to an electronic communication channel where the context and the identity of the user play an innovative role in the transmission and access to information.

Taxonomic classification has always been important to the study of any biological system. Many biological species will go unclassified and become lost forever at the current rate of classification. The current state of computer technology makes image storage and retrieval possible on a global level. As a result, computer-aided taxonomy is now possible. Content based image retrieval techniques utilize visual features of the image for classification. By utilizing image content and computer technology, the gap between taxonomic classification and species destruction is shrinking. This content based study utilizes the Fourier Descriptors of fifteen known landmark features on three Carpiodes species: C.carpio, C.velifer, and C.cyprinus. Classification analysis involves both unsupervised and supervised machine learning algorithms. Fourier Descriptors of the fifteen known landmarks provide for strong classification power on image data. Feature reduction analysis indicates feature reduction is possible. This proves useful for increasing generalization power of classification.

[pt] Quando entregamos e apresentamos aplicações multimídia por meio de uma rede de comunicação, a latência de exibição pode representar um fator central e crítico que afeta a qualidade da apresentação multimídia. Na entrega de uma apresentação multimídia de boa qualidade o sincronismo é prevalecido, consequentemente, os conteúdos são exibidos de forma contínua, conforme as especificações do autor da aplicação. Nesta tese, um plano de pré-busca de conteúdos multimídia é proposto com o intuito de reduzir a latência de exibição e garantir o sincronismo entre os objetos de mídia que fazem parte da apresentação multimídia. O mecanismo proposto considera as aplicações multimídia desenvolvidas na linguagem declarativa NCL e utiliza a vantagem do sincronismo estar baseado em eventos, na determinação da ordem adequada de recuperação dos diferentes objetos de mídia e no cálculo dos seus tempos de início de recuperação. Aspectos importantes a serem considerados em um ambiente de pré-busca são levantados e os diferentes algoritmos que compõem o plano de pré-busca são desenvolvidos. / [en] When delivering and presenting multimedia applications through a communication network, the presentation lag could be a major and critical factor affecting the multimedia presentation quality. In a good quality presentation the synchronism is always preserved, hence all the contents are presented in a continue way according to the authoring specifications. In this dissertation, a multimedia content prefetching plan is proposed in order to minimize the presentation lag and guarantee the synchronism between the media objects, which constitute the multimedia application. The proposed mechanism regards the multimedia applications developed using the NCL declarative language and it uses the events based synchronism advantage to determinate the ideal retrieval order of the media objects and to calculate their start retrieval times. Furthermore, important issues to be considered in a prefetch ambient are raised and the different algorithms that belong to the prefetching plan are developed.

Neste trabalho são apresentados dois novos descritores de forma para tarefas de recuperação de imagens por conteúdo (CBIR) e análise de formas, que são construídos sobre uma extensão do conceito de tensor scale baseada na Transformada de Distância Euclidiana (EDT). Primeiro, o algoritmo de tensor scale é utilizado para extrair informações da forma sobre suas estruturas locais (espessura, orientação e anisotropia) representadas pela maior elipse contida em uma região homogênea centrada em cada pixel da imagem. Nos novos descritores, o limite do intervalo das orientações das elipses do modelo de tensor scale é estendido de 180º para 360º, de forma a melhor discriminar a descrição das estruturas locais. Então, com base em diferentes abordagens de amostragem, visando resumir informações mais relevantes, os novos descritores são construídos. No primeiro descritor proposto, Tensor Scale Sector (TSS), a distribuição das orientações relativas das estruturas locais em setores circulares é utilizada para compor um vetor de características de tamanho fixo, para uma caracterização de formas baseada em região. No segundo descritor, o Tensor Scale Band (TSB), foram considerados histogramas das orientações relativas extraídos de bandas concêntricas, formando também um vetor de características de tamanho fixo, com uma função de distância de tempo linear. Resultados experimentais com diferentes bases de formas (MPEG-7 e MNIST) são apresentados para ilustrar e validar os métodos. TSS demonstra resultados comparáveis aos métodos estado da arte, que geralmente dependem de algoritmos custosos de otimização de correspondências. Já o TSB, com sua função de distância em tempo linear, se demonstra como uma solução adequada para grandes coleções de formas. / In this work, two new shape descriptors are proposed for tasks in Content-Based Image Retrieval (CBIR) and Shape Analysis tasks, which are built upon an extended tensor scale based on the Euclidean Distance Transform (EDT). First, the tensor scale algorithm is applied to extract shape attributes from its local structures (thickness, orientation, and anisotropy) as represented by the largest ellipse within a homogeneous region centered at each image pixel. In the new descriptors, the upper limit of the interval of local orientation of tensor scale ellipses is extended from 180º to 360º, to better discriminate the description of local structures. Then, the new descriptors are built based on different sampling approaches, aiming to summarize the most relevant features. In the first proposed descriptor, Tensor Scale Sector descriptor (TSS), the local distributions of relative orientations within circular sectors are used to compose a fixed-length feature vector, for a region-based shape characterization. For the second method, the Tensor Scale Band (TSB) descriptor, histograms of relative orientations are considered for each circular concentric band, to also compose a fixed-length feature vector, with linear time distance function for matching. Experimental results for different shape datasets (MPEG-7 and MNIST) are presented to illustrate and validate the methods. TSS can achieve high retrieval values comparable to state-of-the-art methods, which usually rely on time-consuming correspondence optimization algorithms, but uses a simpler and faster distance function, while the even faster linear complexity of TSB leads to a suitable solution for very large shape collections.

Com o crescente aumento no número de imagens geradas em mídias digitais surgiu a necessidade do desenvolvimento de novas técnicas de recuperação desses dados. Um critério de busca que pode ser utilizado na recuperação das imagens é o da dissimilaridade, no qual o usuário deseja recuperar as imagens semelhantes à uma imagem de consulta. Para a realização das consultas são empregados vetores de características extraídos das imagens e funções de distância para medir a dissimilaridade entre pares desses vetores. Infelizmente, a busca por conteúdo de imagens em consultas simples tende a gerar resultados que não correspondem ao interesse do usuário misturados aos resultados significativos encontrados, pois em geral há uma descontinuidade semântica entre as características extraídas automaticamente e a subjetividade da interpretação humana. Com o intuito de tratar esse problema, diversos métodos foram propostos para a diminuição da descontinuidade semântica. O foco principal desta tese é o desenvolvimento de métodos escaláveis para a redução da descontinuidade semântica em sistemas recuperação de imagens por conteúdo em tempo real. Nesta sentido, são apresentados: a formalização de consultas por similaridade que permitem a utilização de múltiplos centros de consulta em espaços métricos como base para métodos de realimentação de relevância; um método exato para otimização dessas consultas nesses espaços; e um modelo para tratamento da diversidade em consultas por similaridade e heurísticas para sua otimização / The increasing number of images captured in digital media fostered the developmet of new methods for the recovery of these images. Dissimilarity is a criteria that can be used for image retrieval, where the results are images that are similar to a given reference. The queries are based on feature vectors automatically extracted from the images and on distance functions to measure the dissimilarity between pair of vectors. Unfortunately, the search for images in simple queries may result in images that do not fulfill the user interest together with meaningful images, due to the semantic gap between the image features and to the subjectivity of the human interpretation. This problem leaded to the development of many methods to deal with the semantic gap. The focus of this thesis is the development of scalable methods aiming the semantic gap reduction in real time for content-based image retrieval systems. For this purpose, we present the formal definition of similarity queries based on multiple query centers in metric spaces to be used in relevance feedback methods, an exact method to optimize these queries and a model to deal with diversity in nearest neighbor queries including heuristics for its optimization

O hipocampo é uma estrutura cerebral que possui importância primordial para o sistema de memória humana. Alterações no seus tecidos levam a doenças neurodegenerativas, tais como: epilepsia, esclerose múltipla e demência, entre outras. Para medir a atrofia do hipocampo é necessário isolá-lo do restante do cérebro. A separação do hipocampo das demais partes do cérebro ajuda aos especialistas na análise e o entendimento da redução de seu volume e detecção de qualquer anomalia presente. A extração do hipocampo é principalmente realizada de modo manual, a qual é demorada, pois depende da interação do usuário. A segmentação automática do hipocampo é investigada como uma alternativa para contornar tais limitações. Esta dissertação de mestrado apresenta um novo método de segmentação automático, denominado Modelo de Nuvem de Similaridade (Similarity Cloud Model - SimCM). O processo de segmentação é dividido em duas etapas principais: i) localização por similaridade e ii) ajuste de nuvem. A primeira operação utiliza a nuvem para localizar a posição mais provável do hipocampo no volume destino. A segunda etapa utiliza a nuvem para corrigir o delineamento final baseada em um novo método de cálculo de readequação dos pesos das arestas. Nosso método foi testado em um conjunto de 235 MRI combinando imagens de controle e de pacientes com epilepsia. Os resultados alcançados indicam um rendimento superior tanto em efetividade (qualidade da segmentação) e eficiência (tempo de processamento), comparado com modelos baseados em grafos e com modelos Bayesianos. Como trabalho futuro, pretendemos utilizar seleção de características para melhorar a construção da nuvem e o delineamento dos tecidos / The hippocampus is a particular structure that plays a main role in human memory systems. Tissue modifications of the hippocampus lead to neurodegenerative diseases as epilepsy, multiple sclerosis, and dementia, among others. To measure hippocampus atrophy, it is crucial to get its isolated representation from the whole brain volume. Separating the hippocampus from the brain helps physicians in better analyzing and understanding its volume reduction, and detecting any abnormal behavior. The extraction of the hippocampus is dominated by manual segmentation, which is time consuming mainly because it depends on user interaction. Therefore, automatic segmentation of the hippocampus has being investigated as an alternative solution to overcome such limitations. This master dissertation presents a new automatic segmentation method called Similarity Cloud Model (SimCM) based on hippocampus feature extraction. The segmentation process consists of two main operations: i) localization by similarity, and ii) cloud adjustment. The first operation uses the cloud to localize the most probable position of the hippocampus in a target volume. The second process invokes the cloud to correct the final labeling, based on a new method for arc-weight re-adjustment. Our method has been tested in a dataset of 235 MRIs combining healthy and epileptic patients. Results indicate superior performance, in terms of effectiveness (segmentation quality) and efficiency (processing time), in comparison with similar graph-based and Bayesian-based models. As future work, we intend to use feature selection to improve cloud construction and tissue delineation