Music mood recognition by machine continues to attract attention from both academia and industry. This thesis explores the hypothesis that the music emotion problem is circular, and is a primary step in determining the efficacy of circular regression as a machine learning method for automatic music mood recognition. This hypothesis is tested through experiments conducted using instances of the two commonly accepted models of affect used in machine learning (categorical and two-dimensional), as well as on an original circular model proposed by the author. Polygonal approximations of circular regression are proposed as a practical way to investigate whether the circularity of the annotations can be exploited. An original dataset assembled and annotated for the models is also presented. Next, the architecture and implementation choices of all three models are given, with an emphasis on the new polygonal approximations of circular regression. Experiments with different polygons demonstrate consistent and in some cases significant improvements over the categorical model on a dataset containing ambiguous extracts (ones for which the human annotators did not fully agree upon). Through a comprehensive analysis of the results, errors and inconsistencies observed, evidence is provided that mood recognition can be improved if approached as a circular problem. Finally, a proposed multi-tagging strategy based on the circular predictions is put forward as a pragmatic method to automatically annotate music based on the circular model. / Graduate / 0984 / 0800 / 0413 / zazz101@hotmail.com

L'apprentissage machine (AM) est un outil important dans le domaine de la recherche d'information musicale (Music Information Retrieval ou MIR). De nombreuses tâches de MIR peuvent être résolues en entraînant un classifieur sur un ensemble de caractéristiques. Pour les tâches de MIR se basant sur l'audio musical, il est possible d'extraire de l'audio les caractéristiques pertinentes à l'aide de méthodes traitement de signal. Toutefois, certains aspects musicaux sont difficiles à extraire à l'aide de simples heuristiques. Afin d'obtenir des caractéristiques plus riches, il est possible d'utiliser l'AM pour apprendre une représentation musicale à partir de l'audio. Ces caractéristiques apprises permettent souvent d'améliorer la performance sur une tâche de MIR donnée. Afin d'apprendre des représentations musicales intéressantes, il est important de considérer les aspects particuliers à l'audio musical dans la conception des modèles d'apprentissage. Vu la structure temporelle et spectrale de l'audio musical, les représentations profondes et multiéchelles sont particulièrement bien conçues pour représenter la musique. Cette thèse porte sur l'apprentissage de représentations de l'audio musical. Des modèles profonds et multiéchelles améliorant l'état de l'art pour des tâches telles que la reconnaissance d'instrument, la reconnaissance de genre et l'étiquetage automatique y sont présentés. / Machine learning (ML) is an important tool in the field of music information retrieval (MIR). Many MIR tasks can be solved by training a classifier over a set of features. For MIR tasks based on music audio, it is possible to extract features from the audio with signal processing techniques. However, some musical aspects are hard to extract with simple heuristics. To obtain richer features, we can use ML to learn a representation from the audio. These learned features can often improve performance for a given MIR task. In order to learn interesting musical representations, it is important to consider the particular aspects of music audio when building learning models. Given the temporal and spectral structure of music audio, deep and multi-scale representations are particularly well suited to represent music. This thesis focuses on learning representations from music audio. Deep and multi-scale models that improve the state-of-the-art for tasks such as instrument recognition, genre recognition and automatic annotation are presented.

En aquesta tesi presentem un mètode general per al reconeixement automàtic d’instruments musicals partint d’un senyal d’àudio. A diferència de molts enfocs relacionats, el nostre evita restriccions artificials o artificioses pel que fa al disseny algorísmic, les dades proporcionades al sistema, o el context d’aplicació. Per tal de fer el problema abordable, limitem el procés a l’operació més bàsica consistent a reconèixer l’instrument predominant en un breu fragment d’àudio. Així ens estalviem la separació de fonts sonores en la mescla i, més específicament, predim una font sonora a partir del timbre general del so analitzat. Per tal de compensar aquesta restricció incorporem, addicionalment, informació derivada d’una anàlisi musical jeràrquica: primer incorporem context temporal a l’hora d’extraure etiquetes dels instruments, després incorporem aspectes formals de la peça que poden ajudar al reconeixement de l’instrument, i finalment incloem informació general gràcies a l’explotació de les associacions entre gèneres musicals i instruments. / In this dissertation we present a method for the automatic recognition of musical instruments from music audio signal. Unlike most related approaches, our specific conception mostly avoids laboratory constraints on the method’s algorithmic design, its input data, or the targeted application context. To account for the complex nature of the input signal, we limit the basic process in the processing chain to the recognition of a single predominant musical instrument from a short audio fragment. We thereby prevent resolving the mixture and rather predict one source from the timbre of the sound. To compensate for this restriction we further incorporate information derived from a hierarchical music analysis; we first incorporate musical context to extract instrumental labels from the time-varying model decisions. Second, the method incorporates information regarding the piece’s formal aspects into the process. Finally, we include information from the collection level by exploiting associations between musical genres and instrumentations.

Submitted by Johnny Rodrigues (johnnyrodrigues@ufcg.edu.br) on 2018-04-09T19:37:26Z No. of bitstreams: 1 ANDRYW MARQUES RAMOS - DISSERTAÇÃO PPGCC 2014..pdf: 16053506 bytes, checksum: fdeece58c13c7b38ceb6c8b06f9d516b (MD5) / Made available in DSpace on 2018-04-09T19:37:26Z (GMT). No. of bitstreams: 1 ANDRYW MARQUES RAMOS - DISSERTAÇÃO PPGCC 2014..pdf: 16053506 bytes, checksum: fdeece58c13c7b38ceb6c8b06f9d516b (MD5) Previous issue date: 2014-09-05 / Abuscapornovidadesmusicais,sejamelasmúsicas,álbunsouartistas,éumaspectocentral no hábito das pessoas quando se trata de música. E esta procura aumentou principalmente porcausadagrandequantidadedemúsicadisponívelecomfácilacessoproporcionadopelo avanço de tecnologias como Last.FM, Spotify, Youtube, Itunes, entre outros. Porém, devido a esta grande disponibilidade, nem sempre é fácil a descoberta de novidades que sejam relevantes. Para resolver este problema, muitos esforços foram elaborados. O presente trabalho tenta expandir estes esforços tratando a novidade de maneira multidimensional, de acordo com dois aspectos: familiaridade (o quanto o ouvinte conhece outras músicas/ artistas similares à novidade) e popularidade (o quão essa música / artista é conhecida pelos ouvintes em geral). Esta visão multidimensional da novidade é uma visão mais rica e pode aperfeiçoar ferramentas que dão suporte a descoberta de novidades para ouvintes, como sistemas de recomendação, sites, fóruns, etc. Desta maneira analisamos as preferências dos ouvintes por artistas com novidade (artistas que nunca foram escutados anteriormente pelo ouvinte) baseadas nestes dois aspectos. Para isso foi estudado os hábitos de escuta dos usuários do Last.FM, rede social musical que registra o que os usuários escutam. Os resultados sugerem que não existe uma preferência geral dos ouvintes po ralguma specto das novidades. Os ouvintes tendem a formar grupos baseados nas preferências pelos aspectos das novidades. Estes resultados sugerem um tratamento específico para estes grupos de ouvintes, como um sistema de recomendação que leve em conta estas preferências. Outro estudo realizado neste trabalho compara as preferências dos ouvintes pelos aspectos tanto dos artistas com novidade quanto dos artistas já conhecidos. Este estudo apontou que as preferências dos ouvintes para estes dois âmbitos são diferentes, onde os ouvintes tendem a formar grupos baseados nestas diferentes preferências. Este resultado implica que o âmbito das novidades e o âmbito do que já se conhece não deve ser tratado da mesma maneira. / The search for new music, e.g. songs tracks, albums or artists, is a central aspect in the people’s listening habit. And this pursuit increased because of the large amount of available music and the easy access provided by the advance of technologies like Last.FM, Sportify, Youtube, Itunes. However, due to this high music availability, it is not always easy to discover relevant novelties. This study attempts to expand the studies about music novelties by investigating how the music preferences of listeners are affected by two different aspects of novel artists: familiarity (how much the listener knows other artists similar to the novelty) and popularity (how this artist is known by listeners in general). The study supports this multidimensional view of novelty, which is a richer view and it enables the improvement of tools that support the discovery of music novelties for listeners, as recommender systems, websites,forums,etc. WecollectedandanalyzedhistoricaldatafromLast.fmusers,apopular online music discovery service. The results suggest that there is not a general preference for some aspect of novelty. Listeners tend to form groups based on the preferences for the novelty aspects. These results suggest a specific treatment for these groups of listeners, e.g., a recommendation system considering these preferences. Another study performed compares the listeners preferences by aspects of both novelty artists and artists already known. This study showed that the listeners preferences for these two spheres are different, where listeners tend to form groups based on these different preferences. This result implies that the scope of novelty and the scope of what is already known should not be treated the same way.

Ce mémoire de thèse de doctorat présente, discute et propose des outils de fouille automatique de mégadonnées dans un contexte de classification supervisée musical.L'application principale concerne la classification automatique des thèmes musicaux afin de générer des listes de lecture thématiques.Le premier chapitre introduit les différents contextes et concepts autour des mégadonnées musicales et de leur consommation.Le deuxième chapitre s'attelle à la description des bases de données musicales existantes dans le cadre d'expériences académiques d'analyse audio.Ce chapitre introduit notamment les problématiques concernant la variété et les proportions inégales des thèmes contenus dans une base, qui demeurent complexes à prendre en compte dans une classification supervisée.Le troisième chapitre explique l'importance de l'extraction et du développement de caractéristiques audio et musicales pertinentes afin de mieux décrire le contenu des éléments contenus dans ces bases de données.Ce chapitre explique plusieurs phénomènes psychoacoustiques et utilise des techniques de traitement du signal sonore afin de calculer des caractéristiques audio.De nouvelles méthodes d'agrégation de caractéristiques audio locales sont proposées afin d'améliorer la classification des morceaux.Le quatrième chapitre décrit l'utilisation des caractéristiques musicales extraites afin de trier les morceaux par thèmes et donc de permettre les recommandations musicales et la génération automatique de listes de lecture thématiques homogènes.Cette partie implique l'utilisation d'algorithmes d'apprentissage automatique afin de réaliser des tâches de classification musicale.Les contributions de ce mémoire sont résumées dans le cinquième chapitre qui propose également des perspectives de recherche dans l'apprentissage automatique et l'extraction de caractéristiques audio multi-échelles. / This doctoral dissertation presents, discusses and proposes tools for the automatic information retrieval in big musical databases.The main application is the supervised classification of musical themes to generate thematic playlists.The first chapter introduces the different contexts and concepts around big musical databases and their consumption.The second chapter focuses on the description of existing music databases as part of academic experiments in audio analysis.This chapter notably introduces issues concerning the variety and unequal proportions of the themes contained in a database, which remain complex to take into account in supervised classification.The third chapter explains the importance of extracting and developing relevant audio features in order to better describe the content of music tracks in these databases.This chapter explains several psychoacoustic phenomena and uses sound signal processing techniques to compute audio features.New methods of aggregating local audio features are proposed to improve song classification.The fourth chapter describes the use of the extracted audio features in order to sort the songs by themes and thus to allow the musical recommendations and the automatic generation of homogeneous thematic playlists.This part involves the use of machine learning algorithms to perform music classification tasks.The contributions of this dissertation are summarized in the fifth chapter which also proposes research perspectives in machine learning and extraction of multi-scale audio features.

以往的音樂資訊檢索方法多使用歌詞、曲風、演奏的樂器或一段音頻訊號來當作查詢的媒介，然而，在某些情況下，使用者沒有辦法清楚描述他們想要尋找的歌曲，如：情境式的音樂檢索。本論文提出了一種基於圖像的情境式音樂資訊檢索方法，可以透過輸入圖片來找尋相應的音樂。此方法中我們使用了卷積神經網絡（Convolutional Neural Network）技術來處理圖片，將其轉為低維度的表示法。為了將異質性的多媒體訊息映射到同一個向量空間，資訊網路表示法學習（Network Embedding）技術也被使用，如此一來，可以使用距離計算找回和輸入圖片有關的多媒體訊息。我們相信這樣的方法可以改善異質性資訊間的隔閡（Heterogeneous Gap），也就是指不同種類的多媒體檔案之間無法互相轉換或詮釋。在實驗與評估方面，首先利用從歌詞與歌名得到的關鍵字來搜尋大量圖片當作訓練資料集，接著實作提出的檢索方法，並針對實驗結果做評估。除了對此方法的有效性做測試外，使用者的回饋也顯示此檢索方法和其他方法相比是有效的。同時我們也實作了一個網路原型，使用者可以上傳圖片並得到檢索後的歌曲，實際的使用案例也將在本論文中被展示與介紹。 / Listening to music is indispensable to everyone. Music information retrieval systems help users find their favorite music. A common scenario of music information retrieval systems is to search songs based on user's query. Most existing methods use descriptions (e.g., genre, instrument and lyric) or audio signal of music as the query; then the songs related to the query will be retrieved. The limitation of this scenario is that users might be difficult to describe what they really want to search for. In this paper, we propose a novel method, called ``image2song,'' which allows users to input an image to retrieve the related songs. The proposed method consists of three modules: convolutional neural network (CNN) module, network embedding module, and similarity calculation module. For the processing of the images, in our work the CNN is adopted to learn the representations for images. To map each entity (e.g., image, song, and keyword) into a same embedding space, the heterogeneous representation is learned by network embedding algorithm from the information graph. This method is flexible because it is easy to join other types of multimedia data into the information graph. In similarity calculation module, the Euclidean distance and cosine distance is used as our criterion to compare the similarity. Then we can retrieve the most relevant songs according to the similarity calculation. The experimental results show that the proposed method has a good performance. Furthermore, we also build an online image-based music information retrieval prototype system, which can showcase some examples of our experiments.

Esta dissertação apresenta um estudo comparativo de diferentes métodos computacionais de segmentação estrutural musical, onde o principal objetivo é delimitar fronteiras de seções musicais em um sinal de áudio, e rotulá-las, i.e. agrupar as seções encontradas que correspondem a uma mesma parte musical. São apresentadas novas propostas para segmentação estrutural nãosupervisionada, incluindo métodos para processamento em tempo real, alcançando resultados com taxas de erro inferiores a 12%. O método utilizado compreende um estudo dos descritores sonoros e meios de modelá-los temporalmente, uma exposição das técnicas computacionais de segmentação estrutural e novos métodos de avaliação dos resultados que penalizam tanto a incorreta detecção das fronteiras quanto o número incorreto de rótulos encontrados. O desempenho de cada técnica computacional é calculado utilizando diferentes conjuntos de descritores sonoros e os resultados são apresentados e analisados tanto quantitativa quanto qualitativamente. / A comparative study of different music structural segmentation methods is presented, where the goal is to delimit the borders of musical sections and label them, i.e. group the sections that correspond to the same musical part. Novel proposals for unsupervised segmentation are presented, including methods for real-time segmentation, achieving expressive results, with error ratio less then 12%. Our method consists of a study of sound descriptors, an exposition of the computational techniques for structural segmentation and the description of the evaluation methods utilized, which penalize both incorrect boundary detection and incorrect number of labels. The performance of each technique is calculated using different sound descriptor sets and the results are presented and analysed both from quantitative and qualitative points-of-view.

This work focuses on scientific area called Music Information Retrieval, more precisely it’s subdivision focusing on the recognition of emotions in music called Music Emotion Recognition. The beginning of the work deals with general overview and definition of MER, categorization of individual methods and offers a comprehensive view of this discipline. The thesis also concentrates on the selection and description of suitable parameters for the recognition of emotions, using tools openSMILE and MIRtoolbox. A freely available DEAM database was used to obtain the set of music recordings and their subjective emotional annotations. The practical part deals with the design of a static dimensional regression evaluation system for numerical prediction of musical emotions in music recordings, more precisely their position in the AV emotional space. The thesis publishes and comments on the results obtained by individual analysis of the significance of individual parameters and for the overall analysis of the prediction of the proposed model.

Different music performances of the same piece may significantly differ from each other. Not only the composer and the score defines the listener’s music experience, but the music performance itself is an integral part of this experience. Four parameter classes can be used to describe a performance objectively: tempo and timing, loudness (dynamics), timbre, and pitch. Each of the individual parameters or their combination can generate a unique characteristic performance. The extraction of such objective parameters is one of the difficulties in the field of Music Performance Analysis and Music Information Retrieval. The submitted work summarizes knowledge and methods from both of the fields. The system is applied to extract data from 31 string quartet performances of 2. movement Lento of String Quartet no. 12 F major (1893) by czech romantic composer Antonín Dvořák (1841–1904).

This work contains overview of methods used in the area of Music Information Retrieval, mainly for purposes of searching of musical recordings. Several existing services in the areas of music identification and searching are presented and their methods for unique song identification are described. This work also focuses on possible modifications of these algorithms for searching of cover versions of songs and for the possibility of searching based on voice created examples.