Georgia Institute of Technology
Interpretable Quantitative Evaluation Metrics Of Generative Models In Symbolic Music
Abstract
dc:description.abstractDespite the emergence of innovative architectures claiming improved capabilities in modeling human-level creativity, state-of-the-art generative music systems still struggle with creating musical content that follows technical rules and expectations. The conventional subjective evaluation method for generative models can introduce bias and also lacks transparency, rigor, and reproducibility, emphasizing the need for more quantitative metrics. However, existing approaches to quantitative evaluation have either relied on overly- broad criteria that do not capture higher-level music theoretic properties nor perceptual properties, or are narrowly tailored to the design of a specific model, limiting their generalizability. To address this, this thesis proposes a reproducible and interpretable framework for evaluating the output of symbolic music generation models using musicologically and perceptually informed quantitative metrics. Specifically, I assessed the performance of two prominent models, FolkRNN and Jazz Transformer, by comparing the models’ training data against their generated results through systematic computational music analysis. Benchmark testing revealed this approach to surpass the discriminative capabilities of the widely cited, seminal quantitative metrics proposed by Yang and Lerch, offering a more ecologically-valid assessment of model behavior and highlighting areas for targeted improvement. To further substantiate the perceptual validity of my metrics, I then reported results from a listening study employing a Turing Test disguised as a style-classification task. This experiment tested for the salient musical features that influences individuals’ decision-making in identifying stylistic provenance and provided insights into the perceptual dimensions of style imitation challenges in AI music generation. Together, my findings hold the potential to advance the reliability and validity of AI-generated music by incorporating human perceptual attributes.
Degree
thesis:*- Level thesis:degree_level
- Masters
- Department dc:contributor.department
- Music
- Grantor dc:publisher
- Georgia Institute of Technology
- Year dc:date.issued
- 2024
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Sun, Qianyi
- Advisor dc:contributor.advisor
-
- Arthur, Claire
- Committee members dc:contributor.committeemember
-
- Lerch, Alexander G.
- Wu, Hongchen
Subjects
dc:subject × 6Rights
- Language dc:language.iso
- en_US
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/75633