Back to search

Virginia Tech

Designing Physiological Connections in Artificial Intelligence: Towards integration of an AI collaborator into a musical ensemble for group aesthetics and control using biofeedback with affective states

Abstract

dc:description.abstract

We are artistic and creative human beings. As expert systems encroach on creative or artistic processes, rather than vilify or create certainty, how can we co-create with AI as an agent and have some synergy for the future? While some systems move towards excluding human interaction in favor of AI-generated processes, there is also a growing space for more physiological human-computer interactions. Artificial intelligence in music is often evalu- ated through autonomous generation, leaving unresolved how computational systems can support collaboration while preserving human agency. This dissertation develops Affective Computing in Musical Collaboration (ACMC), an interdisciplinary framework that treats affect as expressed, sensed, reported, negotiated, and acted upon within musical interaction rather than only as a classification label. Informed by participatory and transition-design principles, the project integrated a Russell circumplex interface, low-cost MAX30102 heart- rate and blood-oxygen sensing, Grove skin-conductance sensing, Pd-L2Ork/L2Ork Tweeter, Python, Open Sound Control, event logging, statistical analysis, and machine learning. A related symbolic-generation study using a human-classified emotional MIDI dataset found that continuous-concatenated transformer conditioning performed best among the evaluated conditioning methods. The empirical work followed a validation-to-interaction sequence. Phase 1 involved eight participants and 80 passive-listening trials across ten high-arousal, mixed-valence excerpts. All trials were reconstructed, heart-rate change showed a reproducible association with target valence, and self-reported valence retained modest stimulus ordering. However, baseline-relative directional concordance ranged from 33.8% to 47.5%, below the prespecified 80% progression criterion. Phase 2 was therefore interpreted as an exploratory feasibility study. Fifty participants completed 600 interactive Solo, Transition, and Drop trials. Skin conductance increased during and after interaction, heart rate rose most consistently after events, and SpO2 showed a modest post-event decrease. Drop examples were rated more arousing than Solo and Transition examples, whereas valence varied more by individual example than by musical function. Complete-case System Usability Scale scores averaged 81.80/100, and post-task and qualitative results indicated strong usability, enjoyment, learning, and creative exploration. The findings establish the feasibility and expressive value of an open-source, psychophysiologically responsive musical interface, but not a universal emotion decoder. Physiological and audio-derived affect estimates agreed weakly with independent retrospective ratings; participant-grouped models produced negative held-out R2 values, and function classification remained near chance. The planned Biopac reference-device comparison was not completed, so medical-grade sensor accuracy is not claimed. Ensemble consensus and audience coupling were also not tested. The dissertation contributes the ACMC framework, an open research stack, a validation model separating acquisition, physiological sensitivity, agreement, prediction, usability, and collaboration, and a design direction for personalized, transparent, mixed-initiative musical AI that treats physiology as contextual evidence rather than unquestioned authority.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy
Level thesis:degree_level
doctoral
Discipline thesis:degree_discipline
Individual Interdisciplinary PhD
Department dc:contributor.department
Graduate School
Grantor dc:publisher
Virginia Tech
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Kerobo, Justin Adebayo
Chair dc:contributor.committeechair
  • Algargoosh, Alaa Salman A.
Committee members dc:contributor.committeemember
  • Patrick, Rafael
  • Bukvic, Ivica
  • Scarpa-Friedman, Bruce H.

Subjects

dc:subject × 4

Rights

dc:rights
Statement dc:rights
  • In Copyright
Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Other
vt_gsexam:47708

Chain of custody

source
Harvested from
Virginia Tech
Base URL
vtechworks.lib.vt.edu/oai/request
Last updated
2026-11-18
Source record
OAI-PMH GetRecord
citation

Kerobo, Justin Adebayo. Designing Physiological Connections in Artificial Intelligence: Towards integration of an AI collaborator into a musical ensemble for group aesthetics and control using biofeedback with affective states. doctoral thesis, Virginia Tech, 2026. https://hdl.handle.net/10919/143975