Georgia Institute of Technology
OLIVE: Evaluating Human–AI Co-creation Dynamics through Structured Narrative Feedback Loops
Abstract
dc:description.abstractLarge Language Models can generate text that flows naturally following the structural patterns of storytelling, but lack the inherent internal mechanisms to evaluate whether their contributions advance a story in meaningful ways. This thesis presents OLIVE, a non-training based evaluator-in-the-loop framework for LLM-driven collaborative storytelling. Rather than serving as a fine-tuned system or modifying model weights, OLIVE operates as an external module that evaluates a story across three narrative dimensions (appropriateness, newness, and participation) and uses these internal evaluation mechanisms to steer generation dynamically across a three-act narrative story. A within-subjects study with twelve participants compared OLIVE against a baseline version of the same system without the evaluator loop. Participants co-created stories with each system, completed task-experience questionnaires covering ten constructs adapted from the Mixed-Initiative Creativity Support Index, and took part in semi-structured interviews. Statistical tests yielded no significant differences in the tested population; however signals were present. Descriptively, OLIVE received higher ratings on Contribution, Surprise and Novelty, while Baseline was rated higher on Exploration, Satisfaction, and Immersiveness. System-level metadata showed that OLIVE sessions spent more turns in Exposition and were less likely to reach later narrative phases. Interview responses highlighted that participants experienced OLIVE as a more active collaborator whose contributions were valued for introducing unexpected material, but at the cost of narrative momentum and control. Participants had varied opinions on whether they had approached the system as a co-author to write alongside or as a tool that needs to be steered. These findings suggest that evaluator-in-the-loop steering shifts the collaboration toward a more active but less fluid form of co-authorship, with an increase in narrative contribution and surprise while introducing friction in pacing and control. This tradeoff was perceived differently on whether participants viewed the system as a tool to steer and direct versus a co-author to work alongside.
Degree
thesis:*- Name thesis:degree_name
- Computer Science, MS
- Department thesis:degree_department
- Computer Science
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Gunerli, John Hakan Can
- Advisors dc:contributor.advisor
-
- Magerko, Brian
- Riedl, Mark O.
- Committee members dc:contributor.committeemember
-
- Manoj Deshpande
- Milka Trajkova
- ORCID person:identifier.orcid
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/81608