Back to search

Georgia Institute of Technology

OLIVE: Evaluating Human–AI Co-creation Dynamics through Structured Narrative Feedback Loops

Abstract

dc:description.abstract

Large Language Models can generate text that flows naturally following the structural patterns of storytelling, but lack the inherent internal mechanisms to evaluate whether their contributions advance a story in meaningful ways. This thesis presents OLIVE, a non-training based evaluator-in-the-loop framework for LLM-driven collaborative storytelling. Rather than serving as a fine-tuned system or modifying model weights, OLIVE operates as an external module that evaluates a story across three narrative dimensions (appropriateness, newness, and participation) and uses these internal evaluation mechanisms to steer generation dynamically across a three-act narrative story. A within-subjects study with twelve participants compared OLIVE against a baseline version of the same system without the evaluator loop. Participants co-created stories with each system, completed task-experience questionnaires covering ten constructs adapted from the Mixed-Initiative Creativity Support Index, and took part in semi-structured interviews. Statistical tests yielded no significant differences in the tested population; however signals were present. Descriptively, OLIVE received higher ratings on Contribution, Surprise and Novelty, while Baseline was rated higher on Exploration, Satisfaction, and Immersiveness. System-level metadata showed that OLIVE sessions spent more turns in Exposition and were less likely to reach later narrative phases. Interview responses highlighted that participants experienced OLIVE as a more active collaborator whose contributions were valued for introducing unexpected material, but at the cost of narrative momentum and control. Participants had varied opinions on whether they had approached the system as a co-author to write alongside or as a tool that needs to be steered. These findings suggest that evaluator-in-the-loop steering shifts the collaboration toward a more active but less fluid form of co-authorship, with an increase in narrative contribution and surprise while introducing friction in pacing and control. This tradeoff was perceived differently on whether participants viewed the system as a tool to steer and direct versus a co-author to work alongside.

Degree

thesis:*
Name thesis:degree_name
Computer Science, MS
Department thesis:degree_department
Computer Science
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Gunerli, John Hakan Can
Advisors dc:contributor.advisor
  • Magerko, Brian
  • Riedl, Mark O.
Committee members dc:contributor.committeemember
  • Manoj Deshpande
  • Milka Trajkova
ORCID person:identifier.orcid

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/81608

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Gunerli, John Hakan Can. OLIVE: Evaluating Human–AI Co-creation Dynamics through Structured Narrative Feedback Loops. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/81608