{"id":{"repo_id":"calpoly","oai_identifier":"oai:digitalcommons.calpoly.edu:theses-4816"},"canonical_url":"https://search.ndltd.org/etd/calpoly/oai:digitalcommons.calpoly.edu:theses-4816","repository":{"repo_id":"calpoly","name":"Cal Poly","base_url":"https://digitalcommons.calpoly.edu/do/oai/"},"display":{"title":"Enhancing Non-Player Character Dialogue in Video Gages: An Evaluation of Large Language Model-Generated Responses","abstract":"As video games increasingly emphasize narrative depth and player immersion, the quality of Non-Player Character (NPC) dialogue has become crucial for creating engaging gaming experiences. This thesis investigates the potential of Large Language Models (LLMs) to generate high-quality NPC dialogue by comprehensively evaluating four state-of-the-art models: Gemma 3 27B, Mistral 7B, QWEN 2.5, and LLAMA 3.1. The study employs a mixed-methods approach, combining human evaluation (N=50 participants) with AI-based assessment across five key benchmarks: coherence, personality expression, engagement, style/tone appropriateness, and overall quality. Participants evaluated 32 dialogue samples (8 per model) generated for a fantasy game context featuring two distinct characters. Statistical analysis using ordinal mixed-effects models revealed significant differences in performance across models and benchmarks. Mistral 7B consistently outperformed other models, particularly in personality expression and engagement metrics. The study also examined the correlation between human and AI evaluation methods, finding weak positive correlations across benchmarks, suggesting AI evaluators cannot reliably substitute for human judgment in dialogue quality assessment. These findings have important implications for game developers seeking to leverage LLMs for scalable, dynamic dialogue generation while maintaining narrative quality.","abstract_html":"As video games increasingly emphasize narrative depth and player immersion, the quality of Non-Player Character (NPC) dialogue has become crucial for creating engaging gaming experiences. This thesis investigates the potential of Large Language Models (LLMs) to generate high-quality NPC dialogue by comprehensively evaluating four state-of-the-art models: Gemma 3 27B, Mistral 7B, QWEN 2.5, and LLAMA 3.1. The study employs a mixed-methods approach, combining human evaluation (N=50 participants) with AI-based assessment across five key benchmarks: coherence, personality expression, engagement, style/tone appropriateness, and overall quality. Participants evaluated 32 dialogue samples (8 per model) generated for a fantasy game context featuring two distinct characters. Statistical analysis using ordinal mixed-effects models revealed significant differences in performance across models and benchmarks. Mistral 7B consistently outperformed other models, particularly in personality expression and engagement metrics. The study also examined the correlation between human and AI evaluation methods, finding weak positive correlations across benchmarks, suggesting AI evaluators cannot reliably substitute for human judgment in dialogue quality assessment. These findings have important implications for game developers seeking to leverage LLMs for scalable, dynamic dialogue generation while maintaining narrative quality.","abstract_has_math":false,"creators":["Quach, Lam P"],"institution":null,"degree_name":"MS in Statistics","degree_level":null,"degree_discipline":"Statistics","degree_department":null,"school":null,"contributors":["Hunter Glanz","Statistics","College of Science and Mathematics"],"advisors":[],"committee_chairs":[],"committee_members":[],"year":null,"date_issued":"","date_published":null,"updated_at":"2026-11-19T07:12:41Z","subjects":["Large Language Models","Non-Player Character Dialogue","Video Game Development","Human Evaluation","AI Evaluation","Other Computer Engineering"],"languages":[],"rights":[],"rights_urls":[],"identifier_entries":[],"person_identifier_entries":[]},"links":{"outbound_url":"https://digitalcommons.calpoly.edu/theses/3152","outbound_label":"Repository record","outbound_source":"dc:identifier","record_url":"https://digitalcommons.calpoly.edu/theses/3152","record_label":"Repository record","record_kind":"record","record_source":"dc:identifier","download_url":null,"download_label":null,"download_source":null,"access_status":null,"embargo_until":null,"access_note":null},"source_record":{"url":"https://digitalcommons.calpoly.edu/do/oai/?verb=GetRecord&metadataPrefix=oai_etdms&identifier=oai%3Adigitalcommons.calpoly.edu%3Atheses-4816","prefix":"oai_etdms"},"metadata_groups":[{"id":"people","label":"People","entries":[{"key":"dc:contributor","label":"Contributor","values":["Hunter Glanz","Statistics","College of Science and Mathematics"]},{"key":"dc:creator","label":"Author","values":["Quach, Lam P"]}]},{"id":"academic_context","label":"Academic Context","entries":[{"key":"dc:date.available","label":"Dc Date Available","values":["2025-09-02T07:00:00Z"]},{"key":"dc:date.created","label":"Dc Date Created","values":["2025-09-01T07:00:00Z"]},{"key":"thesis:degree_discipline","label":"Discipline","values":["Statistics"]},{"key":"thesis:degree_name","label":"Degree Name","values":["MS in Statistics"]}]},{"id":"subjects_keywords","label":"Subjects and Keywords","entries":[{"key":"dc:subject","label":"Dc Subject","values":["Large Language Models","Non-Player Character Dialogue","Video Game Development","Human Evaluation","AI Evaluation","Other Computer Engineering"]}]},{"id":"identifiers","label":"Identifiers","entries":[{"key":"dc:identifier","label":"Identifier","values":["https://digitalcommons.calpoly.edu/theses/3152"]}]},{"id":"additional","label":"Additional Metadata","entries":[{"key":"dc:description.abstract","label":"Abstract","values":["As video games increasingly emphasize narrative depth and player immersion, the quality of Non-Player Character (NPC) dialogue has become crucial for creating engaging gaming experiences. This thesis investigates the potential of Large Language Models (LLMs) to generate high-quality NPC dialogue by comprehensively evaluating four state-of-the-art models: Gemma 3 27B, Mistral 7B, QWEN 2.5, and LLAMA 3.1. The study employs a mixed-methods approach, combining human evaluation (N=50 participants) with AI-based assessment across five key benchmarks: coherence, personality expression, engagement, style/tone appropriateness, and overall quality. Participants evaluated 32 dialogue samples (8 per model) generated for a fantasy game context featuring two distinct characters. Statistical analysis using ordinal mixed-effects models revealed significant differences in performance across models and benchmarks. Mistral 7B consistently outperformed other models, particularly in personality expression and engagement metrics. The study also examined the correlation between human and AI evaluation methods, finding weak positive correlations across benchmarks, suggesting AI evaluators cannot reliably substitute for human judgment in dialogue quality assessment. These findings have important implications for game developers seeking to leverage LLMs for scalable, dynamic dialogue generation while maintaining narrative quality."]},{"key":"dc:title","label":"Title","values":["Enhancing Non-Player Character Dialogue in Video Gages: An Evaluation of Large Language Model-Generated Responses"]}]}],"canonical_facts":{"dc:contributor":["Hunter Glanz","Statistics","College of Science and Mathematics"],"dc:creator":["Quach, Lam P"],"dc:date.available":["2025-09-02T07:00:00Z"],"dc:date.created":["2025-09-01T07:00:00Z"],"dc:description.abstract":["As video games increasingly emphasize narrative depth and player immersion, the quality of Non-Player Character (NPC) dialogue has become crucial for creating engaging gaming experiences. This thesis investigates the potential of Large Language Models (LLMs) to generate high-quality NPC dialogue by comprehensively evaluating four state-of-the-art models: Gemma 3 27B, Mistral 7B, QWEN 2.5, and LLAMA 3.1. The study employs a mixed-methods approach, combining human evaluation (N=50 participants) with AI-based assessment across five key benchmarks: coherence, personality expression, engagement, style/tone appropriateness, and overall quality. Participants evaluated 32 dialogue samples (8 per model) generated for a fantasy game context featuring two distinct characters. Statistical analysis using ordinal mixed-effects models revealed significant differences in performance across models and benchmarks. Mistral 7B consistently outperformed other models, particularly in personality expression and engagement metrics. The study also examined the correlation between human and AI evaluation methods, finding weak positive correlations across benchmarks, suggesting AI evaluators cannot reliably substitute for human judgment in dialogue quality assessment. These findings have important implications for game developers seeking to leverage LLMs for scalable, dynamic dialogue generation while maintaining narrative quality."],"dc:identifier":["https://digitalcommons.calpoly.edu/theses/3152"],"dc:subject":["Large Language Models","Non-Player Character Dialogue","Video Game Development","Human Evaluation","AI Evaluation","Other Computer Engineering"],"dc:title":["Enhancing Non-Player Character Dialogue in Video Gages: An Evaluation of Large Language Model-Generated Responses"],"thesis:degree_discipline":["Statistics"],"thesis:degree_name":["MS in Statistics"]},"updated_at":"2026-11-19T07:12:41Z"}