Carleton University
Bidirectional Generation of Behavior-Driven Development Artifacts Using Fine-Tuning and Retrieval-Augmented Generation
Abstract
dc:description.abstractThis thesis investigates bidirectional generation between semi-structured records and structured Behavior-Driven Development (BDD) scenarios using Large Language Models (LLMs). It compares fine-tuning (CodeT5+ and Qwen2.5-Coder) with Retrieval-Augmented Generation (RAG) (Meta-Llama-3.1-8B, DeepSeek-Coder) across a curated dataset of 2,100 aligned pairs. Results demonstrate that generation quality is heavily influenced by task direction and context management. Fine-tuned models significantly outperformed RAG-based methods, achieving F1 scores up to 0.9549 and Exact Match scores of 0.8119. RAG struggled to maintain the strict structural fidelity required for Gherkin-based BDD artifacts, failing to achieve exact matches. These findings suggest that fine-tuning remains the most reliable strategy for generating technically accurate BDD artifacts, as it better handles the rigid structural constraints of automated testing specifications compared to in-context prompting alone.
Degree
thesis:*- Name thesis:degree_name
- Master of Applied Science (M.App.Sc.)
- Level thesis:degree_level
- Master's
- Discipline thesis:degree_discipline
- Engineering, Electrical and Computer
- Grantor dc:publisher
- Carleton University
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Biniek, Artin
Subjects
dc:subject × 4Rights
dc:rights- Statement dc:rights
-
- Copyright © 2026 the author(s). Theses may be used for non-commercial research, educational, or related academic purposes only. Such uses include personal study, distribution to students, research and scholarship. Theses may only be shared by linking to the Carleton University Institutional Repository and no part may be copied without proper attribution to the author; no part may be used for commercial purposes directly or indirectly via a for-profit platform; no adaptation or derivative works are permitted without consent from the copyright owner.
- Language dc:language.iso
- en