Institutional Repository of Vilnius University
Developing educational content using artificial intelligence /
Abstract
dc:descriptionCreating programming exercises is time-consuming for educators, as each exercise needs a correct solution, valid test cases, and a clear problem description. This study evaluates three large language models, Google's Gemini 2.5 Flash, Anthropic's Claude Haiku 4.5, and OpenAI's GPT-5.4-mini, for automatically generating C programming exercises, comparing their technical correctness and educational quality. A software system was developed that sends identical prompts to each model and validates exercises through a unified pipeline covering parsing, compilation, and test case execution. A dataset of 240 exercises was generated across four topics of increasing complexity: variables and operators, conditionals and loops, arrays and strings, and pointers and memory allocation. Of these, 212 (88.3%) passed automated validation, with Gemini achieving the highest pass rate (93.75%), followed by Claude (86.25%) and GPT (85%). The dominant cause of failure was incorrect expected values in test cases rather than logically wrong solution code, indicating that models can write correct C code but struggle to trace through their own solutions. Human evaluation by 15 evaluators with C programming experience assessed exercises for clarity, concept targeting, difficulty, and usability. Gemini scored highest on clarity and usability (93.3% usable without modification), GPT produced clear but consistently too easy exercises, and Claude generated more demanding exercises with less clear descriptions (70.8% usable as is). Clarity and usability declined across all models on the most advanced topic. The results show that current budget-tier large language models can generate C programming exercises that are suitable for classroom use, but human review is still necessary, particularly for advanced topics, difficulty calibration, and test case accuracy. A concrete improvement would be executing generated solutions and using actual output as expected test case values rather than relying on models to predict them.
Degree
thesis:*- Grantor dc:publisher
- Institutional Repository of Vilnius University
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Jemeljanovas, Šarūnas,
Subjects
dc:subject × 1Rights
dc:rights- Statement dc:rights
-
- info:eu-repo/semantics/openAccess
- Language dc:language
- eng
Identifiers
dc:identifier.*- Repository record dc:identifier
- https://repository.vu.lt/VU:ELABAETD308337086&prefLang=en_US