Back to search

Dublin City University

Structured by ADDIE, Guided by Freire: Developing Critical MT Quality Assessment Training

Abstract

dc:description.abstract

This research develops and evaluates a pedagogical intervention for teaching Machine Translation Quality Assessment (MTQA) to postgraduate students in Natural Language Processing (NLP). While Translation Studies (TS) has a long tradition of exploring MTQA, MTQA pedagogy is underdeveloped in NLP education. In both academic and industry contexts, MTQA is most commonly carried out using Automatic Evaluation Metrics (AEMs). These metrics are valued for their speed, low cost, and scalability, but they fail to capture important linguistic, contextual, and ethical aspects of translation quality. Human Evaluation (HE), while capable of providing more reliable and nuanced insights, is underused because it is complex, resource-intensive, and time-consuming. As Neural Machine Translation (NMT) continues to produce mistranslations, omissions, and biases, and Large Language Models (LLMs) raise new risks such as hallucinations and instability across system updates, the need for rigorous evaluation practices becomes even more critical. To address this imbalance, this study draws on the ADDIE instructional design framework and Freirean critical pedagogy to structure a training course on MTQA. A systematic review of literature and a survey of educators identified gaps in how MTQA is taught and reported. Based on these findings, a syllabus and course were designed, combining theoretical foundations with hands-on practice in HE, AEMs, evaluation design, and ethics. The course was implemented with postgraduate NLP students, who participated in lectures, lab-based evaluation tasks, and critical reflection on the methodological and ethical dimensions of MTQA. Effectiveness was assessed through a mixed-methods design, including pre-, midand post-tests analysed with Bayesian linear mixed modelling, classroom observation, and thematic analysis of student reflections. The results show that the students increased their confidence in conducting HE, developed a greater critical awareness of the the strengths and weaknesses of each type of evaluation, and demonstrated the ability to apply rigorous evaluation practices to MT systems. By designing, delivering, and evaluating this course, the study demonstrates that critical MTQA training is both feasible and effective. It contributes a structured pedagogical model for embedding responsible and transparent evaluation practices into NLP curricula, promoting more human-centred and accountable approaches to the development of systems focused on AI-driven translation.

Degree

thesis:*
Name dc:type.qualificationname
phd
Level dc:type.qualificationlevel
doctoral
Grantor dc:publisher.institution
Dublin City University
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Cavalheiro Camargo, João Lucas

Rights

Language dc:language
en

Chain of custody

source
Harvested from
Dublin City University
Base URL
doras.dcu.ie/cgi/oai2
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Cavalheiro Camargo, João Lucas. Structured by ADDIE, Guided by Freire: Developing Critical MT Quality Assessment Training. doctoral thesis, Dublin City University, 2026.