Georgia Institute of Technology
Detecting and Mitigating Pedagogical Risks in Large Language Models with Visual Analytics
Abstract
dc:description.abstractThe advent of large language models (LLMs) has catalyzed a surge in LLM-powered educational technologies, enabling transformational advances that can empower learner agency, deliver personalized study materials, and promote active learning. Data scientists in education, called learning engineers, are at the forefront of this technological revolution, leading efforts to integrate LLMs into adaptive learning experiences. Yet persistent pedagogical risks, from bias and hallucinations to unfair grading and misalignment with instructional goals, highlight a critical gap. Existing tools for selecting, fine-tuning, and evaluating LLMs are not designed to address the unique challenges of educational contexts, limiting the broader adoption of LLM-powered tools for learning. To address this gap, this thesis explores how human-in-the-loop visual analytics can equip learning engineers to better discover, understand, and address these risks by combining automated analysis with interactive visualizations that support more informed, iterative decision-making throughout the LLM development lifecycle. Specifically, this thesis contributes to visual analytics, human computer interaction, artificial intelligence, and educational technology across several related thrusts. First, KnowledgeVIS enables visual exploration of latent biases in foundation LLMs during model selection, towards uncovering harmful LLM biases and stereotypes. Second, iScore visualizes internal and external model perturbations to diagnosis unintended LLM decision-making criteria during fine-tuning. Third, our novel framework for measuring and visualizing LLM trustworthiness issues makes pedagogical risks interpretable and traceable during LLM evaluation. Fourth, TrustyVis implements modular visual analytics components for reusable integration into real-world workflows, broadening access to visual analytics for learning engineers.
Degree
thesis:*- Name thesis:degree_name
- Human-Centered Computing, PhD
- Department thesis:degree_department
- Interactive Computing
- Grantor
- Georgia Institute of Technology
- Year dc:date.issued
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Coscia, Adam Joseph
- Advisor dc:contributor.advisor
-
- Endert, Alex
- ORCID person:identifier.orcid
Rights
- Language dc:language.iso
- English
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- https://hdl.handle.net/1853/82145