Back to search

Cornell University

Evaluating language models applied to student thinking about experiments

Abstract

dc:description.abstract

Recent advancements in natural language processing (NLP) have enabled education researchers to analyze text-based open-response data efficiently and at scale. Here we investigate the use of these methods for a variety of education research questions across multiple data sets. First, we compare the application of various language models to perform large-scale identification of experimental skills in students' typed lab notes through sentence-level labeling. We find that fine-tuned higher-resource models often perform better than fine-tuned lower-resource models, but few-shot implementations of higher-resource models do not perform better. Second, we investigate methods to assess the trustworthiness of education claims made using machine-coded data. We propose a four-part method for making such claims with supervised natural language processing, grounded in physics experimental lab practices, including quantification of uncertainty to calibrate measurements. We provide evidence for this method using data from two distinct short response survey questions with two distinct coding schemes, and work through a real-world example of using these practices to machine code a data set unseen by human coders. We then implement both a supervised analysis and an unsupervised analysis to analyze data from a survey that assesses student thinking about measurement in both quantum and classical contexts. In the supervised portion, we train models to apply a single consistent coding scheme across five open-ended questions on this survey, and we build on results from the previous chapter by demonstrating a second method to calibrate measurements. In the unsupervised portion, we perform a cluster analysis on student responses to a question that assesses reasoning about sources of uncertainty in four experimental physics scenarios. We conclude our investigations into methods to automate analysis of open-response questions with a study that does not apply natural language processing methods at all. Rather, this study evaluates closed-response versions of questions that assess students' reasoning compared to open-response versions. We conduct an experiment to test if asking students their reasoning in an open-response version of the question changes their answer to a subsequent closed-response version of the question, and we do not find an effect. In the same experiment, we investigate whether we measure students' experimental reasoning differently on the closed-response version compared to the open-response version. We discuss the differences we find between the two formats, suggesting the open-response and closed-response versions of the questions measure different aspects of student thinking. Together, these analyses form a foundation for education researchers to evaluate a range of both established and emerging methods for conducting research at speed and scale. Simultaneously, these analyses expand the toolkit available to education researchers to assess student thinking about experiments.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Physics
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Physics
Grantor
Cornell University
Year dc:date.issued
2025

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Fussell, Rebeckah
Chair dc:contributor.chair
  • Holmes, Natasha
Committee members dc:contributor.committeemember
  • Elser, Veit
  • Thom-Levy, Julia

Subjects

dc:subject × 5

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
17422667
Dc Identifier Other
ProQuest Submission ID: 15230
ProQuest Publication ID: 32170919

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Fussell, Rebeckah. Evaluating language models applied to student thinking about experiments. Doctor of Philosophy thesis, Cornell University, 2025. https://hdl.handle.net/1813/120819