University of South Florida
A Psychometric Analysis of Natural Language Inference Using Transformer Language Models
Abstract
dc:descriptionLarge language models (LLMs) are poised to transform both academia and industry. But the excitement around these generative AIs has also been met with concern for the true extent of their capabilities. This dissertation helps to address these questions by examining the capabilities of LLMs using the tools of psychometrics. We focus on analyzing the capabilities of LLMs on the task of natural language inference (NLI), a foundational benchmark often used to evaluate new models. We demonstrate that LLMs can reliably predict the psychometric properties of NLI items were those items administered to humans. Through a series of experiments, we show that LLMs can improve the validity and reliability of NLI by both helping to refine the operationalization of theconstruct and by automatically generating new items with superior validity evidence. Finally, in a related line of work, we demonstrate that LLMs can predict the age at which children acquire words in their vocabulary. Our research demonstrates the potential of applying the tools of psychometrics to the analysis of generative AI and paves the way for creating better AI assessments.
Degree
thesis:*- Name thesis:degree_name
- Doctor of Philosophy (Ph.D.)
- Level thesis:degree_level
- Dissertation
- Discipline thesis:degree_discipline
- Computer Science and Engineering
- Grantor
- University of South Florida
- Year dc:date
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Laverghetta, Antonio, Jr.
- Contributors dc:contributor
-
- John Licato, Ph.D.
- Shaun Canavan, Ph.D.
- Kelsey Merlo, Ph.D.
- Nicole Beckage, Ph.D.
- Ismail Uysal, Ph.D.
- Michael Maness, Ph.D.
Subjects
dc:subject × 8Identifiers
dc:identifier.*- Repository record dc:identifier
- https://digitalcommons.usf.edu/etd/10121