Back to search

University of South Florida

A Psychometric Analysis of Natural Language Inference Using Transformer Language Models

Abstract

dc:description

Large language models (LLMs) are poised to transform both academia and industry. But the excitement around these generative AIs has also been met with concern for the true extent of their capabilities. This dissertation helps to address these questions by examining the capabilities of LLMs using the tools of psychometrics. We focus on analyzing the capabilities of LLMs on the task of natural language inference (NLI), a foundational benchmark often used to evaluate new models. We demonstrate that LLMs can reliably predict the psychometric properties of NLI items were those items administered to humans. Through a series of experiments, we show that LLMs can improve the validity and reliability of NLI by both helping to refine the operationalization of theconstruct and by automatically generating new items with superior validity evidence. Finally, in a related line of work, we demonstrate that LLMs can predict the age at which children acquire words in their vocabulary. Our research demonstrates the potential of applying the tools of psychometrics to the analysis of generative AI and paves the way for creating better AI assessments.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy (Ph.D.)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Computer Science and Engineering
Grantor
University of South Florida
Year dc:date
2023

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Laverghetta, Antonio, Jr.
Contributors dc:contributor
  • John Licato, Ph.D.
  • Shaun Canavan, Ph.D.
  • Kelsey Merlo, Ph.D.
  • Nicole Beckage, Ph.D.
  • Ismail Uysal, Ph.D.
  • Michael Maness, Ph.D.

Subjects

dc:subject × 8

Identifiers

dc:identifier.*
Repository record dc:identifier
https://digitalcommons.usf.edu/etd/10121

Chain of custody

source
Harvested from
University of South Florida
Base URL
digitalcommons.usf.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Laverghetta, Antonio, Jr.. A Psychometric Analysis of Natural Language Inference Using Transformer Language Models. Dissertation thesis, University of South Florida, 2023. https://digitalcommons.usf.edu/etd/10121