Back to search

Rochester Institute of Technology

Toward Reliable Computational Social Science: Inconsistency-Aware Methods for Human Annotation and AI Inference

Abstract

dc:description.abstract

As artificial intelligence (AI) becomes increasingly common in computational social science, \textit{inconsistency} has emerged as a key challenge. AI models often contradict themselves when given equivalent inputs, disagree with other models on the same data, and diverge from human judgments in seemingly opaque ways. Human annotators exhibit their own inconsistencies, both within individuals and across groups shaped by differing values and identities. Rather than treating these inconsistencies simply as noise, this dissertation argues that they contain meaningful signals that can be leveraged to improve learning efficiency, strengthen evaluation, and increase the reliability of large-scale social measurement. To study this phenomenon, we first develop a taxonomy of five forms of inconsistency: \textit{intra-model}, \textit{inter-model}, \textit{intra-annotator}, \textit{inter-annotator}, and \textit{human-AI}. Each of these is examined through empirical studies on real-world datasets from various social science domains. First, in intra-model inconsistency, we introduce \textit{inconsistency sampling}, an active learning strategy that treats a model's own contradictions as informative training signals, and apply this method to natural language inference-based analysis of cable news coverage of policing. Second, to study inter-model inconsistency, we conduct a noise audit of nine offensive speech classifiers on more than three million social media comments. We also use \textit{model inconsistency sampling} to analyze police coverage in local news across 10 U.S. cities, and in Indian parliamentary discourse on education. In intra-annotator inconsistency, we introduce \texttt{ARTICLE}, a framework that estimates annotator reliability through self-consistency rather than majority agreement, allowing minority perspectives to be preserved. To investigate inter-annotator inconsistency, we introduce the concept of \textit{vicarious offense}, a paradigm that reveals how political groups perceive, and often misperceive, what offends members of the other groups. Finally, in human-AI inconsistency, we show that generative AI systems and humans differ in how they weigh situational factors when making social decisions that entail moral responsibility and immediate personal risk. Our findings highlight the importance of inconsistency-aware methods for the responsible use of generative AI in socially consequential domains.

Degree

thesis:*
Name thesis:degree_name
Computing and Information Sciences (Ph.D.)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Computing and Information Sciences Ph.D, Department of

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Dutta, Sujan
Contributors dc:contributor
  • Ashiqur R. KhudaBukhsh

Subjects

dc:subject × 6

Identifiers

dc:identifier.*
Repository record dc:identifier
https://repository.rit.edu/theses/12675

Chain of custody

source
Harvested from
Rochester Institute of Technology
Base URL
repository.rit.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Dutta, Sujan. Toward Reliable Computational Social Science: Inconsistency-Aware Methods for Human Annotation and AI Inference. Dissertation thesis, https://repository.rit.edu/theses/12675