Back to search

University of North Texas

PROCOMP: A Problem-Driven Multi-Agent Framework for Comparative Paper Ranking

Abstract

dc:description

The exponential proliferation of scientific publishing has imposed an unprecedented burden on conventional peer‑review mechanisms, giving rise to reviewer fatigue, heterogeneous assessments, and limited scalability. To evaluate the research quickly and fairly in modern academic communication, we introduce ProComp, a multi-agent large language model (LLM) framework aimed at automating structured manuscript evaluation by emulating the distributed expertise inherent in human reviewers. ProComp is different from single-prompt, monolithic automated review systems because it breaks the evaluation process into smaller parts, each with its own agent that looks at a different part of scholarly quality. This makes the process clearer, more accurate, and more detailed. The architecture uses specialized agents for problem formulation, the validity of the methodology, and the contribution of the research. A meta-reviewer then combines all of these results together to make a final analysis. ProComp uses Retrieval-Augmented Generation (RAG) to improve contextual grounding. This lets agents evaluate in the right citation and literature networks. The system provides authors with helpful feedback in the form of both qualitative comments and numerical scores that follow standard review standards. We looked at research articles of NeuroIPS conference and used metrics like mean absolute error (MAE), root mean squared error (RMSE), and how well agent scores matched up with human judgments. Our multi-agent mechanism had a lower absolute error (MAE = 0.97). The structured baselines, such as zero-shot and AgentReview, on the other hand, had competitive error rates (MAE = 1.08). Ablation studies show that each agent with RAG is needed, and removing them made performance worse; the full multi-agent configuration reduces error by approximately 37% (MAE 1.04 → 0.65). These results show that multi-agent, RAG-enabled systems could help reduce the workload of reviewers, make evaluations more consistent, and support large-scale, high-quality scholarly evaluation.

Degree

thesis:*
Grantor dc:publisher
University of North Texas
Year dc:date
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Mamidala, Laxmi Shravani, 1997-
Contributors dc:contributor
  • Xiao, Ting
  • Li,Yuan
  • Yuan, Jing

Subjects

dc:subject × 4

Rights

dc:rights
Statement dc:rights
  • Public
  • Mamidala, Laxmi Shravani, 1997-
  • Copyright
  • Copyright is held by the author, unless otherwise noted. All rights Reserved.
Language dc:language
English

Identifiers

dc:identifier.*
Identifier
local-cont-no: submission_4859
ark: ark:/67531/metadc2588708

Chain of custody

source
Harvested from
University of North Texas
Base URL
digital.library.unt.edu/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Mamidala, Laxmi Shravani, 1997-. PROCOMP: A Problem-Driven Multi-Agent Framework for Comparative Paper Ranking. University of North Texas, 2026. https://digital.library.unt.edu/ark:/67531/metadc2588708/