University of North Texas
PROCOMP: A Problem-Driven Multi-Agent Framework for Comparative Paper Ranking
Abstract
dc:descriptionThe exponential proliferation of scientific publishing has imposed an unprecedented burden on conventional peer‑review mechanisms, giving rise to reviewer fatigue, heterogeneous assessments, and limited scalability. To evaluate the research quickly and fairly in modern academic communication, we introduce ProComp, a multi-agent large language model (LLM) framework aimed at automating structured manuscript evaluation by emulating the distributed expertise inherent in human reviewers. ProComp is different from single-prompt, monolithic automated review systems because it breaks the evaluation process into smaller parts, each with its own agent that looks at a different part of scholarly quality. This makes the process clearer, more accurate, and more detailed. The architecture uses specialized agents for problem formulation, the validity of the methodology, and the contribution of the research. A meta-reviewer then combines all of these results together to make a final analysis. ProComp uses Retrieval-Augmented Generation (RAG) to improve contextual grounding. This lets agents evaluate in the right citation and literature networks. The system provides authors with helpful feedback in the form of both qualitative comments and numerical scores that follow standard review standards. We looked at research articles of NeuroIPS conference and used metrics like mean absolute error (MAE), root mean squared error (RMSE), and how well agent scores matched up with human judgments. Our multi-agent mechanism had a lower absolute error (MAE = 0.97). The structured baselines, such as zero-shot and AgentReview, on the other hand, had competitive error rates (MAE = 1.08). Ablation studies show that each agent with RAG is needed, and removing them made performance worse; the full multi-agent configuration reduces error by approximately 37% (MAE 1.04 → 0.65). These results show that multi-agent, RAG-enabled systems could help reduce the workload of reviewers, make evaluations more consistent, and support large-scale, high-quality scholarly evaluation.
Degree
thesis:*- Grantor dc:publisher
- University of North Texas
- Year dc:date
- 2026
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Mamidala, Laxmi Shravani, 1997-
- Contributors dc:contributor
-
- Xiao, Ting
- Li,Yuan
- Yuan, Jing
Subjects
dc:subject × 4Rights
dc:rights- Statement dc:rights
-
- Public
- Mamidala, Laxmi Shravani, 1997-
- Copyright
- Copyright is held by the author, unless otherwise noted. All rights Reserved.
- Language dc:language
- English
Identifiers
dc:identifier.*- Identifier
-
local-cont-no: submission_4859
ark: ark:/67531/metadc2588708