Back to search

University of Minnesota

Active Evaluation and Targeted Data Collection for Generalist Robot Policies

Abstract

dc:description.abstract

Generalist robot manipulation policies trained on large, diverse datasets have shown remarkable promise across a wide range of tasks. However, rigorously evaluating these policies remains a fundamental challenge. Real-world performance depends on a large combinatorial space of task factors including object poses and camera viewpoints, making full, exhaustive evaluation intractable. Additionally, real hardware evaluation is slow and resource-intensive, and current practice relies on narrow, researcher-designed test suites that can miss critical failure modes and misrepresent true deployment readiness. We propose an active evaluation and improvement framework that addresses this challenge by treating policy evaluation as a sequential experimental design problem. Our approach fits a probabilistic surrogate model over a structured space of task factors and adaptively selects evaluation configurations to maximize information gain over the policy’s performance distribution, allowing for sample-efficient characterization of policy behavior across unseen conditions and a systematic identification of failure-prone regions. We compare our approach to standard random testing with 2331 real-world evaluations across 3 tasks with 4 domain factor variations and find that our approach typically saves the evaluator at least 20-40% of trials compared to typical random testing. We further explore how to use the identified failure regions of the factor design space to guide targeted data collection for policy fine-tuning, closing the loop between evaluation and data collection.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Liao, Andrew

Subjects

dc:subject × 6

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Citation
Liao, Andrew. (2026). Active Evaluation and Targeted Data Collection for Generalist Robot Policies. Retrieved from the University Digital Conservancy, https://hdl.handle.net/11299/281679.

Chain of custody

source
Harvested from
University of Minnesota
Base URL
conservancy.umn.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Liao, Andrew. Active Evaluation and Targeted Data Collection for Generalist Robot Policies. 2026. https://hdl.handle.net/11299/281679