CUAVerifierBench

Summary: CUAVerifierBench is the first benchmark specifically designed to evaluate verifier quality for computer use agents, providing both process and outcome human labels to measure how well automated systems can assess agent trajectory success. The benchmark enables researchers to measure verifier performance against human agreement baselines using Cohen's kappa metrics.

Overview

CUAVerifierBench addresses a critical gap in Computer Use Agents evaluation by providing the first dedicated benchmark for measuring Trajectory Verification quality. Unlike existing benchmarks that focus on agent performance, CUAVerifierBench evaluates the verifiers themselves - the systems responsible for determining whether an agent successfully completed its task.

The benchmark includes human-annotated labels for both process and outcome evaluation, enabling researchers to measure how well their verifiers align with human judgment. This separation of Process vs Outcome Rewards is crucial because an agent might execute perfectly but fail due to environmental factors beyond its control, or conversely, might achieve the goal despite poor execution.

Released by Microsoft Research as part of their Universal Verifier development, CUAVerifierBench exposes significant weaknesses in existing verification systems. Previous verifiers like WebVoyager and WebJudge showed False Positive Rates of 45%+ and 22%+ respectively, while the Universal Verifier achieved 1-8% FPR with Cohen's κ ≈ 0.7 human agreement.

Key Details

  • First dedicated verifier benchmark: The only benchmark specifically targeting verifier quality rather than agent performance
  • Dual labeling system: Provides separate human annotations for process quality (execution) and outcome success (goal achievement)
  • Human agreement baseline: Enables measurement against Inter-annotator Agreement standards using Cohen's kappa (κ ≈ 0.7 represents near-human agreement)
  • Multimodal evaluation: Incorporates Screenshot Context Management for visual evidence assessment across long interaction sequences
  • Hallucination detection cases: Includes scenarios where agents fabricate actions or claim unsupported facts, enabling Hallucination Detection testing
  • Conditional criteria handling: Accounts for tasks with adaptive requirements (e.g., "buy organic if available, else non-organic")
  • Benchmarks existing systems: Systematically measures and exposes high error rates in previous verifiers
  • Rubric Design foundation: Uses structured, specific, non-overlapping criteria for evaluation consistency
  • Process vs outcome separation: Distinguishes between execution quality and goal achievement, critical for fair assessment
  • Context efficiency: Tests verifier ability to select relevant screenshots rather than processing all visual data

Relationships

  • Computer Use Agents — provides evaluation framework for these autonomous systems that operate computers via screenshots and actions
  • Trajectory Verification — the core task that CUAVerifierBench measures across multi-step agent interactions
  • Process vs Outcome Rewards — benchmark structure separates these two evaluation dimensions to handle environmental vs agent failures
  • Inter-annotator Agreement — provides baseline for measuring verifier-human alignment using Cohen's kappa metrics
  • Hallucination Detection — includes test cases for identifying when agents fabricate actions or claim unsupported facts
  • Universal Verifier — the Microsoft system that achieved near-human performance (κ ≈ 0.7) on this benchmark
  • Screenshot Context Management — tests verifier ability to efficiently process visual evidence across long sequences
  • Rubric Design — benchmark validates structured criteria approaches for consistent multi-step task evaluation
  • WebVoyager — existing verifier system benchmarked with 45%+ false positive rate
  • WebJudge — another existing verifier system showing 22%+ false positive rate
  • Agent Evaluation — broader field that CUAVerifierBench contributes specialized verifier assessment to
  • False Positive Rate — key metric that CUAVerifierBench measures, showing dramatic improvements possible

Sources