Screenshot Relevance Matrix

Summary: A scoring system that selects the most relevant screenshots for each rubric criterion when evaluating computer use agents, rather than using all screenshots or simple truncation methods. This approach optimizes context management by providing targeted visual evidence for each evaluation dimension.

Overview

The Screenshot Relevance Matrix addresses a critical challenge in Trajectory Verification for Computer Use Agents: how to efficiently process large volumes of screenshot evidence when evaluating agent performance against structured rubrics. Instead of overwhelming the verifier with all available screenshots or arbitrarily truncating the sequence, this system scores each screenshot against specific rubric criteria to select the top-k most relevant visual evidence for each evaluation dimension.

This targeted approach is part of Microsoft's Universal Verifier system's Screenshot Context Management strategy, enabling more accurate verification while managing computational constraints. The matrix ensures that verifiers receive the most pertinent visual evidence for each aspect of performance being evaluated.

Key Details

• Criterion-specific selection: Each rubric criterion receives its own set of most relevant screenshots rather than a global selection
• Top-k methodology: Selects a fixed number of highest-scoring screenshots per criterion to maintain consistent context size
• Part of Universal Verifier: Integrated into the broader system that achieves Cohen's κ ≈ 0.7 agreement with humans
• Addresses scale challenges: Computer use trajectories can generate dozens or hundreds of screenshots during complex tasks
• Supports rubric design: Works with specific, non-overlapping rubrics that separate Process vs Outcome Rewards
• Enables targeted evaluation: Different criteria may require different types of visual evidence (e.g., navigation steps vs final results)

Relationships

• Trajectory Verification — core application domain for the matrix system
• Rubric Design — provides the criteria structure that drives screenshot selection
• Screenshot Context Management — broader category of techniques this matrix exemplifies
• Computer Use Agents — the systems being evaluated through this screenshot selection process
• Hallucination Detection — benefits from having relevant visual evidence to verify agent claims
• Process vs Outcome Rewards — different reward types may require different screenshot evidence

Sources

• sources/the-art-of-building-verifiers-for-computer-use-agents — introduced the concept as part of the Universal Verifier system's context management approach