Environment Blockers
Summary: External factors that prevent Computer Use Agents from achieving their goals despite executing actions correctly. These uncontrollable failures distinguish between agent execution quality and environmental constraints that block success.
Overview
Environment Blockers represent a critical distinction in Agent Evaluation between controllable and uncontrollable failure modes. While Process vs Outcome Rewards can diverge for multiple reasons, environment blockers specifically occur when an agent executes its task correctly but external factors prevent goal achievement.
This concept is fundamental to fair evaluation of Computer Use Agents, as it separates execution quality from environmental constraints beyond the agent's control. The Universal Verifier system explicitly accounts for this distinction, recognizing that high process scores can coexist with low outcome scores when environment blockers interfere.
Key Details
- Independence from execution quality: Environment blockers occur regardless of how well an agent performs its actions
- External nature: These factors originate outside the agent's control or influence
- Impact on outcome evaluation: Can cause legitimate task failures that shouldn't penalize the agent's performance assessment
- Evaluation implications: Proper Trajectory Verification must distinguish between agent errors and environmental constraints
- Examples include: Website downtime, authentication failures, inventory unavailability, network connectivity issues, or third-party service disruptions
- Measurement challenge: Requires sophisticated Rubric Design to identify when environmental factors rather than agent errors cause task failure
Relationships
- Process vs Outcome Rewards — environment blockers explain why these can diverge even with correct execution
- Computer Use Agents — systems that must navigate environments with potential blockers
- Universal Verifier — evaluation system designed to distinguish controllable from uncontrollable failures
- Trajectory Verification — process that must account for environment blockers in assessment
- Agent Evaluation — field requiring fair attribution of failure causes
- Rubric Design — evaluation frameworks that must capture environmental vs execution failures
Sources
- sources/the-art-of-building-verifiers-for-computer-use-agents — introduced the concept as part of distinguishing controllable vs uncontrollable failures in agent evaluation