Let's Think in Two Steps:
Mitigating Agreement Bias in MLLMs with
Self-Grounded Verification

Verifiers — functions assigning rewards to agent behavior — have been key for AI progress in domains such as math and board games. However, extending these gains to domains without clear-cut success criteria (e.g., computer use) remains a challenge: while humans can recognize suitable outcomes, translating this intuition into scalable rules is non-trivial. Multimodal Large Language Models (MLLMs) emerge as a promising solution, given their world knowledge, human-preference alignment, and reasoning skills. We evaluate MLLMs as verifiers of agent trajectories across web navigation, computer use, and robotic manipulation, and identify a critical limitation: agreement bias, a strong tendency for MLLMs to favor information in their context window, often generating chains of thought to rationalize flawed behavior. This bias is pervasive across models, resilient to test-time scaling, and can impact several methods using MLLMs as evaluators. Notably, it occurs despite MLLMs showing strong, human-aligned priors on desired behavior. To address this, we propose Self-Grounded Verification (SGV), a lightweight method that enables more effective use of MLLMs' knowledge and reasoning by harnessing their own sampling mechanisms via unconditional and conditional generation. SGV operates in two steps: first, the MLLM is elicited to retrieve broad priors about task completion, independent of the data under evaluation. Then, conditioned on the self-generated priors, it reasons over and evaluates a candidate trajectory. Enhanced with SGV, MLLM verifiers show gains of up to 20 points in accuracy and failure detection rates, and can perform real-time supervision of heterogeneous agents, boosting task completion of a GUI specialist in OSWorld, a diffusion policy in robomimic, and a ReAct agent in VisualWebArena — setting a new state of the art on the benchmark, surpassing the previous best by 48%.

Several breakthroughs in artificial intelligence can be viewed from the lens of search guided by verifiers — functions assigning rewards to agent behavior aligned with desired criteria. However, while domains such as mathematics, programming, and board games benefit from relatively well-defined criteria to evaluate agent behavior, such clarity diminishes in open-ended settings. Evaluation in such scenarios often requires nuanced criteria and reasoning over possibly long sequences of multimodal inputs. Multimodal large language models (MLLMs) emerge as a promising solution to bridge this gap.

We consider two digital agent environments — VisualWebArena and OSWorld — and the long-horizon robot manipulation suite robomimic. Together, they span over 1,200 tasks across varied domains, where evaluating agent behavior demands nuanced criteria and multimodal reasoning.

In the offline setting, we evaluate how often MLLM verifiers match oracle verifiers in evaluation of agent trajectories. In the online setting, we assess the ability of MLLM verifiers to evaluate and provide real-time feedback to guide agents towards task completion.

To address agreement bias, we introduce Self-Grounded Verification (SGV). SGV operates in two steps: first, the model is elicited to retrieve a broad set of priors associated with successful task completion, conditioned only on minimal context to frame the task without access to the data under evaluation. In the second step, conditioned on its self-generated priors, the model then evaluates a candidate trajectory. SGV leads to more balanced and accurate verification across environments and policies, while introducing minimal token overhead and being easy to integrate into pipelines relying on MLLM verifiers.

Intuitively, by modulating unconditional and conditional generation, SGV harnesses MLLMs' own sampling mechanisms to enable more effective use of their world knowledge, alignment, and reasoning capabilities. By conditioning only on essential information in the first step, the model is induced to fully explore its high-dimensional probability distribution to retrieve information relevant to the task at hand and independent of the data under evaluation. In the second step, the model performs more focused generation, reasoning about the trajectory while grounded in its own trajectory-independent beliefs about what success looks like — effectively sampling from a conditional distribution induced by its own priors. We hypothesize that MLLMs, given their vast world knowledge and generalization abilities, can generate priors that are well-aligned with human notions of success, and that by decoupling inference in this manner, these priors can serve as an impartial baseline for grounding the verification.