  Large-scale generative models are shown to be useful for sampling meaningful
candidate solutions, yet they often overlook task constraints and user
preferences. Their full power is better harnessed when the models are coupled
with external verifiers and the final solutions are derived iteratively or
progressively according to the verification feedback. In the context of
embodied AI, verification often solely involves assessing whether goal
conditions specified in the instructions have been met. Nonetheless, for these
agents to be seamlessly integrated into daily life, it is crucial to account
for a broader range of constraints and preferences beyond bare task success
(e.g., a robot should grasp bread with care to avoid significant deformations).
However, given the unbounded scope of robot tasks, it is infeasible to
construct scripted verifiers akin to those used for explicit-knowledge tasks
like the game of Go and theorem proving. This begs the question: when no sound
verifier is available, can we use large vision and language models (VLMs),
which are approximately omniscient, as scalable Behavior Critics to catch
undesirable robot behaviors in videos? To answer this, we first construct a
benchmark that contains diverse cases of goal-reaching yet undesirable robot
policies. Then, we comprehensively evaluate VLM critics to gain a deeper
understanding of their strengths and failure modes. Based on the evaluation, we
provide guidelines on how to effectively utilize VLM critiques and showcase a
practical way to integrate the feedback into an iterative process of policy
refinement. The dataset and codebase are released at:
https://guansuns.github.io/pages/vlm-critic.
