  We introduce a new task called Adaptable Error Detection (AED), which aims to
identify behavior errors in few-shot imitation (FSI) policies based on visual
observations in novel environments. The potential to cause serious damage to
surrounding areas limits the application of FSI policies in real-world
scenarios. Thus, a robust system is necessary to notify operators when FSI
policies are inconsistent with the intent of demonstrations. This task
introduces three challenges: (1) detecting behavior errors in novel
environments, (2) identifying behavior errors that occur without revealing
notable changes, and (3) lacking complete temporal information of the rollout
due to the necessity of online detection. However, the existing benchmarks
cannot support the development of AED because their tasks do not present all
these challenges. To this end, we develop a cross-domain AED benchmark,
consisting of 322 base and 153 novel environments. Additionally, we propose
Pattern Observer (PrObe) to address these challenges. PrObe is equipped with a
powerful pattern extractor and guided by novel learning objectives to parse
discernible patterns in the policy feature representations of normal or error
states. Through our comprehensive evaluation, PrObe demonstrates superior
capability to detect errors arising from a wide range of FSI policies,
consistently surpassing strong baselines. Moreover, we conduct detailed
ablations and a pilot study on error correction to validate the effectiveness
of the proposed architecture design and the practicality of the AED task,
respectively.
