  The ability to reflect on and correct failures is crucial for robotic systems
to interact stably with real-life objects.Observing the generalization and
reasoning capabilities of Multimodal Large Language Models (MLLMs), previous
approaches have aimed to utilize these models to enhance robotic systems
accordingly.However, these methods typically focus on high-level planning
corrections using an additional MLLM, with limited utilization of failed
samples to correct low-level contact poses. To address this gap, we propose an
Autonomous Interactive Correction (AIC) MLLM, which makes use of previous
low-level interaction experiences to correct SE(3) pose predictions.
Specifically, AIC MLLM is initially fine-tuned to acquire both pose prediction
and feedback prompt comprehension abilities.We carefully design two types of
prompt instructions through interactions with objects: 1) visual masks to
highlight unmovable parts for position correction, and 2)textual descriptions
to indicate potential directions for rotation correction.During inference, a
Feedback Information Extraction module is introduced to recognize the failure
cause, allowing AIC MLLM to adaptively correct the pose prediction using the
corresponding prompts. To further enhance manipulation stability, we devise a
Test Time Adaptation strategy that enables AIC MLLM to better adapt to the
current scene configuration.Finally, extensive experiments are conducted in
both simulated and real-world environments to evaluate the proposed method. The
results demonstrate that our AIC MLLM can efficiently correct failure samples
by leveraging interaction experience prompts.Real-world demonstration can be
found at https://sites.google.com/view/aic-mllm
