  Learning from feedback reduces the hallucination of multimodal large language
models (MLLMs) by aligning them with human preferences. While traditional
methods rely on labor-intensive and time-consuming manual labeling, recent
approaches employing models as automatic labelers have shown promising results
without human intervention. However, these methods heavily rely on costly
proprietary models like GPT-4V, resulting in scalability issues. Moreover, this
paradigm essentially distills the proprietary models to provide a temporary
solution to quickly bridge the performance gap. As this gap continues to
shrink, the community is soon facing the essential challenge of aligning MLLMs
using labeler models of comparable capability. In this work, we introduce
RLAIF-V, a novel framework that aligns MLLMs in a fully open-source paradigm
for super GPT-4V trustworthiness. RLAIF-V maximally exploits the open-source
feedback from two perspectives, including high-quality feedback data and online
feedback learning algorithm. Extensive experiments on seven benchmarks in both
automatic and human evaluation show that RLAIF-V substantially enhances the
trustworthiness of models without sacrificing performance on other tasks. Using
a 34B model as labeler, RLAIF-V 7B model reduces object hallucination by 82.9\%
and overall hallucination by 42.1\%, outperforming the labeler model.
Remarkably, RLAIF-V also reveals the self-alignment potential of open-source
MLLMs, where a 12B model can learn from the feedback of itself to achieve less
than 29.5\% overall hallucination rate, surpassing GPT-4V (45.9\%) by a large
margin. The results shed light on a promising route to enhance the efficacy of
leading-edge MLLMs.
