  Modern vision models are trained on very large noisy datasets. While these
models acquire strong capabilities, they may not follow the user's intent to
output the desired results in certain aspects, e.g., visual aesthetic,
preferred style, and responsibility. In this paper, we target the realm of
visual aesthetics and aim to align vision models with human aesthetic standards
in a retrieval system. Advanced retrieval systems usually adopt a cascade of
aesthetic models as re-rankers or filters, which are limited to low-level
features like saturation and perform poorly when stylistic, cultural or
knowledge contexts are involved. We find that utilizing the reasoning ability
of large language models (LLMs) to rephrase the search query and extend the
aesthetic expectations can make up for this shortcoming. Based on the above
findings, we propose a preference-based reinforcement learning method that
fine-tunes the vision models to distill the knowledge from both LLMs reasoning
and the aesthetic models to better align the vision models with human
aesthetics. Meanwhile, with rare benchmarks designed for evaluating retrieval
systems, we leverage large multi-modality model (LMM) to evaluate the aesthetic
performance with their strong abilities. As aesthetic assessment is one of the
most subjective tasks, to validate the robustness of LMM, we further propose a
novel dataset named HPIR to benchmark the alignment with human aesthetics.
Experiments demonstrate that our method significantly enhances the aesthetic
behaviors of the vision models, under several metrics. We believe the proposed
algorithm can be a general practice for aligning vision models with human
values.
