  With rapid progress made in language qualities such as fluency and
consistency via large language models (LLMs), there has been increasing
interest in assessing alignment with diverse human preferences. Traditional
metrics heavily rely on lexical similarity with human-written references and
have been observed to suffer from a poor correlation with human evaluation.
Furthermore, they ignore the diverse preferences of humans, a key aspect in
evaluating open-ended tasks like story generation. Inspired by these
challenges, we introduce an interpretable open-ended evaluation framework PerSE
to assess the alignment with a specific human preference. It is tuned to deduce
the specific preference from a given personal profile and evaluate the
alignment between the generation and the personal preference. PerSE also
explains its assessment by a detailed comment or several fine-grained scores.
This enhances its interpretability, making it more suitable to tailor a
personalized generation. Our 13B LLaMA-2-based PerSE shows a 15.8% increase in
Kendall correlation and a 13.7% rise in accuracy on zero-shot reviewers
compared to GPT-4. It also outperforms GPT-4 by 46.01% in the Kendall
correlation on new domains, indicating its transferability.
