  Open-domain dialogue systems have started to engage in continuous
conversations with humans. Those dialogue systems are required to be adjusted
to the human interlocutor and evaluated in terms of their perspective. However,
it is questionable whether the current automatic evaluation methods can
approximate the interlocutor's judgments. In this study, we analyzed and
examined what features are needed in an automatic response evaluator from the
interlocutor's perspective. The first experiment on the Hazumi dataset revealed
that interlocutor awareness plays a critical role in making automatic response
evaluation correlate with the interlocutor's judgments. The second experiment
using massive conversations on X (formerly Twitter) confirmed that dialogue
continuity prediction can train an interlocutor-aware response evaluator
without human feedback while revealing the difficulty in evaluating generated
responses compared to human responses.
