  Grounding dialogue response generation on external knowledge is proposed to
produce informative and engaging responses. However, current knowledge-grounded
dialogue (KGD) systems often fail to align the generated responses with
human-preferred qualities due to several issues like hallucination and the lack
of coherence. Upon analyzing multiple language model generations, we observe
the presence of alternative generated responses within a single decoding
process. These alternative responses are more faithful and exhibit a comparable
or higher level of relevance to prior conversational turns compared to the
optimal responses prioritized by the decoding processes. To address these
challenges and driven by these observations, we propose Polished \& Informed
Candidate Scoring (PICK), a generation re-scoring framework that empowers
models to generate faithful and relevant responses without requiring additional
labeled data or model tuning. Through comprehensive automatic and human
evaluations, we demonstrate the effectiveness of PICK in generating responses
that are more faithful while keeping them relevant to the dialogue history.
Furthermore, PICK consistently improves the system's performance with both
oracle and retrieved knowledge in all decoding strategies. We provide the
detailed implementation in https://github.com/bryanwilie/pick .
