  Retrieving proper domain knowledge from an external database lies at the
heart of end-to-end task-oriented dialog systems to generate informative
responses. Most existing systems blend knowledge retrieval with response
generation and optimize them with direct supervision from reference responses,
leading to suboptimal retrieval performance when the knowledge base becomes
large-scale. To address this, we propose to decouple knowledge retrieval from
response generation and introduce a multi-grained knowledge retriever (MAKER)
that includes an entity selector to search for relevant entities and an
attribute selector to filter out irrelevant attributes. To train the retriever,
we propose a novel distillation objective that derives supervision signals from
the response generator. Experiments conducted on three standard benchmarks with
both small and large-scale knowledge bases demonstrate that our retriever
performs knowledge retrieval more effectively than existing methods. Our code
has been made publicly
available.\footnote{https://github.com/18907305772/MAKER}
