  Users often ask dialogue systems ambiguous questions that require
clarification. We show that current language models rarely ask users to clarify
ambiguous questions and instead provide incorrect answers. To address this, we
introduce CLAM: a framework for getting language models to selectively ask for
clarification about ambiguous user questions. In particular, we show that we
can prompt language models to detect whether a given question is ambiguous,
generate an appropriate clarifying question to ask the user, and give a final
answer after receiving clarification. We also show that we can simulate users
by providing language models with privileged information. This lets us
automatically evaluate multi-turn clarification dialogues. Finally, CLAM
significantly improves language models' accuracy on mixed ambiguous and
unambiguous questions relative to SotA.
