  Existing dialogue models may encounter scenarios which are not
well-represented in the training data, and as a result generate responses that
are unnatural, inappropriate, or unhelpful. We propose the "Ask an Expert"
framework in which the model is trained with access to an "expert" which it can
consult at each turn. Advice is solicited via a structured dialogue with the
expert, and the model is optimized to selectively utilize (or ignore) it given
the context and dialogue history. In this work the expert takes the form of an
LLM. We evaluate this framework in a mental health support domain, where the
structure of the expert conversation is outlined by pre-specified prompts which
reflect a reasoning strategy taught to practitioners in the field. Blenderbot
models utilizing "Ask an Expert" show quality improvements across all expert
sizes, including those with fewer parameters than the dialogue model itself.
Our best model provides a $\sim 10\%$ improvement over baselines, approaching
human-level scores on "engingingness" and "helpfulness" metrics.
