  Despite recent advancements in language models (LMs), their application to
dialogue management (DM) problems and ability to carry on rich conversations
remain a challenge. We use reinforcement learning (RL) to develop a dialogue
agent that avoids being short-sighted (outputting generic utterances) and
maximizes overall user satisfaction. Most existing RL approaches to DM train
the agent at the word-level, and thus, have to deal with a combinatorially
complex action space even for a medium-size vocabulary. As a result, they
struggle to produce a successful and engaging dialogue even if they are
warm-started with a pre-trained LM. To address this issue, we develop a
RL-based DM using a novel mixture of expert language model (MoE-LM) that
consists of (i) a LM capable of learning diverse semantics for conversation
histories, (ii) a number of {\em specialized} LMs (or experts) capable of
generating utterances corresponding to a particular attribute or personality,
and (iii) a RL-based DM that performs dialogue planning with the utterances
generated by the experts. Our MoE approach provides greater flexibility to
generate sensible utterances with different intents and allows RL to focus on
conversational-level DM. We compare it with SOTA baselines on open-domain
dialogues and demonstrate its effectiveness both in terms of the diversity and
sensibility of the generated utterances and the overall DM performance.
