  Biomedical Question Answering (BQA) has attracted increasing attention in
recent years due to its promising application prospect. It is a challenging
task because the biomedical questions are professional and usually vary widely.
Existing question answering methods answer all questions with a homogeneous
model, leading to various types of questions competing for the shared
parameters, which will confuse the model decision for each single type of
questions. In this paper, in order to alleviate the parameter competition
problem, we propose a Mixture-of-Expert (MoE) based question answering method
called MoEBQA that decouples the computation for different types of questions
by sparse routing. To be specific, we split a pretrained Transformer model into
bottom and top blocks. The bottom blocks are shared by all the examples, aiming
to capture the general features. The top blocks are extended to an MoE version
that consists of a series of independent experts, where each example is
assigned to a few experts according to its underlying question type. MoEBQA
automatically learns the routing strategy in an end-to-end manner so that each
expert tends to deal with the question types it is expert in. We evaluate
MoEBQA on three BQA datasets constructed based on real examinations. The
results show that our MoE extension significantly boosts the performance of
question answering models and achieves new state-of-the-art performance. In
addition, we elaborately analyze our MoE modules to reveal how MoEBQA works and
find that it can automatically group the questions into human-readable
clusters.
