  Recent work in open-domain question answering (ODQA) has shown that
adversarial poisoning of the search collection can cause large drops in
accuracy for production systems. However, little to no work has proposed
methods to defend against these attacks. To do so, we rely on the intuition
that redundant information often exists in large corpora. To find it, we
introduce a method that uses query augmentation to search for a diverse set of
passages that could answer the original question but are less likely to have
been poisoned. We integrate these new passages into the model through the
design of a novel confidence method, comparing the predicted answer to its
appearance in the retrieved contexts (what we call Confidence from Answer
Redundancy, i.e. CAR). Together these methods allow for a simple but effective
way to defend against poisoning attacks that provides gains of nearly 20% exact
match across varying levels of data poisoning/knowledge conflicts.
