  Ideally, dialogue systems should generate responses that are faithful to the
knowledge contained in relevant documents. However, many models generate
hallucinated responses instead that contradict it or contain unverifiable
information. To mitigate such undesirable behaviour, it has been proposed to
fine-tune a `negative expert' on negative examples and subtract its parameters
from those of a pre-trained model. However, intuitively, this does not take
into account that some parameters are more responsible than others in causing
hallucinations. Thus, we propose to weigh their individual importance via (an
approximation of) the Fisher Information matrix, which measures the uncertainty
of their estimate. We call this method Elastic Weight Removal (EWR). We
evaluate our method -- using different variants of Flan-T5 as a backbone
language model -- on multiple datasets for information-seeking dialogue
generation and compare our method with state-of-the-art techniques for
faithfulness, such as CTRL, Quark, DExperts, and Noisy Channel reranking.
Extensive automatic and human evaluation shows that EWR systematically
increases faithfulness at minor costs in terms of other metrics. However, we
notice that only discouraging hallucinations may increase extractiveness, i.e.
shallow copy-pasting of document spans, which can be undesirable. Hence, as a
second main contribution, we show that our method can be extended to
simultaneously discourage hallucinations and extractive responses. We publicly
release the code for reproducing EWR and all baselines.
