  The right to be forgotten has been legislated in many countries but the
enforcement in machine learning would cause unbearable costs: companies may
need to delete whole models learned from massive resources due to single
individual requests. Existing works propose to remove the knowledge learned
from the requested data via its influence function which is no longer naturally
well-defined in Bayesian inference. This paper proposes a {\it Bayesian
inference forgetting} (BIF) framework to realize the right to be forgotten in
Bayesian inference. In the BIF framework, we develop forgetting algorithms for
variational inference and Markov chain Monte Carlo. We show that our algorithms
can provably remove the influence of single datums on the learned models.
Theoretical analysis demonstrates that our algorithms have guaranteed
generalizability. Experiments of Gaussian mixture models on the synthetic
dataset and Bayesian neural networks on the real-world data verify the
feasibility of our methods. The source code package is available at
\url{https://github.com/fshp971/BIF}.
