  Since training deep neural networks takes significant computational
resources, extending the training dataset with new data is difficult, as it
typically requires complete retraining. Moreover, specific applications do not
allow costly retraining due to time or computational constraints. We address
this issue by proposing a novel Bayesian update method for deep neural networks
by using a last-layer Laplace approximation. Concretely, we leverage
second-order optimization techniques on the Gaussian posterior distribution of
a Laplace approximation, computing the inverse Hessian matrix in closed form.
This way, our method allows for fast and effective updates upon the arrival of
new data in a stationary setting. A large-scale evaluation study across
different data modalities confirms that our updates are a fast and competitive
alternative to costly retraining. Furthermore, we demonstrate its applicability
in a deep active learning scenario by using our update to improve existing
selection strategies.
