  In this report we consider the following problem: Given a trained model that
is partially faulty, can we correct its behaviour without having to train the
model from scratch? In other words, can we ``debug" neural networks similar to
how we address bugs in our mathematical models and standard computer code. We
base our approach on the hypothesis that debugging can be treated as a two-task
continual learning problem. In particular, we employ a modified version of a
continual learning algorithm called Orthogonal Gradient Descent (OGD) to
demonstrate, via two simple experiments on the MNIST dataset, that we can
in-fact \textit{unlearn} the undesirable behaviour while retaining the general
performance of the model, and we can additionally \textit{relearn} the
appropriate behaviour, both without having to train the model from scratch.
