  Regulations introduced by General Data Protection Regulation (GDPR) in the EU
or California Consumer Privacy Act (CCPA) in the US have included provisions on
the \textit{right to be forgotten} that mandates industry applications to
remove data related to an individual from their systems. In several real world
industry applications that use Machine Learning to build models on user data,
such mandates require significant effort both in terms of data cleansing as
well as model retraining while ensuring the models do not deteriorate in
prediction quality due to removal of data. As a result, continuous removal of
data and model retraining steps do not scale if these applications receive such
requests at a very high frequency. Recently, a few researchers proposed the
idea of \textit{Machine Unlearning} to tackle this challenge. Despite the
significant importance of this task, the area of Machine Unlearning is
under-explored in Natural Language Processing (NLP) tasks. In this paper, we
explore the Unlearning framework on various GLUE tasks \cite{Wang:18}, such as,
QQP, SST and MNLI. We propose computationally efficient approaches (SISA-FC and
SISA-A) to perform \textit{guaranteed} Unlearning that provides significant
reduction in terms of both memory (90-95\%), time (100x) and space consumption
(99\%) in comparison to the baselines while keeping model performance constant.
