  Real-world natural language processing (NLP) models need to be continually
updated to fix the prediction errors in out-of-distribution (OOD) data streams
while overcoming catastrophic forgetting. However, existing continual learning
(CL) problem setups cannot cover such a realistic and complex scenario. In
response to this, we propose a new CL problem formulation dubbed continual
model refinement (CMR). Compared to prior CL settings, CMR is more practical
and introduces unique challenges (boundary-agnostic and non-stationary
distribution shift, diverse mixtures of multiple OOD data clusters,
error-centric streams, etc.). We extend several existing CL approaches to the
CMR setting and evaluate them extensively. For benchmarking and analysis, we
propose a general sampling algorithm to obtain dynamic OOD data streams with
controllable non-stationarity, as well as a suite of metrics measuring various
aspects of online performance. Our experiments and detailed analysis reveal the
promise and challenges of the CMR problem, supporting that studying CMR in
dynamic OOD streams can benefit the longevity of deployed NLP models in
production.
