  Do all instances need inference through the big models for a correct
prediction? Perhaps not; some instances are easy and can be answered correctly
by even small capacity models. This provides opportunities for improving the
computational efficiency of systems. In this work, we present an explorative
study on 'model cascading', a simple technique that utilizes a collection of
models of varying capacities to accurately yet efficiently output predictions.
Through comprehensive experiments in multiple task settings that differ in the
number of models available for cascading (K value), we show that cascading
improves both the computational efficiency and the prediction accuracy. For
instance, in K=3 setting, cascading saves up to 88.93% computation cost and
consistently achieves superior prediction accuracy with an improvement of up to
2.18%. We also study the impact of introducing additional models in the cascade
and show that it further increases the efficiency improvements. Finally, we
hope that our work will facilitate development of efficient NLP systems making
their widespread adoption in real-world applications possible.
