  State-of-the-art deep learning models have achieved significant performance
levels on various benchmarks. However, the excellent performance comes at a
cost of inefficient computational cost. Light-weight architectures, on the
other hand, achieve moderate accuracies, but at a much more desirable latency.
This paper presents a new method of jointly using the large accurate models
together with the small fast ones. To this end, we propose an Energy-Based
Joint Reasoning (EBJR) framework that adaptively distributes the samples
between shallow and deep models to achieve an accuracy close to the deep model,
but latency close to the shallow one. Our method is applicable to
out-of-the-box pre-trained models as it does not require an architecture change
nor re-training. Moreover, it is easy to use and deploy, especially for cloud
services. Through a comprehensive set of experiments on different down-stream
tasks, we show that our method outperforms strong state-of-the-art approaches
with a considerable margin. In addition, we propose specialized EBJR, an
extension of our method where we create a smaller specialized side model that
performs the target task only partially, but yields an even higher accuracy and
faster inference. We verify the strengths of our methods with both theoretical
and experimental evaluations.
