  Collaborative inference enables resource-constrained edge devices to make
inferences by uploading inputs (e.g., images) to a server (i.e., cloud) where
the heavy deep learning models run. While this setup works cost-effectively for
successful inferences, it severely underperforms when the model faces input
samples on which the model was not trained (known as Out-of-Distribution (OOD)
samples). If the edge devices could, at least, detect that an input sample is
an OOD, that could potentially save communication and computation resources by
not uploading those inputs to the server for inference workload. In this paper,
we propose a novel lightweight OOD detection approach that mines important
features from the shallow layers of a pretrained CNN model and detects an input
sample as ID (In-Distribution) or OOD based on a distance function defined on
the reduced feature space. Our technique (a) works on pretrained models without
any retraining of those models, and (b) does not expose itself to any OOD
dataset (all detection parameters are obtained from the ID training dataset).
To this end, we develop EARLIN (EARLy OOD detection for Collaborative
INference) that takes a pretrained model and partitions the model at the OOD
detection layer and deploys the considerably small OOD part on an edge device
and the rest on the cloud. By experimenting using real datasets and a prototype
implementation, we show that our technique achieves better results than other
approaches in terms of overall accuracy and cost when tested against popular
OOD datasets on top of popular deep learning models pretrained on benchmark
datasets.
