  The success of deep neural networks (DNNs) is heavily dependent on
computational resources. While DNNs are often employed on cloud servers, there
is a growing need to operate DNNs on edge devices. Edge devices are typically
limited in their computational resources, yet, often multiple edge devices are
deployed in the same environment and can reliably communicate with each other.
In this work we propose to facilitate the application of DNNs on the edge by
allowing multiple users to collaborate during inference to improve their
accuracy. Our mechanism, coined {\em edge ensembles}, is based on having
diverse predictors at each device, which form an ensemble of models during
inference. To mitigate the communication overhead, the users share quantized
features, and we propose a method for aggregating multiple decisions into a
single inference rule. We analyze the latency induced by edge ensembles,
showing that its performance improvement comes at the cost of a minor
additional delay under common assumptions on the communication network. Our
experiments demonstrate that collaborative inference via edge ensembles
equipped with compact DNNs substantially improves the accuracy over having each
user infer locally, and can outperform using a single centralized DNN larger
than all the networks in the ensemble together.
