  Due to the surge of cloud-assisted AI services, the problem of designing
resilient prediction serving systems that can effectively cope with
stragglers/failures and minimize response delays has attracted much interest.
The common approach for tackling this problem is replication which assigns the
same prediction task to multiple workers. This approach, however, is very
inefficient and incurs significant resource overheads. Hence, a learning-based
approach known as parity model (ParM) has been recently proposed which learns
models that can generate parities for a group of predictions in order to
reconstruct the predictions of the slow/failed workers. While this
learning-based approach is more resource-efficient than replication, it is
tailored to the specific model hosted by the cloud and is particularly suitable
for a small number of queries (typically less than four) and tolerating very
few (mostly one) number of stragglers. Moreover, ParM does not handle Byzantine
adversarial workers. We propose a different approach, named Approximate Coded
Inference (ApproxIFER), that does not require training of any parity models,
hence it is agnostic to the model hosted by the cloud and can be readily
applied to different data domains and model architectures. Compared with
earlier works, ApproxIFER can handle a general number of stragglers and scales
significantly better with the number of queries. Furthermore, ApproxIFER is
robust against Byzantine workers. Our extensive experiments on a large number
of datasets and model architectures also show significant accuracy improvement
by up to 58% over the parity model approaches.
