  In the context of prediction-as-a-service, concerns about the privacy of the
data and the model have been brought up and tackled via secure inference
protocols. These protocols are built up by using single or multiple
cryptographic tools designed under a variety of different security assumptions.
  In this paper, we introduce SECO, a secure inference protocol that enables a
user holding an input data vector and multiple server nodes deployed with a
split neural network model to collaboratively compute the prediction, without
compromising either party's data privacy. We extend prior work on secure
inference that requires the entire neural network model to be located on a
single server node, to a multi-server hierarchy, where the user communicates to
a gateway server node, which in turn communicates to remote server nodes. The
inference task is split across the server nodes and must be performed over an
encrypted copy of the data vector.
  We adopt multiparty homomorphic encryption and multiparty garbled circuit
schemes, making the system secure against dishonest majority of semi-honest
servers as well as protecting the partial model structure from the user. We
evaluate SECO on multiple models, achieving the reduction of computation and
communication cost for the user, making the protocol applicable to user's
devices with limited resources.
