  Due to increasing privacy concerns, neural network (NN) based secure
inference (SI) schemes that simultaneously hide the client inputs and server
models attract major research interests. While existing works focused on
developing secure protocols for NN-based SI, in this work, we take a different
approach. We propose NASS, an integrated framework to search for tailored NN
architectures designed specifically for SI. In particular, we propose to model
cryptographic protocols as design elements with associated reward functions.
The characterized models are then adopted in a joint optimization with
predicted hyperparameters in identifying the best NN architectures that balance
prediction accuracy and execution efficiency. In the experiment, it is
demonstrated that we can achieve the best of both worlds by using NASS, where
the prediction accuracy can be improved from 81.6% to 84.6%, while the
inference runtime is reduced by 2x and communication bandwidth by 1.9x on the
CIFAR-10 dataset.
