  Recent work on stealing machine learning (ML) models from inference engines
with physical side-channel attacks warrant an urgent need for effective
side-channel defenses. This work proposes the first $\textit{fully-masked}$
neural network inference engine design.
  Masking uses secure multi-party computation to split the secrets into random
shares and to decorrelate the statistical relation of secret-dependent
computations to side-channels (e.g., the power draw). In this work, we
construct secure hardware primitives to mask $\textit{all}$ the linear and
non-linear operations in a neural network. We address the challenge of masking
integer addition by converting each addition into a sequence of XOR and AND
gates and by augmenting Trichina's secure Boolean masking style. We improve the
traditional Trichina's AND gates by adding pipelining elements for better
glitch-resistance and we architect the whole design to sustain a throughput of
1 masked addition per cycle.
  We implement the proposed secure inference engine on a Xilinx Spartan-6
(XC6SLX75) FPGA. The results show that masking incurs an overhead of 3.5\% in
latency and 5.9$\times$ in area. Finally, we demonstrate the security of the
masked design with 2M traces.
