  Outsourcing deep neural networks (DNNs) inference tasks to an untrusted cloud
raises data privacy and integrity concerns. While there are many techniques to
ensure privacy and integrity for polynomial-based computations, DNNs involve
non-polynomial computations. To address these challenges, several
privacy-preserving and verifiable inference techniques have been proposed based
on replacing the non-polynomial activation functions such as the rectified
linear unit (ReLU) function with polynomial activation functions. Such
techniques usually require polynomials with integer coefficients or polynomials
over finite fields. Motivated by such requirements, several works proposed
replacing the ReLU function with the square function. In this work, we
empirically show that the square function is not the best degree-2 polynomial
that can replace the ReLU function even when restricting the polynomials to
have integer coefficients. We instead propose a degree-2 polynomial activation
function with a first order term and empirically show that it can lead to much
better models. Our experiments on the CIFAR and Tiny ImageNet datasets on
various architectures such as VGG-16 show that our proposed function improves
the test accuracy by up to 10.4% compared to the square function.
