  The Vision Transformer (ViT) architecture has emerged as the backbone of
choice for state-of-the-art deep models for computer vision applications.
However, ViTs are ill-suited for private inference using secure multi-party
computation (MPC) protocols, due to the large number of non-polynomial
operations (self-attention, feed-forward rectifiers, layer normalization). We
propose PriViT, a gradient based algorithm to selectively "Taylorize"
nonlinearities in ViTs while maintaining their prediction accuracy. Our
algorithm is conceptually simple, easy to implement, and achieves improved
performance over existing approaches for designing MPC-friendly transformer
architectures in terms of achieving the Pareto frontier in latency-accuracy. We
confirm these improvements via experiments on several standard image
classification tasks. Public code is available at
https://github.com/NYU-DICE-Lab/privit.
