  This paper proposes GuardNN, a secure DNN accelerator that provides
hardware-based protection for user data and model parameters even in an
untrusted environment. GuardNN shows that the architecture and protection can
be customized for a specific application to provide strong confidentiality and
integrity guarantees with negligible overhead. The design of the GuardNN
instruction set reduces the TCB to just the accelerator and allows
confidentiality protection even when the instructions from a host cannot be
trusted. GuardNN minimizes the overhead of memory encryption and integrity
verification by customizing the off-chip memory protection for the known memory
access patterns of a DNN accelerator. GuardNN is prototyped on an FPGA,
demonstrating effective confidentiality protection with ~3% performance
overhead for inference.
