  Despite tremendous success in many application scenarios, deep learning faces
serious intellectual property (IP) infringement threats. Considering the cost
of designing and training a good model, infringements will significantly
infringe the interests of the original model owner. Recently, many impressive
works have emerged for deep model IP protection. However, they either are
vulnerable to ambiguity attacks, or require changes in the target network
structure by replacing its original normalization layers and hence cause
significant performance drops. To this end, we propose a new passport-aware
normalization formulation, which is generally applicable to most existing
normalization layers and only needs to add another passport-aware branch for IP
protection. This new branch is jointly trained with the target model but
discarded in the inference stage. Therefore it causes no structure change in
the target model. Only when the model IP is suspected to be stolen by someone,
the private passport-aware branch is added back for ownership verification.
Through extensive experiments, we verify its effectiveness in both image and 3D
point recognition models. It is demonstrated to be robust not only to common
attack techniques like fine-tuning and model compression, but also to ambiguity
attacks. By further combining it with trigger-set based methods, both black-box
and white-box verification can be achieved for enhanced security of deep
learning models deployed in real systems. Code can be found at
https://github.com/ZJZAC/Passport-aware-Normalization.
