  The knowledge of a deep learning model may be transferred to a student model,
leading to intellectual property infringement or vulnerability propagation.
Detecting such knowledge reuse is nontrivial because the suspect models may not
be white-box accessible and/or may serve different tasks. In this paper, we
propose ModelDiff, a testing-based approach to deep learning model similarity
comparison. Instead of directly comparing the weights, activations, or outputs
of two models, we compare their behavioral patterns on the same set of test
inputs. Specifically, the behavioral pattern of a model is represented as a
decision distance vector (DDV), in which each element is the distance between
the model's reactions to a pair of inputs. The knowledge similarity between two
models is measured with the cosine similarity between their DDVs. To evaluate
ModelDiff, we created a benchmark that contains 144 pairs of models that cover
most popular model reuse methods, including transfer learning, model
compression, and model stealing. Our method achieved 91.7% correctness on the
benchmark, which demonstrates the effectiveness of using ModelDiff for model
reuse detection. A study on mobile deep learning apps has shown the feasibility
of ModelDiff on real-world models.
