  Large neural models are often compressed before deployment. Model compression
is necessary for many practical reasons, such as inference latency, memory
footprint, and energy consumption. Compressed models are assumed to be
miniature versions of corresponding large neural models. However, we question
this belief in our work. We compare compressed models with corresponding large
neural models using four model characteristics: prediction errors, data
representation, data distribution, and vulnerability to adversarial attack. We
perform experiments using the BERT-large model and its five compressed
versions. For all four model characteristics, compressed models significantly
differ from the BERT-large model. Even among compressed models, they differ
from each other on all four model characteristics. Apart from the expected loss
in model performance, there are major side effects of using compressed models
to replace large neural models.
