  Large Language Models (LLMs) have achieved great success in solving difficult
tasks across many domains, but such success comes with a high computation cost,
and inference latency. As developers and third parties customize these models,
the need to provide efficient inference has increased. Many efforts have
attempted to reduce inference cost through model compression techniques such as
pruning and distillation. However, these techniques either require labeled
data, or are time-consuming as they require the compressed model to be
retrained to regain accuracy. In this paper, we propose a gradient-free
structured pruning framework that uses only unlabeled data. An evaluation on
the GLUE and SQuAD benchmarks using BERT$_{BASE}$ and DistilBERT illustrates
the effectiveness of the proposed approach. By only using the weights of the
pre-trained model and unlabeled data, in a matter of a few minutes on a single
GPU, up to 40% of the original FLOP count can be reduced with less than a 4%
accuracy loss across all tasks considered.
