  State-of-the-art semantic segmentation models are characterized by high
parameter counts and slow inference times, making them unsuitable for
deployment in resource-constrained environments. To address this challenge, we
propose \textsc{Auto-Compressing Subset Pruning}, \acosp, as a new online
compression method. The core of \acosp consists of learning a channel selection
mechanism for individual channels of each convolution in the segmentation model
based on an effective temperature annealing schedule. We show a crucial
interplay between providing a high-capacity model at the beginning of training
and the compression pressure forcing the model to compress concepts into
retained channels. We apply \acosp to \segnet and \pspnet architectures and
show its success when trained on the \camvid, \city, \voc, and \ade datasets.
The results are competitive with existing baselines for compression of
segmentation models at low compression ratios and outperform them significantly
at high compression ratios, yielding acceptable results even when removing more
than $93\%$ of the parameters. In addition, \acosp is conceptually simple, easy
to implement, and can readily be generalized to other data modalities, tasks,
and architectures. Our code is available at
\url{https://github.com/merantix/acosp}.
