  Vision transformer based models bring significant improvements for image
segmentation tasks. Although these architectures offer powerful capabilities
irrespective of specific segmentation tasks, their use of computational
resources can be taxing on deployed devices. One way to overcome this challenge
is by adapting the computation level to the specific needs of the input image
rather than the current one-size-fits-all approach. To this end, we introduce
ECO-M2F or EffiCient TransfOrmer Encoders for Mask2Former-style models. Noting
that the encoder module of M2F-style models incur high resource-intensive
computations, ECO-M2F provides a strategy to self-select the number of hidden
layers in the encoder, conditioned on the input image. To enable this
self-selection ability for providing a balance between performance and
computational efficiency, we present a three step recipe. The first step is to
train the parent architecture to enable early exiting from the encoder. The
second step is to create an derived dataset of the ideal number of encoder
layers required for each training example. The third step is to use the
aforementioned derived dataset to train a gating network that predicts the
number of encoder layers to be used, conditioned on the input image.
Additionally, to change the computational-accuracy tradeoff, only steps two and
three need to be repeated which significantly reduces retraining time.
Experiments on the public datasets show that the proposed approach reduces
expected encoder computational cost while maintaining performance, adapts to
various user compute resources, is flexible in architecture configurations, and
can be extended beyond the segmentation task to object detection.
