  The recently introduced Segment Anything Model (SAM) combines a clever
architecture and large quantities of training data to obtain remarkable image
segmentation capabilities. However, it fails to reproduce such results for
Out-Of-Distribution (OOD) domains such as medical images. Moreover, while SAM
is conditioned on either a mask or a set of points, it may be desirable to have
a fully automatic solution. In this work, we replace SAM's conditioning with an
encoder that operates on the same input image. By adding this encoder and
without further fine-tuning SAM, we obtain state-of-the-art results on multiple
medical images and video benchmarks. This new encoder is trained via gradients
provided by a frozen SAM. For inspecting the knowledge within it, and providing
a lightweight segmentation solution, we also learn to decode it into a mask by
a shallow deconvolution network.
