  Segment Anything Model (SAM) has garnered significant attention in
segmentation tasks due to their zero-shot generalization ability. However, a
broader application of SAMs to real-world practice has been restricted by their
low inference speed and high computational memory demands, which mainly stem
from the attention mechanism. Existing work concentrated on optimizing the
encoder, yet has not adequately addressed the inefficiency of the attention
mechanism itself, even when distilled to a smaller model, which thus leaves
space for further improvement. In response, we introduce SAM-Lightening, a
variant of SAM, that features a re-engineered attention mechanism, termed
Dilated Flash Attention. It not only facilitates higher parallelism, enhancing
processing efficiency but also retains compatibility with the existing
FlashAttention. Correspondingly, we propose a progressive distillation to
enable an efficient knowledge transfer from the vanilla SAM without costly
training from scratch. Experiments on COCO and LVIS reveal that SAM-Lightening
significantly outperforms the state-of-the-art methods in both run-time
efficiency and segmentation accuracy. Specifically, it can achieve an inference
speed of 7 milliseconds (ms) per image, for images of size 1024*1024 pixels,
which is 30.1 times faster than the vanilla SAM and 2.1 times than the
state-of-the-art. Moreover, it takes only 244MB memory, which is 3.5\% of the
vanilla SAM. The code and weights are available at
https://anonymous.4open.science/r/SAM-LIGHTENING-BC25/.
