  Recent sparse detectors with multiple, e.g. six, decoder layers achieve
promising performance but much inference time due to complex heads. Previous
works have explored using dense priors as initialization and built
one-decoder-layer detectors. Although they gain remarkable acceleration, their
performance still lags behind their six-decoder-layer counterparts by a large
margin. In this work, we aim to bridge this performance gap while retaining
fast speed. We find that the architecture discrepancy between dense and sparse
detectors leads to feature conflict, hampering the performance of
one-decoder-layer detectors. Thus we propose Adaptive Sparse Anchor Generator
(ASAG) which predicts dynamic anchors on patches rather than grids in a sparse
way so that it alleviates the feature conflict problem. For each image, ASAG
dynamically selects which feature maps and which locations to predict, forming
a fully adaptive way to generate image-specific anchors. Further, a simple and
effective Query Weighting method eases the training instability from
adaptiveness. Extensive experiments show that our method outperforms
dense-initialized ones and achieves a better speed-accuracy trade-off. The code
is available at \url{https://github.com/iSEE-Laboratory/ASAG}.
