  Transformer-based detectors have shown success in computer vision tasks with
natural images. These models, exemplified by the Deformable DETR, are optimized
through complex engineering strategies tailored to the typical characteristics
of natural scenes. However, medical imaging data presents unique challenges
such as extremely large image sizes, fewer and smaller regions of interest, and
object classes which can be differentiated only through subtle differences.
This study evaluates the applicability of these transformer-based design
choices when applied to a screening mammography dataset that represents these
distinct medical imaging data characteristics. Our analysis reveals that common
design choices from the natural image domain, such as complex encoder
architectures, multi-scale feature fusion, query initialization, and iterative
bounding box refinement, do not improve and sometimes even impair object
detection performance in medical imaging. In contrast, simpler and shallower
architectures often achieve equal or superior results. This finding suggests
that the adaptation of transformer models for medical imaging data requires a
reevaluation of standard practices, potentially leading to more efficient and
specialized frameworks for medical diagnosis.
