  Both CNN-based and Transformer-based object detection with bounding box
representation have been extensively studied in computer vision and medical
image analysis, but circular object detection in medical images is still
underexplored. Inspired by the recent anchor free CNN-based circular object
detection method (CircleNet) for ball-shape glomeruli detection in renal
pathology, in this paper, we present CircleFormer, a Transformer-based circular
medical object detection with dynamic anchor circles. Specifically, queries
with circle representation in Transformer decoder iteratively refine the
circular object detection results, and a circle cross attention module is
introduced to compute the similarity between circular queries and image
features. A generalized circle IoU (gCIoU) is proposed to serve as a new
regression loss of circular object detection as well. Moreover, our approach is
easy to generalize to the segmentation task by adding a simple segmentation
branch to CircleFormer. We evaluate our method in circular nuclei detection and
segmentation on the public MoNuSeg dataset, and the experimental results show
that our method achieves promising performance compared with the
state-of-the-art approaches. The effectiveness of each component is validated
via ablation studies as well. Our code is released at
https://github.com/zhanghx-iim-ahu/CircleFormer.
