  Localization of anatomical landmarks is essential for clinical diagnosis,
treatment planning, and research. In this paper, we propose a novel deep
network, named feature aggregation and refinement network (FARNet), for the
automatic detection of anatomical landmarks. To alleviate the problem of
limited training data in the medical domain, our network adopts a deep network
pre-trained on natural images as the backbone network and several popular
networks have been compared. Our FARNet also includes a multi-scale feature
aggregation module for multi-scale feature fusion and a feature refinement
module for high-resolution heatmap regression. Coarse-to-fine supervisions are
applied to the two modules to facilitate the end-to-end training. We further
propose a novel loss function named Exponential Weighted Center loss for
accurate heatmap regression, which focuses on the losses from the pixels near
landmarks and suppresses the ones from far away. Our network has been evaluated
on three publicly available anatomical landmark detection datasets, including
cephalometric radiographs, hand radiographs, and spine radiographs, and
achieves state-of-art performances on all three datasets. Code is available at:
\url{https://github.com/JuvenileInWind/FARNet}
