  In this paper, we propose a general approach to optimize anchor boxes for
object detection. Nowadays, anchor boxes are widely adopted in state-of-the-art
detection frameworks. However, these frameworks usually pre-define anchor box
shapes in heuristic ways and fix the sizes during training. To improve the
accuracy and reduce the effort of designing anchor boxes, we propose to
dynamically learn the anchor shapes, which allows the anchors to automatically
adapt to the data distribution and the network learning capability. The
learning approach can be easily implemented with stochastic gradient descent
and can be plugged into any anchor box-based detection framework. The extra
training cost is almost negligible and it has no impact on the inference time
or memory cost. Exhaustive experiments demonstrate that the proposed anchor
optimization method consistently achieves significant improvement ($\ge 1\%$
mAP absolute gain) over the baseline methods on several benchmark datasets
including Pascal VOC 07+12, MS COCO and Brainwash. Meanwhile, the robustness is
also verified towards different anchor initialization methods and the number of
anchor shapes, which greatly simplifies the problem of anchor box design.
