  It is always well believed that parsing an image into constituent visual
patterns would be helpful for understanding and representing an image.
Nevertheless, there has not been evidence in support of the idea on describing
an image with a natural-language utterance. In this paper, we introduce a new
design to model a hierarchy from instance level (segmentation), region level
(detection) to the whole image to delve into a thorough image understanding for
captioning. Specifically, we present a HIerarchy Parsing (HIP) architecture
that novelly integrates hierarchical structure into image encoder. Technically,
an image decomposes into a set of regions and some of the regions are resolved
into finer ones. Each region then regresses to an instance, i.e., foreground of
the region. Such process naturally builds a hierarchal tree. A tree-structured
Long Short-Term Memory (Tree-LSTM) network is then employed to interpret the
hierarchal structure and enhance all the instance-level, region-level and
image-level features. Our HIP is appealing in view that it is pluggable to any
neural captioning models. Extensive experiments on COCO image captioning
dataset demonstrate the superiority of HIP. More remarkably, HIP plus a
top-down attention-based LSTM decoder increases CIDEr-D performance from 120.1%
to 127.2% on COCO Karpathy test split. When further endowing instance-level and
region-level features from HIP with semantic relation learnt through Graph
Convolutional Networks (GCN), CIDEr-D is boosted up to 130.6%.
