  Existing methods for image captioning are usually trained by cross entropy
loss, which leads to exposure bias and the inconsistency between the optimizing
function and evaluation metrics. Recently it has been shown that these two
issues can be addressed by incorporating techniques from reinforcement
learning, where one of the popular techniques is the advantage actor-critic
algorithm that calculates per-token advantage by estimating state value with a
parametrized estimator at the cost of introducing estimation bias. In this
paper, we estimate state value without using a parametrized value estimator.
With the properties of image captioning, namely, the deterministic state
transition function and the sparse reward, state value is equivalent to its
preceding state-action value, and we reformulate advantage function by simply
replacing the former with the latter. Moreover, the reformulated advantage is
extended to n-step, which can generally increase the absolute value of the mean
of reformulated advantage while lowering variance. Then two kinds of rollout
are adopted to estimate state-action value, which we call self-critical n-step
training. Empirically we find that our method can obtain better performance
compared to the state-of-the-art methods that use the sequence level advantage
and parametrized estimator respectively on the widely used MSCOCO benchmark.
