  Exposure bias has been regarded as a central problem for auto-regressive
language models (LM). It claims that teacher forcing would cause the test-time
generation to be incrementally distorted due to the training-generation
discrepancy. Although a lot of algorithms have been proposed to avoid teacher
forcing and therefore alleviate exposure bias, there is little work showing how
serious the exposure bias problem actually is. In this work, we focus on the
task of open-ended language generation, propose metrics to quantify the impact
of exposure bias in the aspects of quality, diversity, and consistency. Our key
intuition is that if we feed ground-truth data prefixes (instead of prefixes
generated by the model itself) into the model and ask it to continue the
generation, the performance should become much better because the
training-generation discrepancy in the prefix is removed. Both automatic and
human evaluations are conducted in our experiments. On the contrary to the
popular belief in exposure bias, we find that the the distortion induced by the
prefix discrepancy is limited, and does not seem to be incremental during the
generation. Moreover, our analysis reveals an interesting self-recovery ability
of the LM, which we hypothesize to be countering the harmful effects from
exposure bias.
