  Language models achieve impressive results in tasks involving complex
multistep reasoning, but scaling these capabilities further traditionally
requires expensive collection of more annotated data. In this work, we explore
the potential of improving the capabilities of language models without new
data, merely using automated feedback to the validity of their predictions in
arithmetic reasoning (self-training). We find that models can substantially
improve in both single-round (offline) and online self-training. In the offline
setting, supervised methods are able to deliver gains comparable to preference
optimization, but in online self-training, preference optimization shows to
largely outperform supervised training thanks to superior stability and
robustness on unseen types of problems.
