  Existing open-source helpfulness preference datasets do not specify what
makes some responses more helpful and others less so. Models trained on these
datasets can incidentally learn to model dataset artifacts (e.g. preferring
longer but unhelpful responses only due to their length). To alleviate this
problem, we collect HelpSteer, a multi-attribute helpfulness dataset annotated
for the various aspects that make responses helpful. Specifically, our
37k-sample dataset has annotations for correctness, coherence, complexity, and
verbosity in addition to overall helpfulness of responses. Training Llama 2 70B
using the HelpSteer dataset with SteerLM technique produces a model that scores
7.54 on MT Bench, which is currently the highest score for open models that do
not require training data from more powerful models (e.g. GPT4). We release
this dataset with CC-BY-4.0 license at
https://huggingface.co/datasets/nvidia/HelpSteer
