  Reinforcement learning from human feedback (RLHF) is fundamentally limited by
the capacity of humans to correctly evaluate model output. To improve human
evaluation ability and overcome that limitation this work trains "critic"
models that help humans to more accurately evaluate model-written code. These
critics are themselves LLMs trained with RLHF to write natural language
feedback highlighting problems in code from real-world assistant tasks. On code
containing naturally occurring LLM errors model-written critiques are preferred
over human critiques in 63% of cases, and human evaluation finds that models
catch more bugs than human contractors paid for code review. We further confirm
that our fine-tuned LLM critics can successfully identify hundreds of errors in
ChatGPT training data rated as "flawless", even though the majority of those
tasks are non-code tasks and thus out-of-distribution for the critic model.
Critics can have limitations of their own, including hallucinated bugs that
could mislead humans into making mistakes they might have otherwise avoided,
but human-machine teams of critics and contractors catch similar numbers of
bugs to LLM critics while hallucinating less than LLMs alone.
