  People tell lies when seeking rewards. Large language models (LLMs) are
aligned to human values with reinforcement learning where they get rewards if
they satisfy human preference. We find that this also induces dishonesty in
helpful and harmless alignment where LLMs tell lies in generating harmless
responses. Using the latest interpreting tools, we detect dishonesty, show how
LLMs can be harmful if their honesty is increased, and analyze such conflicts
at the parameter-level. Given these preliminaries and the hypothesis that
reward-seeking stimulates dishonesty, we theoretically show that the dishonesty
can in-turn decrease the alignment performances and augment reward-seeking
alignment with representation regularization. Extensive results, including
GPT-4 annotated win-rates, perplexities, and cases studies demonstrate that we
can train more honest, helpful, and harmless LLMs. We will make all our codes
and results be open-sourced upon this paper's acceptance.
