  Existing alignment methods share a common topology of information flow, where
reward information is collected from humans, modeled with preference learning,
and used to tune language models. However, this shared topology has not been
systematically characterized, nor have its alternatives been thoroughly
explored, leaving the problems of low data efficiency and unreliable
generalization unaddressed. As a solution, we introduce a theoretical framework
for investigating reward generalization in reinforcement learning from human
feedback (RLHF), focusing on the topology of information flow at both macro and
micro levels. At the macro level, we portray the RLHF information flow as an
autoencoding process over behavior distributions, formalizing the RLHF
objective of distributional consistency between human preference and model
behavior. At the micro level, we present induced Bayesian networks as a theory
of reward generalization in RLHF, introducing fine-grained dataset topologies
into generalization bounds. Combining analysis on both levels, we propose
reward modeling from tree-structured preference information. It is shown to
reduce reward uncertainty by up to $\Theta(\log n/\log\log n)$ times compared
to baselines, where $n$ is the dataset size. Validation on three NLP tasks
shows that our tree-based reward model achieves an average win rate of 65%
against baseline methods, thus improving reward generalization for free via
topology design.
