  Aligning large language models (LLMs) with human objectives is crucial for
real-world applications. However, fine-tuning LLMs for alignment often suffers
from unstable training and requires substantial computing resources. Test-time
alignment techniques, such as prompting and guided decoding, do not modify the
underlying model, and their performance remains dependent on the original
model's capabilities. To address these challenges, we propose aligning LLMs
through representation editing. The core of our method is to view a pre-trained
autoregressive LLM as a discrete-time stochastic dynamical system. To achieve
alignment for specific objectives, we introduce external control signals into
the state space of this language dynamical system. We train a value function
directly on the hidden states according to the Bellman equation, enabling
gradient-based optimization to obtain the optimal control signals at test time.
Our experiments demonstrate that our method outperforms existing test-time
alignment techniques while requiring significantly fewer resources compared to
fine-tuning methods.
