  Recent alignment algorithms such as direct preference optimization (DPO) have
been developed to improve the safety of large language models (LLMs) by
training these models to match human behaviors exemplified by preference data.
However, these methods are both computationally intensive and lacking in
controllability and transparency, making them prone to jailbreaking and
inhibiting their widespread use. Furthermore, these tuning-based methods
require large-scale preference data for training and are susceptible to noisy
preference data. In this paper, we introduce a tuning-free alignment
alternative (DeTox) and demonstrate its effectiveness under the use case of
toxicity reduction. Grounded on theory from factor analysis, DeTox is a
sample-efficient model editing approach that identifies a toxic subspace in the
model parameter space and reduces model toxicity by projecting away the
detected subspace. The toxic sub-space is identified by extracting preference
data embeddings from the language model, and removing non-toxic information
from these embeddings. We show that DeTox is more sample-efficient than DPO,
further showcasing greater robustness to noisy data. Finally, we establish both
theoretical and empirical connections between DeTox and DPO, showing that DeTox
can be interpreted as a denoised version of a single DPO step.
