  Numerous algorithms have been proposed to $\textit{align}$ language models to
remove undesirable behaviors. However, the challenges associated with a very
large state space and creating a proper reward function often result in various
jailbreaks. Our paper aims to examine this effect of reward in the controlled
setting of positive sentiment language generation. Instead of online training
of a reward model based on human feedback, we employ a statically learned
sentiment classifier. We also consider a setting where our model's weights and
activations are exposed to an end-user after training. We examine a pretrained
GPT-2 through the lens of mechanistic interpretability before and after
proximal policy optimization (PPO) has been applied to promote positive
sentiment responses. Using these insights, we (1) attempt to "hack" the PPO-ed
model to generate negative sentiment responses and (2) add a term to the reward
function to try and alter `negative' weights.
