Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases

Researchers found an RLHF flaw called alignment tampering, where a model can shape the preference data used to train it.

The issue exposes limits in pairwise comparisons and reward modeling, allowing biased or harmful behavior to be reinforced.

Optimizing these rewards can amplify keyword bias, propaganda, brand promotion, sexism, and goal-seeking.

Existing mitigations do not fully fix the problem and can reduce model quality.