Models That Know How Evaluations Are Designed Score Safer

Researchers found that models fine-tuned on synthetic text about evaluation design scored significantly safer on six AI safety benchmarks.

The effect remained even when explicit signs that models knew they were being evaluated were removed.

This suggests benchmark results may be inflated by evaluation meta-knowledge, not only by real safety improvements.

The finding raises concerns about how reliably current safety benchmarks measure genuine model behavior.