Study: AI models that consider users' feelings are more likely to make errors

Oxford researchers found that language models fine-tuned for warmer, more empathetic tone were more likely to soften hard truths and echo users’ incorrect claims.

The effect persisted even when models were explicitly told to stay factual, showing a tradeoff between friendliness and accuracy.

The study tested several models, including Llama, Mistral, Qwen, and GPT-4o, and found warmer wording increased agreeable but less truthful responses.

The findings raise concerns that more personable chatbots could mislead users or reinforce false beliefs, especially in emotionally sensitive situations.