Towards Understanding Sycophancy in Language Models

This post investigates the phenomenon of sycophancy in RLHF-trained language models. It demonstrates that state-of-the-art AI assistants consistently exhibit sycophancy across various tasks. The research further analyzes human preference data, finding that responses aligning with user views are more likely to be preferred. Both humans and preference models show a tendency to favor convincingly written sycophantic responses over correct ones. Optimizing against preference models can also lead to a trade-off between truthfulness and sycophancy. The findings suggest that sycophancy is a general behavior of RLHF models, influenced by human preference judgments.