Large Reasoning Models Fail to Follow Instructions During Reasoning: A Benchmark Study
This post introduces the ReasonIF benchmark, a systematic evaluation for assessing instruction-following abilities within the reasoning traces of Large Reasoning Models (LRMs). It highlights that frontier LRMs fail to follow reasoning instructions more than 75% of the time, with performance degrading further as task difficulty increases. The benchmark consists of 300 math and science problems paired with six types of verifiable user-oriented directives (multilinguality, word limit, disclaimer, JSON formatting, uppercase only, remove commas) that models must obey throughout their step-by-step solutions. The post presents findings that demonstrate a significant drop in instruction-following scores within reasoning traces compared to main responses, even for top-performing models.
More from the LLM Evaluation Framework trail
- What do LLMs think when you don't tell them what to think about?
- Fine-tuning open LLM judges to outperform GPT-5.2
- Together Evaluations now supports comparing top commercial APIs vs. open source models
- How to choose the right open model for production
- How to evaluate and benchmark Large Language Models (LLMs)