TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

LLM Reasoning Instruction Following Benchmark

1 post · Oct 2025

What this trail covers

Together AI continues to develop and promote novel evaluation frameworks for LLMs, focusing on their ability to follow instructions throughout their reasoning process. This includes the ReasonIF benchmark, which assesses multilingual reasoning formatting and length control, and aims to improve controllability transparency and safety of LRMs by identifying and addressing failures in reasoning instruction adherence.

All Together AI trails
  1. Large Reasoning Models Fail to Follow Instructions During Reasoning: A Benchmark StudyIntroduced the ReasonIF benchmark dataset to systematically evaluate instruction-following abilities within reasoning traces of Large Reasoning Models (LRMs). The benchmark consists of math and science problems with concrete reasoning instructions across six types: multilinguality, word limit, disclaimer, JSON formatting, uppercase only, and remove commas. Findings indicate that frontier LRMs fail to follow reasoning instructions more than 75% of the time, with performance degrading further as task difficulty increases.