Your Agent Aced the Task. Will It Do It Again?

This post introduces 'consistency guidelines' as a new type of guideline within the ALTK-Evolve system. It details a diagnostic tool called the Consistency Analyzer, which resamples an agent's own recorded trajectory to identify 'flip-prone' decision points where the model was close to making a different choice. This diagnostic is used to generate guidelines that halve the consistency gap (from 24.4pp to 12.0pp) without negatively impacting average accuracy. The post also defines and contrasts 'Mean@k' with 'Pass^k' metrics, highlighting the importance of the latter for measuring agent reliability.