TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

LLM Evaluation Framework

8 posts · Jul 2025 – Feb 2026

What this trail covers

Together Evaluations provides a fast, flexible way to benchmark LLM response quality using leading open-source judge models. It supports three evaluation modes: Classify, Score, and Compare, enabling users to define custom benchmarks and compare models. The platform integrates with serverless inference APIs and offers a Python SDK for programmatic access. This release enhances the ability to make data-driven decisions on model quality, cost, and performance, supporting the evaluation of open-source, fine-tuned, and proprietary models, including those from OpenAI, Anthropic, and Google. It also supports fine-tuned models deployed via LoRA serverless inference or Dedicated Endpoints. New recipes demonstrate how to fine-tune open-source judges to outperform proprietary models and use GEPA for automated prompt optimization.

All Together AI trails
  1. What do LLMs think when you don't tell them what to think about?This post introduces a study on LLM behavior under near-unconstrained generation, using minimal, topic-neutral prompts. It reveals that different model families exhibit distinct topical preferences (e.g., GPT-OSS favors programming/math, Llama leans literary, DeepSeek produces religious content, Qwen generates multiple-choice questions). The study also quantifies depth differences in technical content and analyzes degenerate text outputs as signals for safety and privacy risks, noting model-specific quirks like Llama generating social media URLs. This work contributes to understanding LLM priors beyond task-specific prompting and offers insights for model auditing and safety analysis.
  2. Fine-tuning open LLM judges to outperform GPT-5.2This post details the engineering effort to fine-tune open-source LLM judges (GPT-OSS 120B, Qwen3 235B Instruct) using Direct Preference Optimization (DPO) on the RewardBench 2 dataset. The fine-tuned models demonstrated improved agreement with human preference labels compared to their baselines and outperformed GPT-5.2 in accuracy, cost-effectiveness (15.3x cheaper for GPT-OSS 120B), and speed (14x faster for GPT-OSS 120B). The analysis includes a category-level breakdown of judge performance and provides practical code for implementation.
  3. Together Evaluations now supports comparing top commercial APIs vs. open source modelsThis post announces the integration of proprietary models (OpenAI, Anthropic, Google) as both judges and target models within the Together Evaluations framework. It also adds support for evaluating Together fine-tuned models (LoRA serverless inference, Dedicated Endpoints). New recipes are provided for fine-tuning open-source judges to outperform proprietary ones (e.g., GPT-5.2) and for using GEPA to automate prompt optimization. The core functionality of uploading data, picking evaluation types (classify, score, compare), describing evaluation criteria via system templates, and configuring models (external with API key, or fine-tuned model IDs) remains the same.
  4. How to choose the right open model for productionThis post provides guidance on selecting open-source LLMs for production workloads. It outlines the benefits of open models (transparency, adaptability, control), legal considerations (licensing, origin), and offers a framework for comparing models based on parameter size and capability tiers (low, medium, high). It details trade-offs between cost, speed, and quality, providing example configurations. The post emphasizes the importance of rigorous evaluation using metrics, a golden dataset, and techniques like LLM-as-a-judge, with a step-by-step process for manual review and prompt modification. It also highlights the role of fine-tuning (LoRA SFT, DPO) for task-specific adaptation.
  5. How to evaluate and benchmark Large Language Models (LLMs)This post details the principles of good LLM benchmarking, including difficulty, diversity, usefulness, reproducibility, and data contamination. It also explores multiple-choice and classification benchmarks like MMLU, GSM8K, and HumanEval, providing practical insights into how to evaluate and benchmark LLMs.
  7. Dynamic AI agent testing for the real world with Collinear Simulations and Together EvalsThis post details the integration of Collinear Simulations' TraitMix product with Together Evals API. TraitMix generates dynamic, persona-driven AI agent interactions by composing user traits (e.g., impatience, confusion, sarcasm) to create realistic conversational data. This data is then fed into Together Evals for automated judgment using LLM-as-a-judge, enabling reproducible and scalable evaluation of AI agents across various domains and behavioral dimensions. The integration facilitates a closed loop for agent improvement through RLHF data generation, regression testing, and failure mode discovery.
  8. Large Reasoning Models Fail to Follow Instructions During Reasoning: A Benchmark StudyThis post introduces the ReasonIF benchmark, a systematic evaluation for assessing instruction-following abilities within the reasoning traces of Large Reasoning Models (LRMs). It highlights that frontier LRMs fail to follow reasoning instructions more than 75% of the time, with performance degrading further as task difficulty increases. The benchmark consists of 300 math and science problems paired with six types of verifiable user-oriented directives (multilinguality, word limit, disclaimer, JSON formatting, uppercase only, remove commas) that models must obey throughout their step-by-step solutions. The post presents findings that demonstrate a significant drop in instruction-following scores within reasoning traces compared to main responses, even for top-performing models.
  9. Together Evaluations: Benchmark Models for Your TasksThis post introduces Together Evaluations, an early preview of a framework for benchmarking LLM response quality using LLM judges. It details three evaluation modes (Classify, Score, Compare) and explains the benefits of LLM-as-a-judge over manual annotation and algorithmic metrics. The post outlines a step-by-step guide for using the platform via UI, API, or Python SDK, and provides links to demos and tutorial notebooks for specific use cases like classifying harmful output, scoring response quality, and A/B testing models/prompts. It clarifies that pricing is based on serverless inference costs for candidate and judge models.