TechPrufer
Get started
Together AI
Together AI logoTogether AI · Feature trail

LLM Model Performance and Cost Comparison

4 posts · Aug 2025 – Sep 2026

What this trail covers

Together AI continues to provide in-depth comparisons of LLM performance and cost, now detailing the trade-offs between DeepSeek V4 Flash 0731 and GPT-5.6 Luna on the DeepSWE benchmark. The analysis highlights DeepSeek V4 Flash's strength in cost-effectiveness and cleaner failure modes, while GPT-5.6 Luna excels in accuracy and broader domain coverage. The post advocates for a cascaded routing strategy, using DeepSeek V4 Flash as a first stage to achieve superior accuracy at a lower cost than us. This post also compares OpenAI's new gpt-oss-120B model against o4-mini across practical tests, showing gpt-oss-120B's strengths in code generation and instruction following, while o4-mini excels in creative SVG generation. The results reinforce the value of open-source models for customization, cost-efficiency, and deployment flexibility.

All Together AI trails
  1. The Open Source AI StackThis post introduces the 'MIGHT' stack for open-source AI development, breaking down the components: Model, Inference, Gateways/Routers, Harness, and Tools. It emphasizes the importance of selecting models based on task specificity and cost, contrasting large models like Kimi K3 for complex tasks with smaller models like GLM 5.3 Flash for well-defined ones. It also highlights the role of inference providers and gateways in accessing diverse models and the concept of treating models as tools within a flexible development workflow.
  2. GLM-5.3 vs. GLM-5.3 Flash on DeepSWE: Cost, Coding, and RoutingThis post details a comparative analysis between GLM-5.3 and GLM-5.3 Flash on the DeepSWE benchmark. It quantifies the performance (pass@1, pass@k) and cost differences, showing that GLM-5.3 Flash is 17x cheaper per rollout and faster, with a minimal drop in best-of-k quality (2.6 points at pass@4). The analysis reveals that distillation primarily impacts consistency and the ability to recover from flaky attempts, rather than core capability. A cascade strategy is proposed, running GLM-5.3 Flash first and escalating to GLM-5.3 for rejected answers, achieving 80.9% task completion at $1.70 per task. A key regression identified is the increased baseline test breakage rate for GLM-5.3 Flash (6.9% vs 4.4%).
  3. GLM-5.3 vs. Claude Fable 5 on DeepSWE: Cost, Coding, and RoutingThis post compares GLM-5.3 and Claude Fable 5 on the DeepSWE benchmark, focusing on accuracy, cost, and specific task/language performance. GLM-5.3 demonstrates comparable first-shot accuracy to Fable 5 but at a 5.4x lower cost, with better performance on retries and wider domain coverage. Fable 5 shows a distinct advantage in Rust and serialization tasks. The analysis highlights their high per-task correlation, suggesting they are near-substitutes, and recommends GLM-5.3 as the default with Fable as a specialized escalation.
  4. GLM-5.3 vs. GPT-5.6 Sol on DeepSWE: Cost, Coding, and RoutingThis post provides a detailed comparative analysis of GLM-5.3 and GPT-5.6 Sol on the DeepSWE benchmark, focusing on pass@k metrics, cost per rollout, failure modes, and domain/language-specific performance. It quantifies the trade-offs: GLM-5.3 offers better value and reach (pass@4, coverage) at lower cost and with cleaner failure profiles (fewer regressions), while GPT-5.6 Sol leads in single-shot accuracy (pass@1), speed, and reliability, particularly in protocol conformance. The key technical contribution is the demonstration and quantification of a cascaded routing strategy (GLM-5.3 first, then GPT-5.6 Sol) that achieves 85.9% task completion at $6.61 per task, outperforming either model alone and a perfect one-shot oracle router.
  5. DeepSeek V4 Pro 0813 vs GPT-5.6 Sol on DeepSWE: Cost, Coding, and RoutingThis post provides a detailed comparative analysis of DeepSeek V4 Pro 0813 and GPT-5.6 Sol on the DeepSWE benchmark, focusing on performance metrics (pass@1, pass@k), cost per rollout, speed, token usage, and failure modes. It quantifies the trade-offs between the two models, demonstrating that a cascaded approach (DeepSeek V4 Pro 0813 first, escalating to GPT-5.6 Sol on failure) achieves higher overall task completion (83.0%) at a significantly lower cost ($3.35 per task) compared to using GPT-5.6 Sol alone (72.7% at $8.37 per task). It also details specific failure modes for each model (e.g., GPT-5.6 Sol's regression rate) and performance differences across task domains and programming languages.
  7. DeepSeek V4 Pro 0813 vs Claude Fable 5 on DeepSWE: Cost, Coding, and RoutingThis post provides a detailed comparison of DeepSeek V4 Pro 0813 and Claude Fable 5 on the DeepSWE benchmark, focusing on cost, coding performance, and routing strategies. It quantifies the performance differences (pass@1, pass@k), cost per rollout, and failure modes of each model. Key findings include Fable's lead in single-shot accuracy and Rust/serialization tasks, while Pro excels in retry-tolerant tasks, concurrency, and durability, and is 90x cheaper. The post demonstrates that a cascaded routing strategy (Pro first, then Fable) achieves higher overall accuracy (82.7%) at a significantly lower cost ($8.28 per task) compared to using Fable alone (69.7% at $21.63 per task). It also highlights the low per-task correlation (0.39) between the two models, indicating genuine complementarity.
  8. DeepSeek-V4 Flash 0731 vs GPT-5.6 Luna on DeepSWE: Cost and CodingThis post introduces a detailed comparison between DeepSeek-V4 Flash 0731 and GPT-5.6 Luna on the DeepSWE benchmark, focusing on accuracy, cost, and failure modes. It quantifies the performance gap (14 points in pass@1), cost difference (6x per rollout, 4.8x value edge for DeepSeek), and failure characteristics (DeepSeek fails more cleanly). Crucially, it demonstrates a cascaded approach where DeepSeek-V4 Flash is used as a first-stage model, escalating to GPT-5.6 Luna only on failure. This cascade achieves 78.9% accuracy at $0.385 per task, outperforming GPT-5.6 Luna alone (67.2% at $0.61) in both accuracy and cost. The analysis also breaks down performance by task domain and programming language, highlighting DeepSeek's strengths in query/config and weaknesses in JavaScript.
  9. What do LLMs think when you don't tell them what to think about?This post introduces a study on near-unconstrained LLM generation, revealing distinct topical preferences and depth differences between model families (GPT-OSS, Llama, DeepSeek, Qwen). It analyzes model behavior with minimal, topic-neutral prompts, observing that GPT-OSS favors programming and math, Llama leans literary, DeepSeek produces religious content, and Qwen generates multiple-choice questions. The study also identifies model-specific degenerate outputs (e.g., repetitive artifacts, URLs) as signals for safety and privacy risks. This contributes to understanding LLM behavior beyond standard benchmarks by examining their learned generative priors.
  10. DeepSeek-V3.1: Hybrid Thinking Model Now Available on Together AIThis post introduces the availability of DeepSeek-V3.1 on Together AI, highlighting its hybrid 'thinking' and 'non-thinking' modes for balancing response speed and reasoning depth. It details the model's technical specifications (671B total parameters, 37B active per token, 128K context), its built-in agent capabilities (code and search), and its performance improvements on benchmarks like MMLU-Redux, GPQA-Diamond, LiveCodeBench, and AIME 2024. The post also emphasizes the production-ready infrastructure, including OpenAI-compatible APIs, fine-tuning availability, batch processing, and dedicated endpoints, all optimized for large mixture-of-experts models.
  11. OpenAI's New Open gpt-oss Models vs o4-mini: A Real-World ComparisonThis post introduces and evaluates OpenAI's new gpt-oss-120B model against o4-mini through five practical tests: terminal snake game development, creative SVG generation, advanced instruction following, mathematical reasoning, and web-enhanced information synthesis. It highlights gpt-oss-120B's superior performance in code generation and instruction following, while o4-mini demonstrated better results in creative SVG generation. The comparison reinforces the advantages of open-source models in terms of customization, cost-efficiency, and deployment flexibility.