Model Evaluation and Benchmarking

7 posts · Feb 2026 – Sep 2026

What this trail covers

The Open TTS Leaderboard provides a scalable and objective evaluation framework for multilingual text-to-speech (TTS) and voice cloning models. It complements human preference scores by using objective metrics like Word Error Rate (WER) for intelligibility, inverse real-time factor (RTFx) and time-to-first-audio (TTFA) for speed, and speaker similarity (SIM) for voice identity preservation. The leaderboard allows for comparison across languages and supports voice cloning evaluation, with a 'Listen' tab for direct comparison of model outputs and a 'Streaming' tab for interactive application performance. Community feedback is encouraged to shape future evaluations.

All Hugging Face trails

Sep 2026MTEB: Massive Text Embedding BenchmarkThis post introduces the Massive Text Embedding Benchmark (MTEB), a comprehensive benchmark for evaluating text embedding models. It details the benchmark's structure, including various tasks (sentence similarity, retrieval, clustering) and datasets, and presents results for several models, including Muennighoff/SGPT-5.8B-weightedmean-msmarco-specb-bitfit. The post emphasizes the importance of standardized evaluation for comparing and selecting effective text embedding models.
Jun 2026Nemotron 3.5 Content Safety: Customizable Multimodal Safety for Global Enterprise AIThis post introduces Nemotron 3.5 Content Safety, a multimodal safety model that can be customized for enterprise AI applications. It supports both image and text content, providing customizable safety controls. The model is available as a 4B parameter model and is updated to include specific safety features for enterprise use cases.
Apr 2026Direct Preference Optimization Beyond ChatbotsThis post introduces Direct Preference Optimization (DPO) as a method for aligning large language models with human preferences, extending its application beyond traditional chatbots to other generative tasks. It details the DPO loss function and its implementation, demonstrating how it can be used to fine-tune models for tasks like summarization or code generation based on preference data. The post also highlights the Dharma-AI/Dharma-OCR-LITE model as an example of a multimodal model that can benefit from such alignment techniques.
Feb 2026Open TTS Leaderboard: Scalable Evaluation for Multilingual Text-to-Speech and Voice CloningIntroduces the Open TTS Leaderboard, a new system for evaluating TTS and voice cloning models using objective metrics: WER/CER for intelligibility (via Qwen3 ASR), RTFx/TTFA for speed (on H200 GPU and CPU), and cosine similarity for speaker similarity (via WavLM embeddings). The leaderboard supports multilingual evaluation and includes a 'Listen' tab for direct comparison of model outputs and a 'Streaming' tab for TTFA analysis. It aims to address the scalability limitations of human preference-based leaderboards and increase the visibility of open-source TTS models.
Advertisement