Universal Image Segmentation

Hugging Face now supports universal image segmentation with Mask2Former and OneFormer, enabling users to segment images into objects, instances, or scenes using a unified approach. This capability leverages advanced transformer architectures to handle diverse segmentation tasks efficiently. The post details the technical implementation of OneFormer, a unified framework for universal image segmentation, highlighting its architecture and performance on various segmentation tasks. This post introduces Falcon Perception, a 0.6B-parameter early-fusion Transformer for open-vocabulary grounding and segmentation from natural language prompts, and Falcon OCR, a 0.3B-parameter model for document understanding. Falcon Perception processes image patches + text in one sequence using a hybrid attention mask, and produces variable numbers of instances with a small, structured token interface and lightweight output heads. On SA-Co, Falcon Perception reaches 68.0 Macro-F1 (vs. 62.3 for SAM 3) with the main remaining gap being presence calibration (MCC 0.64 vs. 0.82). PBench, a diagnostic benchmark, breaks down performance by capability and by dense long-context crowded scenes. Falcon OCR reaches a score of 80.3 and 88.6 on the olmOCR benchmark and OmniDocBench respectively, while having the highest throughput of any open source OCR model. The post also discusses the architecture, training recipe (multi-teacher distillation, large-scale data), and inference optimizations (paged inference engine, Docker and MLX integration for Falcon-OCR).