Falcon Perception

This post introduces Falcon Perception, a 0.6B-parameter early-fusion Transformer for open-vocabulary grounding and segmentation from natural language prompts. It details the architecture, which uses a single backbone processing image patches and text tokens in a shared parameter space from the first layer, with a hybrid attention mask allowing bidirectional attention on image tokens and causal attention on text and task tokens. The "Chain-of-Perception" interface decomposes instance prediction into `<coord>` → `<size>` → `<seg>` steps. Specialized heads use Fourier feature encoding for coordinates/sizes and a dot product with upsampled image features for segmentation masks. It also introduces PBench, a diagnostic benchmark for perception systems. The training recipe involves multi-teacher distillation, large-scale data (54M images, 195M positive expressions, 488M hard negatives), and a three-stage process totaling 700 GT. Results on SA-Co and PBench are presented. Additionally, Falcon OCR, a 0.3B-parameter model, is released, achieving high scores on olmOCR and OmniDocBench with high throughput. Inference optimizations include a paged inference engine and Docker/MLX integration for Falcon-OCR.