Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

This talk advocates for using coding agents with pre-trained, Apache 2.0 licensed models instead of directly deploying large Vision Language Models (VLMs) for real-time computer vision tasks. It introduces a toolkit that leverages a coding agent to act as a 'clueless computer vision engineer.' This agent uses a large VLM for initial labeling (vibe training) on unlabeled datasets, then employs two smaller VLM judges to refine bounding boxes based on minimum agreement, and finally trains a more efficient RF-DETR model. The process is demonstrated to be cost-effective (a few dollars) and achieves strong results on road sign detection and document parsing, even identifying signatures missed by the initial labeling VLM. Key findings highlight judge imbalance, the need for human approval of judge prompts, and agent blunders like flipping traffic signs. The toolkit also includes smaller models for segmentation, pose, depth, and OCR, with future plans for image-guided detection.