AI debugging at Meta with HawkEye

This post introduces HawkEye, a toolkit for monitoring, observability, and debugging of end-to-end machine learning workflows at Meta. It details workflows for isolating top-line product issues to specific model snapshots, correlating degradation with model snapshots and feature importance, tracing feature issues upstream to data pipelines, diagnosing model snapshot issues by comparing weights and biases, and identifying training data problems. HawkEye aims to significantly reduce debugging time and simplify operational workflows for AI/ML systems.