Why 80% Reliability Isn't Good Enough — Felipe Blanes, Amazon AGI Lab

This post introduces the concept of the 'benchmark illusion' in AI agent development, where static evaluations fail to capture real-world performance degradation due to unexpected customer usage. It proposes a customer-centric 'eval flywheel' as a solution, comprising four steps: defining success from the customer's perspective, capturing real-world signals (including direct customer feedback), diagnosing gaps in the model, harness, or product, and using these insights to inform development decisions. The talk also highlights the 'trust cliff,' where reliability below a certain threshold (around 92%) erodes customer trust, and advocates for transparency about model limitations over inflated benchmark scores. It shares surprising use cases from Amazon Leo, Hertz, and Sola, and outlines four principles for maintaining fresh and relevant evaluations.