Taming the ML Firehose: Scaling Feature Consistency

This post details the implementation of a feature logging framework to ensure consistency between ML model training and inference data at Uber Eats. It addresses scale challenges including bandwidth, volume, feature name length, and selective logging by implementing a feature allow list, feature name aliasing via enums, and impression filtering with Flink. The Flink job optimization involved profiling, custom state management with RocksDB, and aggressive eviction strategies to handle large state footprints and high throughput.