How Twitter Built a Massive Machine Learning Pipeline Using Kafka
Sep 25, 2020Peilin Yang
Summary
Introduces a custom Kafka Streams join DSL variant for ML logging pipelines, addressing the need for delayed emission of unmatched left-side records and improved handling of feature-label joins. Details the implementation of three state stores (KeySortedStore/WindowStore, TimeSortedStore/KeyValueStore, JoinedIndicatorStore) and the logic of the normal processor and punctuator to achieve efficient, high-throughput joins for millions of events per second.
More from the Streaming Machine Learning trail
- Focal Systems: Boosting Store Performance with an AI Retail Operating System and Real-Time Data
- Detecting the Unexpected: Built-in Real-Time Anomaly Detection in Confluent Cloud for Apache Flink®
- Real-Time Business Analytics Using AI
- Uniting Machine Learning and Data Streaming Technologies - Part 1
- Streaming Machine Learning and Analytics with Confluent and SymetryML