This release details how the Kafka ecosystem can be leveraged to build scalable machine learning infrastructure, addressing the impedance mismatch between data scientists and production engineers. It highlights the use of Kafka for real-time data ingestion, processing, model deployment, and monitoring. The post explores using Jupyter Notebooks with KSQL and TensorFlow/Keras for model training, and discusses how the same Kafka pipelines can be reused for model inference. It emphasizes Kafka's role in enabling two primary approaches for model deployment: model servers with RPC communication and embedding models directly into Kafka client applications. The post also discusses considerations for embedding models, such as performance, size, and the need for model server features, and evaluates the tradeoffs between the two approaches. Finally, it touches upon cloud-native deployment strategies using Kubernetes to combine the benefits of both approaches.
2023
Real-Time Business Analytics Using AI
10/31/2023
This post details the challenges and solutions for real-time business analytics using AI/ML. It outlines data quality issues (completeness, context, consistency) and scalability challenges (volume, velocity, infrastructure, continuous training). It then introduces Confluent Cloud as a solution, highlighting its managed Kafka Connectors for data completeness, ksqlDB and Flink for data context and transformation (including OpenAI API integration), and Schema Registry/Stream Governance for data consistency. The post also touches upon the importance of a robust data infrastructure for performance and scalability.
Uniting Machine Learning and Data Streaming Technologies - Part 1
3/28/2023
This post, Part 1 of a series, examines the current state of the machine learning and data streaming markets, identifying opportunities at their intersection and socio-technical barriers to unification. It argues that operationalizing ML requires operationalizing data, emphasizing the need for a powerful data platform and a cultural shift. The post highlights the challenges in ML adoption, such as the gap between limited production and widespread deployments, the struggle to become data-driven, and cultural impediments. It draws parallels between the ML market transformation and shifts in traditional tech (data science mimicking computer science, data engineering mimicking software engineering, MLOps mimicking DevOps, data mesh to data warehouses as microservices to monoliths). The post positions Apache Kafka as the central nervous system for ML-powered applications, detailing its role in data gathering, context provision, sensing, inference, and actuation. It emphasizes Kafka's suitability for real-time analytics, contrasting the Kafka protocol's efficiency with HTTP for data transfer and discussing the speed trade-off in stream processing. The post also explains Kafka's utility for historical data through event sourcing and infinite retention, supported by tiered storage. Finally, it introduces Kafka Connect as a key integration tool for ML workflows, listing several relevant connectors.
2021
Streaming Machine Learning and Analytics with Confluent and SymetryML
9/8/2021
This post introduces SymetryML's streaming machine learning capabilities and its integration with Confluent Cloud. It details the use of a SymetryML Confluent Connector and Kafka Streams for real-time anomaly detection, including a step-by-step guide and configuration examples. The post also discusses the architecture and benefits of streaming ML over traditional batch ML.
2020
Machine Learning and AI at Scale with MAADS-VIPER and Apache Kafka
12/11/2020
Introduces Transactional Machine Learning (TML) as a paradigm for integrating data streams with AutoML, using Apache Kafka as the backbone. Details the MAADS-VIPER connector for creating transactional training datasets from Kafka streams, MAADS-HPDE for AutoML, and the MAADS Python library for low-code development. Contrasts TML with conventional ML, emphasizing Kafka's role in data fluidity and stream joining. Mentions Confluent Tiered Storage for data rollback and the architectural benefits of microservices for TML solutions.
How Twitter Built a Massive Machine Learning Pipeline Using Kafka
9/25/2020
This post details the development of a custom Kafka Streams join DSL variant to address specific requirements of Twitter's machine learning logging pipeline. It describes the custom state store implementations (KeySortedStore, TimeSortedStore, JoinedIndicatorStore) and their data models, as well as the procedures for normal processing and the punctuator for emitting matched records. The focus is on handling feature-label joins with delayed label arrival and maintaining serving order, achieving high throughput for millions of events per second.
2019
Machine Learning and Real-Time Analytics in Apache Kafka Applications
10/31/2019
This post details two approaches for deploying machine learning models in Kafka applications: using model servers with RPC and embedding models directly into Kafka client applications. It provides code examples for both scenarios using TensorFlow and Kafka Streams, and discusses the trade-offs between the two methods. It also covers considerations for embedding models, such as performance and size, and briefly touches on cloud-native deployment with Kubernetes.
Bust the Burglars – Machine Learning with TensorFlow and Apache Kafka | Confluent
7/16/2019
This post details building a burglar alarm system using network-connected cameras, Apache Kafka, Kafka Streams, and TensorFlow. It covers image acquisition and preprocessing (grayscale conversion, resizing) using a custom Kafka Connect converter, building a TensorFlow model for burglar detection via transfer learning and TensorFlow for Java, and sending alerts using a Telegram sink connector. It also discusses integrating with smart locks via a REST source connector for filtering alerts.
Machine Learning with Python, Jupyter, KSQL and TensorFlow | Confluent
2/6/2019
This post focuses on the practical application of the Kafka ecosystem for building machine learning infrastructure, specifically addressing the impedance mismatch between data scientists and production engineers. It details how Kafka, Kafka Connect, and Kafka Streams/KSQL can be used for data ingestion, preprocessing, model deployment, and monitoring. The post also explores a specific workflow involving Jupyter Notebooks, KSQL, and TensorFlow/Keras for model training and inference, demonstrating how Kafka pipelines can be reused for both stages. It positions Kafka as a complementary technology that enhances existing ML solutions by providing a scalable, reliable, and real-time data backbone.
2018
Using Apache Kafka to Drive Cutting-Edge Machine Learning | Confluent
11/19/2018
This post expands on the use of Kafka for machine learning by detailing specific implementation patterns and architectural considerations. It elaborates on embedded model deployment within Kafka Streams applications and via RPC to dedicated model servers, discussing their respective tradeoffs. It also details hybrid cloud architectures for training and inference, and the use of KSQL for preprocessing and embedding ML models via User-Defined Functions (UDFs), including an example of a deep learning UDF for anomaly detection.
2017
Predicting Flight Arrivals with the Apache Kafka Streams API
10/18/2017
This post demonstrates how to use the Kafka Streams API, specifically KTables and GlobalKTables, to perform real-time machine learning predictions and online model retraining. It details the architecture for consuming flight data, joining it with a GlobalKTable holding serialized machine learning models (OnlineLogisticRegression), and using these to predict flight arrival status. It also explains how to update these models in real-time by publishing new serialized models to a Kafka topic that feeds the GlobalKTable. The post highlights the benefits of this approach for continuous model improvement without offline batch training.
Build and Deploy Scalable Machine Learning in Production with Kafka
9/29/2017
This post details how to build, operate, and monitor analytic models in a scalable, mission-critical way by leveraging Apache Kafka as a central, scalable, mission-critical nervous system and Apache Kafka's Streams API to build intelligent streaming applications. It outlines the machine learning development lifecycle (Build, Validate, Operate, Monitor, Continuous Loop) and presents a reference architecture using Kafka as an intermediary for feature data collection, model building, and prediction serving. It also provides a specific example architecture separating model training (using Hadoop and H2O.ai) from model inference (using Kafka Streams), and briefly mentions online model training as an alternative.
