Uniting Machine Learning and Data Streaming Technologies - Part 1

What this post added

This post, Part 1 of a series, examines the current state of the machine learning and data streaming markets, identifying opportunities at their intersection and socio-technical barriers to unification. It argues that operationalizing ML requires operationalizing data, emphasizing the need for a powerful data platform and a cultural shift. The post highlights the challenges in ML adoption, such as the gap between limited production and widespread deployments, the struggle to become data-driven, and cultural impediments. It draws parallels between the ML market transformation and shifts in traditional tech (data science mimicking computer science, data engineering mimicking software engineering, MLOps mimicking DevOps, data mesh to data warehouses as microservices to monoliths). The post positions Apache Kafka as the central nervous system for ML-powered applications, detailing its role in data gathering, context provision, sensing, inference, and actuation. It emphasizes Kafka's suitability for real-time analytics, contrasting the Kafka protocol's efficiency with HTTP for data transfer and discussing the speed trade-off in stream processing. The post also explains Kafka's utility for historical data through event sourcing and infinite retention, supported by tiered storage. Finally, it introduces Kafka Connect as a key integration tool for ML workflows, listing several relevant connectors.