Flink AI Model Inference

2/13/2025

This post details the implementation of remote AI model inference within Apache Flink on Confluent Cloud. It outlines the architecture where Flink applications make API calls to external model servers for real-time predictions. Key technical aspects covered include preprocessing data in Flink, registering models using SQL DDL statements (e.g., CREATE MODEL), invoking remote endpoints, and utilizing the ML_PREDICT function. It also discusses optimizing network efficiency through batching, implementing robust error handling, using efficient data encoding (Protocol Buffers, Avro), monitoring model drift, and optimizing cloud resources for independent scaling of model servers and Flink applications.

12/10/2024

Confluent Platform 7.8 introduces Confluent Platform for Apache Flink (GA), providing enterprise-grade Flink distribution and control plane for on-prem workloads. This includes Confluent Manager for Apache Flink (CMF) for Kubernetes-native job lifecycle management, resource handling, and automation. The release also enhances platform security with RBAC authorization using mTLS identity, allowing granular access control and service-to-service authentication. Confluent Control Center (C3) receives updates for browsing and searching messages across partitions and producing/consuming schematized messages with validation. Additional features include platform support for Ubuntu 22.04 and Alma Linux 8/9, JBOD support in KRaft mode, and updates to Confluent for Kubernetes (CFK) and CP Ansible Playbooks for enhanced deployment and management capabilities.

10/1/2024

This post introduces and details the Flink Table API, a unified relational API for stream and batch processing, now supported in Confluent Cloud for Apache Flink® and Confluent Platform for Apache Flink®. It explains the Table API's programmatic interface in Java and Python, its relationship and interoperability with other Flink APIs (DataStream, SQL), and its benefits for serverless deployments on Confluent Cloud. Practical examples and unit testing strategies are provided.

9/16/2024

This post details a practical application of Confluent's data streaming platform, specifically Apache Flink on Confluent Cloud, for accelerating insurance claims processing. It outlines a 5-step architecture for real-time claims, emphasizing the use of Kafka topics for event ingestion (FNOL), data enrichment via temporal joins with customer and policy data streamed from Oracle CDC, and Flink for data filtering and routing. The post highlights the benefits of stream processing over batch jobs for improved customer experience, reduced risk of inaccurate assessments, and enabling parallel processes like fraud detection and ML model training. It also showcases Confluent's managed connectors and Flink SQL capabilities.

8/2/2024

This post details the technical advancements in Apache Flink 1.20, including the introduction of the `DISTRIBUTED BY` clause for unified bucketing strategies in Flink SQL (FLIP-376), the `MATERIALIZED` table concept for managing data freshness and automating refresh jobs (FLIP-435), and parallel aggregations on non-keyed streams for the DataStream API (FLIP-380). It also covers operational improvements like unified file merging for checkpoints (FLIP-306), RocksDB SST file compaction (FLINK-26050), and batch job recovery from JobMaster failures (FLIP-383). The post highlights API deprecations in preparation for Flink 2.0.

7/29/2024

This post details the integration of Flink AI Model Inference within Confluent Cloud for Apache Flink. It explains how Flink SQL can now invoke machine learning models, including generative AI large language models (LLMs) and custom ML/deep learning models, directly from SQL statements. The post outlines the process of creating Flink models as resources, invoking them using the `ML_PREDICT` function, and integrating with various cloud AI services (OpenAI, Azure OpenAI, Amazon Bedrock, Google Vertex AI). A practical example demonstrates using Flink AI for sentiment analysis of product reviews, covering Kafka topic creation, data ingestion, Flink SQL model definition, invocation, and storing results in a downstream table. The feature is currently in Early Access (EA).

3/19/2024

This post announces the general availability of Confluent Cloud for Apache Flink, detailing its serverless architecture, elastic autoscaling, usage-based billing, and 99.99% uptime SLA. It highlights features like Flink SQL, SQL Workspaces, Data Portal integration, Flink Actions, and early access to Java UDFs. The post also emphasizes reliability through exactly-once processing, distributed snapshotting, and integration with Kafka's two-phase commit protocol. Production operations are supported via REST API and Terraform, with enhanced monitoring through the Confluent Cloud Metrics API. The integration with Kafka and Schema Registry is also detailed, including Stream Lineage for data traceability.

2/27/2024

Introduces 'Topic Actions', a new feature for Confluent Cloud's serverless Flink service, providing turn-key stream processing workloads for common use cases like message deduplication and field masking. These Actions are implemented as templated Flink SQL statements that can be configured via a user-friendly interface or exposed via an 'Open SQL editor' for customization. The post also details the addition of Terraform support for deploying Flink statements programmatically and the expansion of Flink service availability to GCP, in addition to existing AWS and Azure support.

12/12/2023

This post details the serverless architecture of Confluent Cloud for Apache Flink, focusing on fine-grained scaling and fast rescaling. It explains the use of compute pools for elastic resource allocation, the DS2 algorithm for determining optimal parallelism and task sizes, and Flink's Adaptive Scheduler and Declarative Resource Management for non-disruptive rescaling operations. It also highlights specific Apache Flink JIRA issues (FLINK-32326, FLINK-32345, FLINK-33341, FLINK-31238) that contribute to faster rescaling, particularly for jobs with small state. The post also elaborates on the cost-effectiveness of the usage-based billing model and its benefits for variable workloads.

11/29/2023

This post elaborates on the synergy between data streaming and AI/ML, highlighting how Confluent Cloud, particularly with its managed Apache Flink service, streamlines the integration of streaming data into AI/ML models. It details how Confluent Cloud enables embedding ML models in event streams for reduced latency, performing simulations and analysis with real-time data, and supporting asynchronous communication for AI/ML applications. It also mentions the deployment of AI/ML models at the edge using librdkafka and on mobile applications. The post emphasizes Confluent Cloud's capabilities in autoscaling, observability, and automatic model updates, and how Flink optimizes resource consumption for low latency and fault tolerance, facilitating real-time AI/ML pipelines.

11/8/2023

This post introduces SQL Workspaces, a new browser-based interface for Confluent Cloud's serverless Apache Flink service. It details how Flink SQL is integrated with Confluent Cloud environments, clusters, and topics, mapping them to Flink's catalog, database, and table abstractions. The post highlights features like auto-saving workspaces, multi-editor support for concurrent development and debugging of streaming SQL statements, and real-time runtime metrics. It also describes contextual entry points within the Confluent Cloud UI (e.g., from Kafka topic views and Data Portal) that allow users to directly launch SQL Workspaces and query data.

10/12/2023

This post details the evaluation of stream processing engines, specifically Apache Flink and Kafka Streams, discussing their strengths, weaknesses, and use cases. It emphasizes the importance of choosing the right data streaming platform and stream processing engine based on business needs, latency, data volume, processing requirements, and SLAs. The post also highlights the challenges of building robust streaming frameworks and the benefits of managed services in simplifying stream processing operations.

9/26/2023

This post introduces the open preview of Confluent Cloud for Apache Flink, emphasizing its serverless, cloud-native nature and seamless integration with Kafka. It highlights the benefits of using Flink for stream processing without the operational burden of self-management. The post also details the introduction of Enterprise clusters, which offer private networking via AWS PrivateLink for secure, isolated connections. Additionally, it announces cost savings on data storage, updates to the Confluent Terraform provider including HashiCorp Sentinel integration and a Resource Importer, and previews the upcoming Data Portal for Stream Governance.

9/26/2023

This post announces the general availability of Confluent Cloud for Apache Flink, re-architecting Flink as a cloud-native, serverless service. Key technical contributions include: serverless Flink with elastic autoscaling (scale-to-zero) and usage-based billing; Flink compute pools for managing elastic compute resources, allowing for budgetary caps and workload segmentation; a unified Kafka connector for Flink SQL simplifying operations across Kafka clusters; and an integrated SQL editor and workspace for enhanced developer experience. The service aims to provide simplicity, security, and scalability comparable to Confluent's Kafka offering, with a focus on decoupling compute from storage for independent scaling.

9/14/2023

This post details building a Java pipeline with Flink and Kafka for clickstream data analysis. It covers setting up a Maven project with Flink dependencies (flink-streaming-java, flink-json, flink-connector-kafka), building an uber JAR using the maven-shade-plugin, defining POJO data structures for input (ClickStreamRecord) and output (ClickStreamAnalytics), creating a Flink entry point (ClickStreamJob), and initializing the StreamExecutionEnvironment. It also demonstrates loading Kafka configuration properties from a file and outlines the process of consuming from Kafka, performing keyBy and windowing operations, and producing results.

9/12/2023

This post introduces Flink SQL as a declarative way to process and analyze data streams, covering its relation to other Flink APIs, dynamic tables, continuous queries, and core features like joins, aggregations, windows, and pattern recognition. It provides syntax examples for creating Kafka tables and performing SQL operations, highlighting its ANSI SQL compliance and optimization techniques.

8/29/2023

This post details how Apache Flink supports three key categories of stream processing use cases: event-driven applications, real-time analytics, and streaming data pipelines. For event-driven applications, it highlights Flink's capabilities in fraud detection, anomaly detection, alerting, real-time routing, business process monitoring, and IoT applications, emphasizing its advanced windowing, per-event stateful processing, and Complex Event Processing (CEP) library with MATCH_RECOGNIZE. For real-time analytics, it discusses use cases like ad performance tracking, content performance monitoring, Telco network quality monitoring, and pattern analysis, underscoring Flink's low-latency processing, Flink ML library, time-series analysis support, and dynamic tables for materialized views. The post also touches upon streaming data pipelines as a foundational category.

8/15/2023

This post introduces the core concepts of Apache Flink, its powerful runtime, flexible APIs (DataStream, Table API, Flink SQL, Stateful Functions), and its unified approach to stream and batch processing, highlighting its production-readiness with features like a flexible metrics system and extensive observability. It also details the benefits of using Flink for Kafka use cases, focusing on reduced latency, increased innovation and reusability, resource efficiency, and expressiveness.

4/4/2023

This post details the technical challenges and solutions for integrating Python's ML ecosystem with JVM-based data streaming platforms like Kafka and Flink. It explores the use of SQL as an intermediary, the mechanisms for inter-process communication (Py4J, gRPC, PemJa), and the role of Apache Arrow. It also discusses the current state of ML experiment management platform integrations with Kafka, noting the lack of direct managed integrations and the reliance on custom connectors or SDKs. Code examples are provided for building streaming ML solutions using the Confluent Python Client.

1/6/2023

This post announces Confluent's acquisition of Immerok and the strategic decision to integrate a fully managed Apache Flink offering into Confluent Cloud. It outlines the rationale for choosing Flink, highlighting its strengths in multi-language support, principled processing model, state management, and fault tolerance. The post emphasizes the goal of making stream processing as easy as batch processing and positions Flink as the de facto standard for building applications that process and react to data streams, working seamlessly with Kafka.

9/2/2016

This post provides a detailed comparison between Apache Flink and Kafka Streams, focusing on their architectural differences and ideal use cases. It elaborates on Flink's cluster-based deployment, job lifecycle management, and its capabilities for high throughput, low latency, and exactly-once semantics. It contrasts this with Kafka Streams, which is presented as an embeddable library that leverages Kafka for coordination, load balancing, and fault tolerance, simplifying stream processing for application developers. Key differences in deployment, lifecycle, coordination, data sources/sinks, and semantic guarantees are outlined.