TechPrufer
Get started
Confluent
Confluent logoConfluent · Feature trail

AI Developer Tools

9 posts · May 2023 – Sep 2026

What this trail covers

Confluent Cloud enhances AI-ready streaming capabilities across data, pipelines, and operations. This includes a dbt adapter for Apache Flink and Materialized Tables, a managed Model Context Protocol (MCP) server and Confluent Agent Skills for AI coding agents, and the General Availability (GA) of Streaming Agents and Real-time Context Engine for event-driven AI. The VS Code extension now integrates with GitHub Copilot chat participant APIs to provide engineers with safe, fast access to data str

All Confluent trails
  1. Building an Agentic SOC on a StreamThis post details the architecture and implementation of an "Agentic SOC" that leverages AI agents to autonomously investigate security alerts. It describes a system comprising a central triage coordinator, specialized evidence agents, an adversarial evaluator, and a self-learning knowledge base. Key technical contributions include treating source-specific quirks as data rather than code for agent specialization, implementing an adversarial review process with a secondary agent to find flaws in primary agent conclusions, and treating alerts as a continuous stream with priority lanes for critical events. The system streams agent telemetry back into Confluent Cloud topics for real-time context and monitoring.
  2. New in Confluent Intelligence and AI Tools: Making Agents Native to the Stream, Expanded Model Support, New Agent Skills, and CopilotThis post announces several updates to Confluent's AI and developer tools. Key contributions include: 1. Expanded model support in Flink SQL functions (`AI_FORECAST`, `AI_DETECT_ANOMALIES`) for IBM Granite time-series models (TTM, FlowState, PatchTST-FM) and Google TimesFM, enabling direct integration of these models for anomaly detection and forecasting on streaming data. 2. Updates to the fully managed MCP server, adding write tools for restarting and updating connector configurations, and upcoming Flink tools for managing Flink SQL. 3. New open-source Agent Skills for AI coding assistants, including a migration skill for assessing MSK environments and planning migrations to Confluent Cloud, and skills for Java Kafka producer/consumer and Flink UDFs. 4. Confluent Copilot for Confluent Cloud (EA) provides a hosted, natural-language interface for building and operating Confluent Cloud. 5. Real-Time Context Engine (GA) now includes enhanced metrics for usage, performance, health, and cost visibility (table size, scan rate, throughput, query rate). 6. Lightning queries (EA) offer a REST API for low-latency queries against live state. These updates focus on making AI agents more context-aware by leveraging real-time streaming data and simplifying the development and operationalization of AI/ML workloads on Confluent Cloud.
  3. Agentic Event-Driven Systems ArchitectureThis post introduces the architectural patterns for agentic event-driven systems, which combine event streaming, stateful processing, and agentic decision layers to form closed-loop AI systems capable of operating independently at scale. It details an eight-layer architecture: Event Producers, Streaming Backbone (using platforms like Kafka), Stateful Stream Processing (using engines like Flink), Shared State & Context Layer, Agent Execution Layer (LLM-based agents, ML models, rules engines), Orchestration & Policy Engine, Command & Event Emission, and Observability & Governance. The core concept is the closed-loop control pattern where actions generate outcome events that feed back into the decision process, enabling continuous adaptation and autonomy.
  4. Confluent AI Developer Tools Now GA: MCP Servers & Agent SkillsThis post announces the General Availability (GA) of Confluent AI Developer Tools, including an open-source local Model Context Protocol (MCP) server, a managed MCP server hosted by Confluent Cloud, and Agent Skills. The local MCP server provides read/write access to Confluent environments (Cloud and local Kafka) for AI assistants to discover topics, schemas, connectors, Flink pools, and perform actions like creating topics, deploying Flink SQL, and configuring connectors. The managed MCP server offers read-only access to Confluent Cloud for exploration and diagnostics, with a connector troubleshooting suite. Agent Skills package Confluent domain expertise (Schema Registry, Kafka Streams, Python Kafka Client, CDC to Tableflow) for AI coding tools, enabling platform-aware assistance without direct MCP connection, though they can leverage it. The post details a demo of using Agent Skills and MCP together to build an end-to-end CDC pipeline from MySQL to Iceberg using natural language prompts.
  5. Confluent Cloud Q2 2026: dbt adapter, Confluent Intelligence updatesThis post introduces a managed Model Context Protocol (MCP) server for Confluent Cloud, making Confluent resources discoverable by MCP clients for natural language interaction. It also launches Confluent Agent Skills, open-source skills encoding Confluent best practices for AI coding agents, with a three-tier governance model. The open-source local MCP server for Apache Kafka and Confluent Platform is expanded for hybrid development. Additionally, it announces the GA of Streaming Agents and Real-time Context Engine, and the dbt adapter for Confluent Cloud for Apache Flink® and Materialized Tables, enabling SQL-based stream processing workflows and simplifying Flink pipeline evolution with persistent, database-like objects. Process Table Functions (PTFs) are in Early Access, allowing custom stateful stream processing logic in Java. External connectivity for UDFs is also in EA, enabling Flink applications to interact with external REST APIs, databases, or AI services.
  7. New in Confluent Intelligence: Real-Time Context Engine Upgrade, New Model Support, ML Functions, and MoreThis post announces the General Availability (GA) of the Real-Time Context Engine with enhanced query capabilities (filters, ranges, compound queries, projections, ordering) directly on live tables, reducing the need for external databases. It also marks the GA of Streaming Agents with enterprise-grade reliability and an Agent Management Console for UI-driven agent creation and operation. New ML functions for PII detection and sentiment analysis are introduced in Early Access (EA), and multivariate anomaly detection moves to Open Preview (OP). Support for TimesFM (EA), Anthropic (GA), and Fireworks AI (GA) models is expanded. The post details the technical advancements in these areas, focusing on enabling real-time, context-rich AI systems on Confluent's streaming platform.
  8. New in Confluent Intelligence: A2A, Multivariate Anomaly Detection, and MoreIntroduces Agent2Agent (A2A) Integration for Streaming Agents, enabling collaboration with external agents over Kafka. Adds Multivariate Anomaly Detection for built-in ML functions in Flink SQL to detect anomalies across multiple correlated metrics. Expands vector search capabilities to include Azure Cosmos DB and Amazon S3 Vectors, integrating them into Flink's external tables and search fabric. Enables AWS and Azure Private Link for secure, VPC-to-VPC connectivity from Flink to external models, databases, and vector stores, keeping AI traffic off the public internet. Provides vendor-backed support for the open-source MCP server.
  9. Thunai Automates Customer Support with AI Agents and Data StreamingThunai leverages Confluent's data streaming platform to power its agentic AI platform for customer support automation. The post details how real-time data ingestion from various customer interaction channels (voice, chat, email) into Kafka provides fresh, reliable context for AI agents. This enables features like L1 task deflection, real-time agent assistance, and operational monitoring. The architecture relies on streaming data to continuously update Thunai Brain, their knowledge base, and supports safe agentic actions through robust integrations. Confluent's Kafka and Flink services are highlighted as crucial for achieving low latency, scalability, reliability, and governance in their AI-driven customer support solution.
  10. Real-Time Context for Snowflake Intelligence with ConfluentThis post details the integration of Confluent Intelligence with Snowflake Cortex agents. It introduces the Real-Time Context Engine as a managed service for serving enriched streaming data to AI systems via the Model Context Protocol (MCP). It also highlights Streaming Agents, built on Flink, for creating event-driven agents that can perform LLM reasoning, RAG, and generate real-time embeddings. The post also mentions Tableflow for converting Kafka topics into Apache Iceberg tables for Snowflake.
  11. Faster, Smarter, More Context-AwareStreaming AgentsThis post introduces new capabilities for Confluent's Streaming Agents, focusing on simplified agent definition with iterative tool calling, enhanced observability and debugging through structured immutable logs and Flink-powered recovery, and the Real-Time Context Engine for serving fresh context via MCP with built-in authentication, RBAC, and audit logging. It also highlights integrations with cloud providers (AWS, Azure, GCP), vector databases, and frameworks (Anthropic, LlamaIndex), and mentions system integrators. Use cases like real-time fraud prevention, dynamic customer support, and intelligent supply chain optimization are detailed.
  12. Confluent + Google Cloud: Real-Time Backbone for Agentic AIThis post details the integration of Confluent's data streaming platform with Google Cloud to power real-time backbone for agentic AI systems. It explains how Confluent facilitates real-time data flow for Generative AI, enabling LLMs and other AI services with up-to-date information. The post elaborates on the Model Context Protocol (MCP) for agent-to-resource interaction and Agent-to-Agent (A2A) protocol for inter-agent communication, highlighting how Confluent acts as a proxy to standardize and manage these interactions. Key technical aspects include using Kafka topics for decoupled, event-driven communication, ensuring message durability and asynchronous processing, and exposing existing microservices and Flink jobs as MCP tools or A2A agents via a unified registry and protocol transformation. The post also emphasizes the role of Confluent in providing real-time context and stability for complex agent systems.
  13. Introducing Streaming Agents on Confluent CloudIntroduces Streaming Agents on Confluent Cloud, enabling event-driven agents to be built, deployed, and orchestrated natively on Apache Flink. These agents can monitor and act on real-time business events, unifying stream processing and AI workflows. Key features include seamless integration with models and tools, access to real-time context for dynamic decision-making, and built-in security and governance. Under the hood, it leverages Flink jobs for event-driven replayability, observability, and governance. Specific capabilities highlighted are model inference via remote endpoints, real-time embeddings for RAG, built-in ML functions (forecast, anomaly detection), tool calling with MCP, secure connections for external systems, and external tables/vector search for data enrichment.
  14. AI & Developer Experience: Building Smarter with CopilotThis post details the integration of Confluent's VS Code extension with GitHub Copilot's chat participant APIs, enabling data streaming engineers to leverage AI for project scaffolding and generation. It outlines the architecture of the LanguageModelTool API integration, the message types involved (User, Assistant), and the file structure for the participant's tools and summarizers. The focus is on providing a safe and efficient developer experience by allowing AI to call tools, such as generating projects with pre-filled configurations.
  15. How to build a multi-agent orchestrator using Flink and KafkaThis post details the implementation of an event-driven orchestrator for multi-agent systems using Apache Flink and Apache Kafka. It describes how Flink can interpret messages, leverage LLMs for agent selection via external model inference, and route tasks to specific agents. The architecture uses Kafka as a shared memory backbone and Flink for real-time processing and routing. A practical example of an AI-driven sales development representative (SDR) system is provided, showcasing how agents interact dynamically without hard-coded dependencies.
  16. Why Google’s Agent2Agent Protocol Needs Apache KafkaThis post explores how Apache Kafka can serve as a scalable, event-driven communication layer for Google's Agent2Agent (A2A) protocol, extending its point-to-point HTTP/JSON-RPC/SSE model. It argues that Kafka's capabilities for loose coupling, multiple consumers, durable storage, and real-time event flow are crucial for building robust enterprise AI agent ecosystems, similar to how event-driven architectures transformed microservices. The post outlines three potential integration patterns: Kafka as a transport layer, Kafka for task routing and fan-out, and a hybrid orchestration pattern using Kafka to drive multi-agent workflows.
  17. Powering AI Agents with Real-Time Data Using Anthropic’s MCP and ConfluentThis post introduces a Confluent-built Model Context Protocol (MCP) server that integrates with Anthropic's MCP standard. This server allows AI agents to access real-time data from Confluent Cloud, manage Kafka topics, execute Flink SQL queries, and configure connectors using natural language. The implementation includes 20 built-in tools and a clear process for adding new tools, demonstrating interactions with Goose AI framework for tasks like listing topics, saving conversations, sampling data, and managing retention policies.
  18. Getting Started with Confluent for VS CodeThis post announces the General Availability of Confluent for VS Code, an open-source extension that streamlines development for streaming data engineers. Key features include project setup templates, direct connection to various Kafka deployments (Confluent Cloud, Platform, WarpStream, self-managed Kafka), real-time topic visibility and debugging, local Kafka/Schema Registry cluster execution via Docker, expanded language support (Python, Go, .NET) for project templates, manual message production, shareable topic links, and schema evolution support. The extension aims to accelerate local development and improve the overall developer experience for Kafka users.
  19. Building AI Agents and Copilots with Confluent, Airy, and Apache FlinkThis post details how Airy built AI agents and copilots by integrating Confluent, Apache Flink, and AI Model Inference. It addresses challenges in data governance, security, accuracy of GenAI outputs, and cost-effective infrastructure. The solution leverages Confluent connectors for data ingestion, Flink for stream processing and embedding generation, and Stream Governance for data contracts and schema evolution. It highlights the use of Flink AI Model Inference to simplify the stack and mentions Tableflow for Iceberg integration. The benefits include low-latency streaming, governed data streams, accurate AI decisions, and cost-effective scaling.
  20. Automating Podcast Promotion with AI and Event-Driven DesignThis post details the architecture and implementation of an AI-powered LinkedIn post generator for podcasts. It showcases an event-driven architecture using Confluent Cloud (Kafka and Flink) to decouple a Next.js frontend from an AI workflow involving OpenAI's Whisper and GPT models. Key technical contributions include: defining a Flink SQL CREATE MODEL statement for OpenAI's GPT-4, using Flink SQL views to construct prompts by joining Kafka topics (podcast transcript and request details), and employing HTTP Sink Connectors to move data between Kafka and external APIs for audio processing and cache writing. The post emphasizes the benefits of EDA for scalability, modularity, and future-proofing AI applications.
  21. Reworkd Scales Real-Time Web Scraping With Data Streaming and GenAIThis post details how Reworkd uses agentic AI and GenAI to automate and scale real-time web scraping and processing. It describes their architecture which leverages the Confluent Data Streaming Platform for streaming web data into Kafka topics, using Schema Registry for data governance, and performing shift-left stream processing for validation and transformation. The agentic AI workflow automates code generation and testing for web scraping, utilizing RAG to create embeddings and understand webpage context for efficient scraping code. Data is processed in real-time on Confluent for deduplication and transformation, enabling customers to integrate it into their workflows via a REST API.
  22. Getting Started With Confluent for VS CodeThis post announces the Early Access Program for Confluent for VS Code, an extension that integrates Confluent's streaming capabilities directly into the VS Code IDE. It aims to streamline development workflows for Kafka and Flink developers by providing features such as project templates, Kafka topic management (listing, creating, editing, deleting, browsing messages), integration with Schema Registry, and live debugging of data streams alongside code. The extension is open-source and available on the VS Code Marketplace.
  23. How Developers Can Use Generative AI to Improve Data QualityThis post details how generative AI can be leveraged to improve data quality by automating the creation of metadata (schemas, descriptions, provenance) which is essential for trustworthy AI applications. It explains the process of using generative AI models with examples to induce schemas and generate metadata, emphasizing the need for human review to correct AI errors and ensure accuracy. The post advocates for a data streaming platform as an optimal pattern for applying metadata at the time of data production and creating reusable data products. It also touches upon the broader concept of AI governance and its dependence on robust data governance.
  24. Automatic Detect PII in Real Time with Machine LearningThis post details the development and deployment of a PII Detection stream processing application, PII UDFs/UDTs for ksqlDB, and a PII SMT for Kafka Connect. These components leverage machine learning and NLP to identify and redact Personally Identifiable Information (PII) from data streams in real-time, enhancing data governance and security for SIEM/SOAR solutions. It also introduces the concept of entity-level control for unstructured data and provides configuration examples for custom entity detection.