.NET Kafka Client and TPL Integration

5 posts · Nov 2016 – Dec 2022

What this trail covers

This feature thread tracks the evolution of building streaming applications using Apache Kafka and .NET clients, focusing on leveraging the Task Parallel Library (TPL) for in-process parallelization to maximize throughput. It covers implementing producer and consumer pipelines with Dataflow blocks, managing buffering and parallelism, and ensuring data order guarantees. The goal is to enable .NET developers to build robust and efficient event streaming applications by abstracting low-level concurrency.

All Confluent trails

Dec 2022Build a Streaming Application Using Apache Kafka and .NETThis post details building a streaming application using Apache Kafka, the .NET producer and consumer clients, and the Task Parallel Library (TPL) Dataflow Library. It explains how to use `BufferBlock` and `TransformBlock` to create a pipeline for consuming, processing (including simulated CPU-intensive tasks), and producing data. The post highlights how TPL's Dataflow Library handles work partitioning, thread scheduling, and maintains data order even with increased parallelism, enabling higher throughput for CPU/IO-intensive tasks.
Oct 2019Get Started with Rust and Apache KafkaThis post introduces building streaming applications using Apache Kafka and the .NET client library, leveraging the Task Parallel Library (TPL) for in-process parallelization. It details the use of TPL Dataflow blocks (BufferBlock and TransformBlock) to create a pipeline for consuming, processing (including simulated CPU-intensive tasks), and producing data, emphasizing throughput optimization and maintaining data order. This post also introduces using Rust with Apache Kafka, exploring the `kafka` and `rdkafka` crates for building Kafka producers and consumers. It details the challenges of integrating Avro serialization with Rust clients and demonstrates how to use the `schema_registry_converter` crate for this purpose. Performance tests are included to compare Rust with Java applications using Kafka.
Jun 2019Designing the .NET API for Apache Kafka | ConfluentThis post announces the release of version 1.0 of Confluent's .NET client for Apache Kafka. It details significant API changes, including the introduction of strongly typed configuration classes (ProducerConfig, ConsumerConfig, AdminClientConfig), the builder pattern for client creation (addressing event handler registration and SerDes), and a new general-purpose Message class. The API changes aim for idiomatic .NET design, improved error reporting, and better integration with Confluent Schema Registry, while maintaining performance.
Jun 2017Introduction to Apache Kafka® for Python Programmers | ConfluentThis post introduces the Confluent Python client (`confluent-kafka-python`) which leverages the high-performance C client `librdkafka`. It provides a guide on installing the client, starting a local Kafka cluster using Confluent Platform, producing messages with callbacks and flush/poll methods, and consuming messages with consumer groups and offset management. It also mentions advanced features like partition assignment hooks and Avro serialization with Schema Registry.
Nov 2016Finding a Good Apache Kafka Client | ConfluentThis post details the development of .NET and Go Kafka clients based on the C client (librdkafka). It explains the rationale behind wrapping the C client for performance and feature completeness, and highlights the release of open-source Python and Go clients. It also mentions the existence of Node.js and .NET clients based on librdkafka.
Advertisement