.NET Kafka Client and TPL Integration

12/8/2022

This post details the architecture and implementation of a streaming application using Apache Kafka and the .NET client library. It specifically highlights the integration of the Task Parallel Library (TPL), particularly its Dataflow components (BufferBlock and TransformBlock), to manage concurrency and parallelism for high-throughput event processing. The post explains how to construct a pipeline with source, transform, and target blocks, demonstrating how to wrap Kafka consumers and producers within these blocks and manage backpressure and ordered processing with parallelism.

10/24/2019

This post introduces building streaming applications using Apache Kafka and the .NET client library, leveraging the Task Parallel Library (TPL) for in-process parallelization. It details the use of TPL Dataflow blocks (BufferBlock and TransformBlock) to create a pipeline for consuming, processing (including simulated CPU-intensive tasks), and producing data, emphasizing throughput optimization and maintaining data order. This post also introduces using Rust with Apache Kafka, exploring the `kafka` and `rdkafka` crates for building Kafka producers and consumers. It details the challenges of integrating Avro serialization with Rust clients and demonstrates how to use the `schema_registry_converter` crate for this purpose. Performance tests are included to compare Rust with Java applications using Kafka.

6/7/2017

This post introduces the Confluent Python client (`confluent-kafka-python`) which leverages the high-performance C client `librdkafka`. It provides a guide on installing the client, starting a local Kafka cluster using Confluent Platform, producing messages with callbacks and flush/poll methods, and consuming messages with consumer groups and offset management. It also mentions advanced features like partition assignment hooks and Avro serialization with Schema Registry.

11/2/2016

This post details the development of .NET and Go Kafka clients based on the C client (librdkafka). It explains the rationale behind wrapping the C client for performance and feature completeness, and highlights the release of open-source Python and Go clients. It also mentions the existence of Node.js and .NET clients based on librdkafka.