Kafka Producer Pipeline for Ruby on Rails - Shopify

This post details the initial implementation of a Kafka producer pipeline for Ruby on Rails applications. It describes the challenges of integrating Kafka into a Ruby/Go ecosystem, the decision to avoid direct Ruby-Kafka links due to operational concerns and lack of mature libraries, and the use of SysV message queues as a reliable buffer between Rails and a Go Kafka producer. It also covers the subsequent adaptation to Docker by introducing a TCP-based proxy to communicate with the host-level Kafka producer, avoiding per-container producers.