Safely sample production data into pre-production environments with Logstash
10/1/2024
Introduces a new Logstash pattern for safely sampling production data into pre-production environments. This pattern utilizes UDP for pipeline-to-pipeline communication, enabling a subset of production events to be sent to a pre-production cluster without blocking the production pipeline. It contrasts this with the output isolator pattern, highlighting reduced operational risks and performance overhead by avoiding persistent queues. The post provides detailed Logstash pipeline configurations for both the production (common) and pre-production pipelines, demonstrating the use of a Ruby filter for sampling and UDP output.