Adopting OpenTelemetry for our logging pipeline

What this post added

This post details the migration of Cloudflare's large-scale logging pipeline from syslog-ng to the OpenTelemetry Collector. It covers the motivations for the shift, including the use of Go for easier contributions, improved build chains with Post-Quantum cryptography, enhanced telemetry via Prometheus metrics, and unification with existing OpenTelemetry Collector usage for tracing. The post describes the development of four internal OpenTelemetry Collector components: `cfjs1exporter` for custom log formatting, `fileexporter` with custom permissions and formats, `externaljsonprocessor` for enriching logs with external data, and `ratelimit processor` for service-level rate limiting. It also outlines the migration process for edge and core data centers, including the use of Salt states and manual onboarding for custom configurations. Lessons learned include the importance of robust failover mechanisms for central logging servers and the need for more granular instrumentation around niche services and internet 'badness'.