Adopting OpenTelemetry for our logging pipeline

This post details the migration of Cloudflare's logging pipeline from syslog-ng to OpenTelemetry Collector. Key technical contributions include the development of four internal OpenTelemetry Collector components: cfjs1exporter (to support Cloudflare's internal cfjs1 log format), fileexporter (a fork supporting plain text and RFC 5424 syslog formats with custom permissions), externaljsonprocessor (to fetch external data like data center status and add it as fields to logs), and ratelimit processor (to apply rate limits based on log message fields). The migration process involved using OCB to build an internal distribution and phased rollouts across edge and core data centers, with specific salt states and temporary migration pieces. Lessons learned include the need for robust failover mechanisms in the cfjs1 exporter and the importance of handling niche services and 'Internet badness'.