Introducing Northguard and Xinfra: scalable log storage at LinkedIn

This post introduces Northguard, a new log storage system designed to replace Kafka at LinkedIn, addressing scalability and operability challenges. It details Northguard's data model (records, segments, ranges, topics), its use of log striping for load balancing, and its metadata management via Raft-backed vnodes. The post also contrasts Northguard's range-based approach with indexed partitions, highlighting benefits for stream processing and ordering guarantees. It also introduces Xinfra as a virtualized Pub/Sub layer over Northguard.