TechPrufer
Get started
LinkedIn
LinkedIn logoLinkedIn · Feature trail

Scalable Log Storage System

1 post · Jun 2025

What this trail covers

Northguard is a new log storage system designed for improved scalability and operability, replacing Kafka at LinkedIn. It features a data model based on records, segments, ranges, and topics, with log striping for load balancing. Ranges are used instead of indexed partitions to minimize interruption and facilitate stream processing. Metadata is managed by vnodes, which are Raft-backed replicated state machines.

All LinkedIn trails
  1. Introducing Northguard and Xinfra: scalable log storage at LinkedInThis post introduces Northguard, a new log storage system designed to replace Kafka at LinkedIn, addressing scalability and operability challenges. It details Northguard's data model (records, segments, ranges, topics), its use of log striping for load balancing, and its metadata management via Raft-backed vnodes. The post also contrasts Northguard's range-based approach with indexed partitions, highlighting benefits for stream processing and ordering guarantees. It also introduces Xinfra as a virtualized Pub/Sub layer over Northguard.