TechPrufer
Get started
Netflix
Netflix logoNetflix · Feature trail

Cassandra Data Movement

2 posts · Nov 2020 – Jun 2026

What this trail covers

Netflix has evolved its Cassandra data movement capabilities by replacing the monolithic Casspactor engine with a new layered architecture. This new system leverages a core S3 reading capability (Cassandra Analytics Wrapper) to produce standard Spark DataFrames, enabling a 'Connector Factory' model. This allows for purpose-built, data-model-aware connectors for various Cassandra data abstractions (Key Value, Time Series, etc.), improving efficiency, reducing intermediate storage bloat, enhancing performance, and enabling time travel.

All Netflix trails
  1. The Evolution of Cassandra Data Movement at NetflixThis post details the evolution of Netflix's Cassandra data movement from the monolithic Casspactor to a new layered architecture. The new system replaces Casspactor's fragile metadata dependencies with direct S3 reads and introduces a 'Connector Factory' model. This allows for purpose-built connectors for various Cassandra data abstractions (Key Value, Time Series, etc.) that operate on Spark DataFrames, eliminating intermediate table bloat, handling skewed partitions, and enabling time travel. The architecture also focuses on performance gains and significant cost savings.
  2. Bulldozer: Batch Data Moving from Data Warehouse to Online Key-Value StoresThis post introduces Bulldozer, a self-serve data platform for batch data movement from data warehouses to online key-value stores. Bulldozer leverages Spark for data processing and a Key-Value Data Abstraction Layer (KV DAL) to decouple applications from specific datastore APIs. It supports configuration-based job definitions via YAML, auto-generates Protobuf schemas for serialization, and ensures data integrity and seamless transitions for consumers. The system handles version control of datasets, ensuring atomic writes and seamless reads of new data versions.