Tulip: Schematizing Meta’s data platform

What this post added

Introduced Tulip, a new binary serialization protocol designed for Meta's data platform. Tulip addresses challenges in schema evolution, reliability, and efficiency by replacing legacy formats like Hive Text Delimited and JSON. It utilizes Thrift's TCompactProtocol and assigns field IDs through a data platform management module to ensure forward and backward compatibility, reduce data size (40-85% reduction), and decrease CPU cycles for serialization/deserialization (50-90% reduction).