TechPrufer
Sign inGet started

Blogs›Shopify›Marshal Serialization Replacement

Marshal Serialization Replacement

Marshal Serialization Replacement

2
posts
2022

Shopify has successfully migrated its caching mechanisms away from Ruby's Marshal serialization to MessagePack. This involved developing custom extension types for common Ruby objects like Date, Symbol, Time, DateTime, BigDecimal, and ActiveRecord::Base. The migration was phased, running both formats in parallel for six months, logging unencodable objects to identify necessary extension types and potential cache misuse. The new system uses a version byte prefix to distinguish between MessagePack and Marshal payloads during the transition. The primary benefit is improved safety and reliability by mitigating risks associated with code changes breaking cache payloads.

2022

Caching Without Marshal Part 2: The Path to MessagePack - Shopify

11/15/2022

This post details the technical implementation of replacing Ruby's Marshal serialization with MessagePack for caching. It explains the MessagePack format, its core types, and the crucial concept of extension types for handling custom Ruby objects. Specific examples are provided for encoding Date objects and the challenges of serializing ActiveRecord::Base records with associations, leading to the development of an `ActiveRecordCoder`. The post also outlines the migration strategy, including a phased rollout with a version byte prefix and logging of unencodable objects to identify new extension types or problematic cache usage. The core problem addressed is preventing cache corruption due to code changes by using a generic, extensible serialization format.

Caching Without Marshal Part 1: Marshal from the Inside Out - Shopify

11/11/2022

This post details the risks and limitations of using Ruby's Marshal for serialization in caching, particularly highlighting a production incident caused by Marshal's behavior with class changes. It explains Marshal's internal encoding mechanisms for objects, instance variables, and core types like strings and arrays, and introduces MessagePack as a safer alternative. The post sets the stage for a deeper dive into the implementation of MessagePack for caching.