Marshal Serialization Replacement

2 posts · Nov 2022

What this trail covers

Shopify has successfully migrated its caching mechanisms away from Ruby's Marshal serialization to MessagePack. This involved developing custom extension types for common Ruby objects like Date, Symbol, Time, DateTime, BigDecimal, and ActiveRecord::Base. The migration was phased, running both formats in parallel for six months, logging unencodable objects to identify necessary extension types and potential cache misuse. The new system uses a version byte prefix to distinguish between MessagePack

All Shopify trails

Nov 2022Caching Without Marshal Part 2: The Path to MessagePack - ShopifyThis post details the migration from Ruby's Marshal serialization to MessagePack for caching. It explains the differences between Marshal and MessagePack formats, focusing on MessagePack's generic nature and its support for extension types. The post describes how Shopify implemented custom extension types for Date, Symbol, Time, DateTime, BigDecimal, and ActiveRecord::Base. For ActiveRecord objects, a custom serializer (`ActiveRecordCoder`) was developed to handle attributes and associations, including circular references, which Marshal handles automatically but MessagePack does not. A phased migration strategy was employed, running both formats in parallel for six months, using a version byte prefix to differentiate payloads and logging unencodable objects to identify necessary extension types and problematic cache usage.
Nov 2022Caching Without Marshal Part 1: Marshal from the Inside Out - ShopifyThis post details the internal workings of Ruby's Marshal serialization format, explaining its encoding of objects, instance variables, and circular references. It highlights the risks associated with Marshal's ability to serialize arbitrary Ruby objects, which can lead to production incidents when code changes interact with cached data. The post sets the stage for a subsequent discussion on replacing Marshal with MessagePack.
Advertisement