Decrease PyTorch Model Load Times with CoreWeave’s Tensorizer | CoreWeave

This post introduces CoreWeave Tensorizer, a tool designed to drastically reduce PyTorch model loading times. It achieves this through a bespoke serialization format that enables 'zero-copy' tensor streaming directly from S3/HTTP endpoints or local storage to the GPU. Key technical contributions include: a novel serialization format placing metadata at the beginning of a single binary file for quick loading; 'tensor streaming' that pulls model chunks over the network wire speed to the GPU, minimizing RAM usage; and a comparison demonstrating >5x faster average latency per request compared to Hugging Face when scaling from zero, with fewer pod spin-ups and less RAM required. Tensorizer also offers sharding capabilities via a filter function and HTTP range support, and mitigates security risks associated with PyTorch's default pickle format by using a single binary file.