TechPrufer
Get started
Hugging Face
Hugging Face logoHugging Face · Feature trail

Storage Buckets

1 post · Mar 2026

What this trail covers

Hugging Face introduces Storage Buckets, a new S3-like object storage service integrated with the Hugging Face Hub. Buckets are designed for mutable, non-versioned storage of intermediate ML artifacts such as checkpoints, optimizer states, and logs. They are built on Xet, enabling content deduplication and efficient storage for related files. The service supports pre-warming data to specific cloud regions for improved compute proximity, and can be accessed via the `hf` CLI, Python (`huggingface_hub`), and JavaScript (`@huggingface/hub`). Filesystem integration via `fsspec` allows seamless use with libraries like pandas, Polars, and Dask. Buckets are intended for fast, iterative workflows where version control is not required, complementing the existing versioned repositories for final artifacts.

All Hugging Face trails
  1. Introducing Storage Buckets on the Hugging Face HubThis post introduces Hugging Face Storage Buckets, a new S3-like object storage service for mutable, non-versioned storage of intermediate ML artifacts. It highlights the benefits of using Xet for content deduplication and efficient storage of ML artifacts like checkpoints and logs. The post details how to create, sync, and manage buckets using the `hf` CLI and Python client, and emphasizes filesystem integration via `fsspec` for compatibility with existing data workflows. It also introduces 'pre-warming' to bring data closer to compute regions for improved performance in distributed training and large-scale pipelines.