Boost Training Goodput: How Continuous Checkpointing Optimizes Reliability in Orbax and MaxText- Google Developers Blog

Introduced continuous checkpointing in Orbax and MaxText to optimize training job reliability and goodput. This feature intelligently initiates asynchronous checkpoint saves only upon successful completion of the preceding save, maximizing host machine and I/O bandwidth utilization and minimizing risks from hardware failures or preemptions. Configuration flags like `enable_checkpointing`, `async_checkpointing`, and `enable_continuous_checkpointing` are detailed. Benchmarks demonstrate smaller P50 checkpoint intervals with continuous checkpointing, with an anticipated increase in average training step time. The post also outlines more flexible Orbax options for saving and preserving checkpoints using customizable policies like `minimum_interval_secs` and `EveryNSeconds`, and provides an example of a custom preservation policy based on evaluation results. It addresses concerns regarding multi-slice I/O and recommends co-locating storage clusters with training clusters.