We terminated a TPU mid-training and it recovered in seconds: Introduction to elastic training with MaxText- Google Developers Blog

What this post added

Introduced and demonstrated elastic training for large model training on Cloud TPUs using MaxText, Pathways, and Orbax. This capability allows the training process to recover from hardware failures (e.g., TPU termination) within seconds by catching `JaxRuntimeError` exceptions, restoring from the latest viable checkpoint using Orbax, and resuming training within the same process, significantly reducing downtime compared to traditional job restarts. The post details the three key components enabling this: Pathways' failure detection, the `elastic_retry` decorator from `pathways-utils`, and Orbax's checkpointing mechanism for safe restoration.