We terminated a TPU mid-training and it recovered in seconds: Introduction to elastic training with MaxText- Google Developers Blog

This post introduces and demonstrates elastic training for large language models on Cloud TPUs using JAX, MaxText, Pathways, and Orbax. It details how the system can detect hardware failures (TPU slice termination) as Python exceptions, recover in place by reloading the last viable checkpoint, and resume training with minimal downtime (under two minutes in the example). Key components enabling this are: 1) Pathways detecting the failure and surfacing it as a `jax.errors.JaxRuntimeError`, 2) the `elastic_retry` decorator from `pathways-utils` catching this exception and initiating recovery, and 3) Orbax ensuring only complete checkpoints are restored. The post contrasts this with traditional restart-based recovery, highlighting the savings in teardown and rescheduling costs. It also differentiates elastic training from planned suspend-resume for Spot TPUs.