Trading off compute for memory with activation checkpointing

This post introduces a new activation checkpointing procedure that addresses limitations in existing PyTorch tools. The new procedure involves a planning pass to compile the model on fake tensors, capturing graph structures and min-cut save sets. It then employs a greedy algorithm to iteratively drop saves with the cheapest recompute cost per byte of peak memory freed, until a user-specified memory budget is met. The planner estimates peak memory by traversing forward and backward graphs and tracking activation lifetimes, and also accounts for parameters, buffers, optimizer state, and user-provided reserves. This provides a more accurate and controllable method for managing memory usage during model training.