TechPrufer
Get started
Jane Street
Jane Street logoJane Street · Feature trail

Activation Checkpointing

3 posts · Sep 2026

What this trail covers

This post introduces a new capability for activation checkpointing in model training. It details a novel procedure for finding optimal nodes to checkpoint to trade off compute for memory, addressing limitations in existing PyTorch tools. The procedure involves a planning pass to estimate true memory usage and recomputation costs, followed by a greedy algorithm to select nodes for checkpointing based on the cheapest recompute cost per byte of peak memory freed. This capability aims to provide use cases for autoregressive diffusion models in quantitative finance, specifically for generating market data. The post details the challenges of modeling market data, which exhibits both continuous and discrete characteristics, and explores solutions using diffusion and flow-matching models. It highlights findings on the instability of DDPM and the better performance of flow matching. The post also discusses techniques for handling discontinuities in market data, such as fracturing categorical predictions and using 'atom smoothing'. Finally, it presents results comparing model-generated data to real market data using total variational divergence and a discriminator classifier.

All Jane Street trails
  1. Can you use autoregressive diffusion to generate market data?This post introduces a novel approach to generating market data using autoregressive diffusion models. It details the technical challenges of modeling market data's mixed continuous and discrete nature, comparing DDPM and flow matching, and proposing 'atom smoothing' and fractured categorical predictions to handle discontinuities. The contribution lies in adapting diffusion models for financial time-series generation and exploring specific techniques to address the jaggedness of market data.
  2. Mitigating memorization in LLMsThis post explores mitigating memorization in LLMs by investigating methods like divergence decoding, distillation, prompt engineering, and natural language autoencoders. It details experiments on a cricket match dataset to measure and reduce reliance on memorized pretraining knowledge, with divergence decoding showing promise in reducing pre-cutoff performance without harming post-cutoff performance.
  3. Trading off compute for memory with activation checkpointingThis post introduces a new activation checkpointing procedure that addresses limitations in existing PyTorch tools. The new procedure involves a planning pass to compile the model on fake tensors, capturing graph structures and min-cut save sets. It then employs a greedy algorithm to iteratively drop saves with the cheapest recompute cost per byte of peak memory freed, until a user-specified memory budget is met. The planner estimates peak memory by traversing forward and backward graphs and tracking activation lifetimes, and also accounts for parameters, buffers, optimizer state, and user-provided reserves. This provides a more accurate and controllable method for managing memory usage during model training.