TechPrufer
Get started
Physical Intelligence
Physical Intelligence logoPhysical Intelligence · Feature trail

Real-Time Action Chunking

3 posts · Jan 2025 – Jun 2025

What this trail covers

The FAST tokenizer continues to evolve, enabling faster training and zero-shot generalization of autoregressive transformer policies for dexterous robotic tasks. The latest iteration, Real-Time Chunking (RTC), allows for real-time execution without discontinuities, significantly speeding up execution time and improving robustness to latency, even with high inference delays. RTC is applied without training-time changes to existing VLA models, and can be adapted to diffusion or flow-based VLAs by posing chunk consistency as an inpainting problem.

All Physical Intelligence trails
  1. Real-Time Action Chunking with Large ModelsIntroduced Real-Time Chunking (RTC) as a method to enable real-time execution of Vision-Language-Action (VLA) models without discontinuities. RTC treats chunk consistency as an inpainting problem, adapting diffusion/flow model inpainting algorithms to partially attend to overlapping timesteps between action chunks. This allows for continued execution of previous chunks while new ones are generated, significantly reducing pauses and improving robustness to inference latency (up to +200ms) without requiring training-time changes to existing VLA models.
  2. Teaching Robots to Listen and Think HarderIntroduces the 'Hi Robot' system, a hierarchical inference process for VLA models. This system uses a high-level VLM ('System 2') to reason through complex prompts and language, generating intermediate steps for a low-level VLA model ('System 1'). The high-level policy is trained using synthetic data and can incorporate real-time user feedback. The post presents quantitative results showing improved instruction-following accuracy and task progress compared to flat VLA policies and GPT-4o.
  3. FAST: Efficient Robot Action TokenizationIntroduced the FAST (Efficient Robot Action Tokenization) method, which uses Discrete Cosine Transform (DCT) combined with Byte Pair Encoding (BPE) to compress robot action sequences into discrete tokens. This approach significantly improves training efficiency (5x faster) and enables autoregressive transformer models to handle highly dexterous tasks with precision, achieving performance comparable to diffusion models while allowing for faster training and better generalization.