Editing Files at 1000 Tokens per Second

Introduced a "fast-apply" model and "speculative edits" technique to achieve ~1000 tokens per second for code edits. This involved training custom models (e.g., Llama-3-70b-ft) using synthetic data and optimizing inference for full-file code rewrites, demonstrating significant speedups over vanilla inference and existing speculative decoding methods. The post details the evaluation methodology for prompted rewrites, the rationale for full-file rewrites over diffs, and the training pipeline for the specialized model. Future directions include long-context training and knowledge distillation.