Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost (2026) - Shopify

Introduced 'gisting', a prompt compression technique that reduces system prompt token count by learning special tokens via knowledge distillation. This technique achieved a 4:1 reduction in system prompt tokens for the Sidekick GraphQL agent, leading to a 19% decrease in median time to first token (TTFT), a 38% decrease in median end-to-end request latency, and a 16% increase in throughput. The method involves training gist embeddings by minimizing KL divergence between teacher and student logits, and can be integrated into continual learning loops.