Implement KV cache for inference
Design Task: Key–Value Cache for Transformer Decoder Inference
Context
You are tasked with building an autoregressive inference engine for a Transformer decoder-only model. To optimize performance and avoid redundant computations, you must implement a Key–Value (K/V) caching mechanism at the per-layer level for the self-attention mechanism.
Assume a standard multi-head self-attention decoder with the following parameters:
- Batch size B (or effective batch B_eff when using beams)
- Model dimension d_model
- Number of attention heads H
- Head dimension d_k = d_model / H
- Maximum generation length L_max
- Support for mixed precision (float16/bfloat16)
Requirements
Your design and implementation of the K/V caching system must address the following:
- Tensor Shapes: Clearly specify the tensor shapes required for both the prefill (processing the initial prompt) and the per-step decode phases, accommodating both greedy and beam search.
- API Definition: Define a robust API to manage and maintain these per-layer caches across multiple decoding steps and for batched inputs.
- Memory Management: Handle cache growth and memory limits effectively. This includes considering strategies like preallocation and providing an option for paged or block allocation.
- Search Algorithm Support: Fully support both greedy search and beam search. For beam search, this involves correctly expanding and reordering caches at each step, as well as handling End-Of-Sequence (EOS) tokens.
- Performance Analysis: Include a complexity analysis of your design, detailing expected speedups compared to recomputing self-attention without caching.
- Testing Considerations: Outline tests for correctness, including critical edge cases such as EOS tokens appearing in the middle of a batch, variable prompt lengths within a batch, and ensuring the correctness of cache reordering during beam search.
Provide a teaching-oriented solution, including step-by-step reasoning, relevant formulas, and code-style pseudocode for the API and critical execution paths.
AI interview prep
Core dilemma
Balancing the computational savings of K/V caching against the significant memory overhead and complexity of managing dynamic cache states, especially with batched, variable-length inputs and advanced search algorithms like beam search.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Implement KV cache for inference
Design Task: Key–Value Cache for Transformer Decoder Inference
Context
You are tasked with building an autoregressive inference engine for a Transformer decoder-only model. To optimize performance and avoid redundant computations, you must implement a Key–Value (K/V) caching mechanism at the per-layer level for the self-attention mechanism.
Assume a standard multi-head self-attention decoder with the following parameters:
- Batch size B (or effective batch B_eff when using beams)
- Model dimension d_model
- Number of attention heads H
- Head dimension d_k = d_model / H
- Maximum generation length L_max
- Support for mixed precision (float16/bfloat16)
Requirements
Your design and implementation of the K/V caching system must address the following:
- Tensor Shapes: Clearly specify the tensor shapes required for both the prefill (processing the initial prompt) and the per-step decode phases, accommodating both greedy and beam search.
- API Definition: Define a robust API to manage and maintain these per-layer caches across multiple decoding steps and for batched inputs.
- Memory Management: Handle cache growth and memory limits effectively. This includes considering strategies like preallocation and providing an option for paged or block allocation.
- Search Algorithm Support: Fully support both greedy search and beam search. For beam search, this involves correctly expanding and reordering caches at each step, as well as handling End-Of-Sequence (EOS) tokens.
- Performance Analysis: Include a complexity analysis of your design, detailing expected speedups compared to recomputing self-attention without caching.
- Testing Considerations: Outline tests for correctness, including critical edge cases such as EOS tokens appearing in the middle of a batch, variable prompt lengths within a batch, and ensuring the correctness of cache reordering during beam search.
Provide a teaching-oriented solution, including step-by-step reasoning, relevant formulas, and code-style pseudocode for the API and critical execution paths.