Implement Autoregressive Decoding in PyTorch
Problem: Autoregressive Text Generation in PyTorch
Implement an autoregressive text-generation function using PyTorch.
You will be provided with a pre-trained language model that, for a given input tensor of token IDs, returns logits representing the next-token distribution at every position in the sequence.
The model can be invoked as follows:
logits = model(input_ids)
Where:
input_idsis a PyTorch tensor with shape[batch_size, current_sequence_length].logitsis a PyTorch tensor with shape[batch_size, current_sequence_length, vocab_size].- The relevant logits for predicting the next token are located at
logits[:, -1, :].
Your generation function must support the following common decoding strategies:
- Greedy decoding
- Temperature sampling
- Top-k sampling
- Top-p (nucleus) sampling
The generation process should continue iteratively, predicting one token at a time, until one of the following termination conditions is met:
- A maximum number of new tokens (
max_new_tokens) has been generated. - All sequences within the current batch have generated an
eos_token_id(End-Of-Sequence token ID), if aneos_token_idis provided.
Finally, discuss important implementation details and potential edge cases.
AI interview prep
Core dilemma
Efficiently implementing multiple, distinct probabilistic sampling strategies within a batched, iterative generation loop, while managing dynamic sequence lengths and termination conditions for each sequence in the batch. It's about balancing computational efficiency (batching) with the complexity of per-sequence state management and diverse sampling logic.
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Implement Autoregressive Decoding in PyTorch
Problem: Autoregressive Text Generation in PyTorch
Implement an autoregressive text-generation function using PyTorch.
You will be provided with a pre-trained language model that, for a given input tensor of token IDs, returns logits representing the next-token distribution at every position in the sequence.
The model can be invoked as follows:
logits = model(input_ids)
Where:
input_idsis a PyTorch tensor with shape[batch_size, current_sequence_length].logitsis a PyTorch tensor with shape[batch_size, current_sequence_length, vocab_size].- The relevant logits for predicting the next token are located at
logits[:, -1, :].
Your generation function must support the following common decoding strategies:
- Greedy decoding
- Temperature sampling
- Top-k sampling
- Top-p (nucleus) sampling
The generation process should continue iteratively, predicting one token at a time, until one of the following termination conditions is met:
- A maximum number of new tokens (
max_new_tokens) has been generated. - All sequences within the current batch have generated an
eos_token_id(End-Of-Sequence token ID), if aneos_token_idis provided.
Finally, discuss important implementation details and potential edge cases.