Replicate

Streaming output for language models

Aug 14, 2023
Streaming output for language models
SummaryAI, from the source post

Introduced server-sent event streams for language models, allowing for real-time token-by-token output. Provided Node.js and cURL examples for consuming these streams. Documented how to add streaming support to custom Cog models by yielding tokens.

More from the Language Model Deployment and Fine-tuning trail