Streaming and longer context lengths for LLMs on Workers AI

What this post added

Introduced streaming responses for all LLMs on Workers AI using Server-Sent Events (SSE), enabling token-by-token output for improved user experience. Added a full-precision (fp16) Llama-2 model variant and increased context and sequence lengths for existing models to support longer inputs and outputs.