GPT vs Llama: Migrate to open source LLMs seamlessly

This post introduces a new capability for Baseten: an OpenAI-compatible ChatCompletions API endpoint that acts as a bridge to deploy and serve open-source LLMs. It details the minimal code modifications (three lines) required to switch from using OpenAI's GPT models to open-source alternatives like Llama 3.2 11B. The post explains how to deploy these models on Baseten, obtain model IDs, and configure the OpenAI Python client with a custom `base_url` pointing to Baseten's bridge endpoint. It also covers how to enable streaming inference with these open-source models. The technical delta includes the implementation of the `bridge.baseten.co` endpoint and the demonstration of its compatibility with the OpenAI client library for both standard and streaming API calls.