How to build function calling and JSON mode for open-source and fine-tuned LLMs

This post details the engineering implementation of function calling and structured output for LLMs deployed on Baseten. It explains how a modified Triton inference server ('Briton') uses logit biasing to enforce valid token generation. A state machine, generated from a regex derived from an Outlines-generated schema, is serialized to Protocol Buffers and loaded into the server. This state machine efficiently masks invalid tokens by applying pre-computed token masks based on the current state, ensuring minimal performance impact. The implementation follows OpenAI API specifications for compatibility.