Introducing function calling and structured output for open-source and fine-tuned LLMs

This post introduces built-in support for function calling and structured output (JSON mode) within Baseten's TensorRT-LLM Engine Builder. This feature operates at the model server level, using logit biasing within NVIDIA's Triton inference server to enforce output schemas and tool selection during inference. The implementation guarantees 100% output structure with negligible marginal latency, addressing the challenge of integrating unstructured LLM output with structured application requirements. The functionality is available for all LLMs deployed via the Engine Builder and matches OpenAI API specifications.