Sub-3 millisecond named entity recognition (NER) inference

This post details the technical optimizations made to Baseten Embeddings Inference (BEI) to achieve sub-3 millisecond inference for Named Entity Recognition (NER) models. Key contributions include: a Rust-based web server and tokenizer to minimize per-request overhead, Nginx configuration for a memory-backed and lightweight request path, HTTP/2 connection pooling in the Baseten Performance Client to reuse connections, and the introduction of a `/predict_tokens` default route for NER models. It also provides an example YAML configuration for deploying a BERT-family NER model and a Python script for retrieving token-level predictions.