TechPrufer
Get started
Baseten
Baseten logoBaseten · Feature trail

Model Inference Metrics Export

1 post · Oct 2024 – Dec 2024

What this trail covers

Baseten's platform allows users to export model inference metrics to external observability tools, integrating with platforms like Grafana, Datadrog, New Relic, and Prometheus via OpenTelemetry. This provides a unified view of AI application performance within existing observability stacks, facilitating debugging and optimization. The platform also introduces activity logging for workspace changes and request-level LLM metrics (request size, response size, time to first byte) to aid in performance analysis and debugging. The metrics interface is now customizable, allowing users to organize and prioritize displayed metrics and directly access relevant logs from metric plots.

All Baseten trails
  1. New observability features: activity logging, LLM metrics, and metrics dashboard customizationThis post introduces three new observability features: activity logging for tracking workspace and model changes, request-level LLM metrics (request size, response size, time to first byte) to contextualize inference latency, and a customizable metrics dashboard. The dashboard offers a unified view, drag-and-drop organization of graphs, and one-click access to logs from metric plots. These features enhance the ability to monitor, debug, and optimize AI workloads.
  2. Export your model inference metrics to your favorite observability toolThis post introduces a new integration for exporting model inference metrics from Baseten to external observability platforms. The implementation uses OpenTelemetry and exposes a `/metrics` endpoint compatible with Prometheus scraping. It details the types of metrics available (request count, response time, replica count, hardware usage) and provides configuration examples for Prometheus. The goal is to enable users to incorporate AI model performance into their existing production monitoring and alerting workflows.