TechPrufer
Get started
NVIDIA
NVIDIA logoNVIDIA · Feature trail

Local Network Inference Routing

4 posts · May 2026 – Sep 2026

What this trail covers

This feature thread tracks the evolution of local network inference routing, enabling distributed AI workloads across multiple compatible systems. Initial efforts focused on addressing multi-agent bottlenecks by introducing NVIDIA PAIR, a virtual inference router that distributes independent inference requests to available nodes on a local network. PAIR supports familiar interfaces like Ollama and LM Studio without requiring agent harness changes. It leverages mDNS for discovery, MTLS for encryption, and now TensorRT Edge-LLM on Jetson AGX Thor for accelerated agentic AI inference, achieving significant performance gains in benchmarks like MLPerf Edge Agentic through techniques such as NVFP4 quantization, KV cache reuse, and tree-based multi-token prediction.

All NVIDIA trails
  1. TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor | NVIDIA Technical BlogThis post details the successful application of TensorRT Edge-LLM on a Jetson AGX Thor platform to achieve a 6.4x speedup in the MLPerf Edge Agentic benchmark compared to a llama.cpp reference. Key technical contributions include the use of NVFP4 quantization for weights and activations, FP8 for the KV cache, tree-based multi-token prediction (MTP) for accelerated decoding, and KV cache reuse across agent turns to reduce prefill computation. The implementation achieved 52.33 tokens per second and completed the benchmark's performance workload in 24 minutes and 36 seconds, demonstrating efficient long-context processing and tool-call acceleration on edge devices.
  2. How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra | NVIDIA Technical BlogThis post details how full-stack NIM optimizations, including autotuned kernels, tensor parallelism, prefix and state reuse, scheduler and memory tuning, and MTP speculative decoding, deliver up to 2.5x higher system throughput for Nemotron 3 Ultra on a 4xB200 system compared to a baseline serving stack. It introduces the concept of a Pareto chart for evaluating performance trade-offs between interactivity and throughput, and provides guidance on benchmarking NIM with NVIDIA AIPerf for representative workloads. It also outlines the deployment process for Nemotron 3 Ultra NIM, including selecting optimized profiles for agentic workloads.
  3. High-Throughput Structure Prediction with BioNeMo Inference Runtime | NVIDIA Technical BlogThis post introduces the NVIDIA BioNeMo Inference Runtime (BioIR) for accelerating biomolecular structure-prediction models on NVIDIA GPUs. It details two integration methods: an end-to-end processor and direct PyTorch module integration. The post highlights BioIR's optimization layers (kernel selection, CUDA Graph capture, pipeline scaling via Ray replicas) and presents benchmark results demonstrating a 2.90x improvement in residue-normalized throughput and significant energy savings compared to a torch-compiled open-source implementation. It also provides practical steps for using the serial and Ray executors for validation and scaling.
  4. When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving | NVIDIA Technical BlogThis post introduces NVIDIA Dynamo, an open-source inference framework that implements encode-prefill-decode (EPD) disaggregation for multimodal model serving. It details how separating vision encoding from LLM prefill and decode stages can accelerate inference, especially for image-heavy prompts and quantized MoE models. The post outlines three encoder placement topologies (aggregated, colocated, disaggregated) and analyzes the factors influencing EPD benefits, such as input media load, output sequence length, model size/precision, and traffic mix. It also provides experimental results demonstrating significant improvements in time to first token (TTFT) and end-to-end response time, along with next steps for users to reproduce experiments and further optimize performance.
  5. Building a Memory-Driven Agent with NVIDIA NemoClaw | NVIDIA Technical BlogThis post introduces NVIDIA NemoClaw, a framework for building memory-driven AI agents. It details the 'self model' concept for maintaining structured knowledge (people, projects, priorities) in Markdown and a SQLite ledger for obligations and judgments. It highlights the importance of separating evidence from knowledge and judgment, prioritizing user intent over urgency, allowing user corrections via an audit trail, and enforcing runtime security with NVIDIA OpenShell. Evaluation shows a significant improvement in agent accuracy (90.9% vs 82.8%) and tracking of changing facts (100% vs 60.0%) compared to a RAG baseline. This extends the capability of local network inference routing by providing a structured memory layer for agents operating within such networks.
  7. Frontier Reasoning Reaches the Edge: How to Deploy and Optimize Models on NVIDIA Jetson | NVIDIA Technical BlogThis post details how to deploy and optimize compact open models (Nemotron 3.5 Lightning, Qwen3.8-27B) on NVIDIA Jetson for edge reasoning and agentic capabilities. It introduces NVFP4 quantization and speculative decoding as key optimization techniques, demonstrating up to 6.28x decode throughput speedup. The post also provides guidance on choosing between MoE and dense model architectures based on workload requirements and emphasizes the importance of application-level validation for specific prompts and checkpoints.
  8. NVIDIA PAIR Virtual Inference Router Expands Available Compute on Your Local Network | NVIDIA Technical BlogThis post announces the beta release of NVIDIA PAIR, a virtual inference router designed to distribute AI inference requests across compatible systems on a local network. It addresses multi-agent bottlenecks by routing independent requests to available nodes, supporting Ollama and LM Studio without requiring agent harness modifications. PAIR utilizes mDNS for discovery and MTLS for encryption, with scheduling logic considering node readiness, engine state, model presence, and GPU utilization. A demonstration showed a three-device PAIR cluster completing a five-subagent workload significantly faster than a single device.
  9. Route AI Agents Across Models with NVIDIA NeMo Switchyard | NVIDIA Technical BlogThis post introduces NVIDIA NeMo Switchyard, an intelligent orchestration layer for AI agents that enables dynamic routing of workloads across specialized and frontier models. It details how Switchyard's provider-agnostic SDK, separation of routing logic from model providers, and support for various routing algorithms (tuning-free and tunable) allow for efficient, cost-effective, and controllable agent workflows. The post highlights the decision-making signals for routers (model capabilities, cost, infrastructure) and demonstrates practical benefits through benchmarking and integration examples with LangChain and Cognition.
  10. Run Local Agentic AI Workflows with Meta’s Muse Glimmer on NVIDIA | NVIDIA Technical BlogThis post details the local deployment and performance of Meta's Muse Glimmer, a 30B parameter dense model with a 120K+ context window, on various NVIDIA hardware platforms (GeForce RTX 5090, DGX Spark, DGX Station, Jetson). It highlights the model's suitability for long-running agentic workflows due to its dense architecture, which avoids MoE routing overhead and offers predictable latency and reliability. Performance metrics of over 20K tokens/sec/GPU on Blackwell Ultra are provided. The post also outlines flexible deployment paths using NVIDIA NIM, SGLang, and vLLM, and discusses fine-tuning with NeMo AutoModel and NeMo RL. It emphasizes privacy by design for agentic workflows requiring on-device inference.
  11. Building AI Agents for AR Glasses and XR Devices with NVIDIA XR AI | NVIDIA Technical BlogThis post introduces NVIDIA XR AI, an open-source library and framework for building intelligent agents for AR glasses and XR devices. It extends the concept of local network inference routing to XR by enabling agents to connect to GPU-accelerated AI services for real-time visual and voice interaction, enterprise data access, and tool integration. The architecture is modular, separating media transport, model services (Cosmos, Nemotron), enterprise connectivity (MCP), and agent orchestration (NeMo Agent Toolkit). It details how to get started with a public beta, including cloning the repository, starting AI services (parakeet, Cosmos, Nemotron, Llama-3.1-Nemotron-Nano-8B-v1, NVIDIA-Nemotron-3-Nano-30B-A3B), running a simple multimodal agent, connecting enterprise data via MCP, and adding agent orchestration. The post highlights the use of visual grounding, voice-first interaction, and flexible deployment across various XR devices and clients.
  12. Build On-Device AI Companions with the NVIDIA ACE Game Agent SDK and Unreal Engine 5 Plugins | NVIDIA Technical BlogThis post introduces the NVIDIA ACE Game Agent SDK Beta and associated Unreal Engine 5 plugins, extending the local network inference routing capability. The SDK provides a C/C++ agentic framework with Agent, Chat, and RAG APIs for on-device AI companions in games. The Unreal Engine plugins offer Blueprint and C++ integration for ASR, SLM (with Qwen 3.5 4B), and TTS (with Chatterbox Turbo 350M), enabling low-latency, real-time conversational AI and dynamic NPC behavior directly within game engines. This builds upon existing local inference routing by specifically targeting game development scenarios and integrating with game engines.
  13. Run DiffusionGemma on NVIDIA for Developer-Ready, High-Throughput Text Generation | NVIDIA Technical BlogThis post details the optimization and deployment of DiffusionGemma, a diffusion-based text generation model, on NVIDIA platforms. It highlights the model's ability to generate tokens in parallel, achieving high throughput (up to 1,000 tokens/sec on H100). The post provides guidance on running DiffusionGemma on various NVIDIA hardware (H100, DGX Spark, DGX Station, RTX/RTX PRO) using tools like Hugging Face Transformers and vLLM. It also covers enterprise deployment via NVIDIA NIM with an OpenAI-compatible API and fine-tuning capabilities using NVIDIA NeMo AutoModel. The model supports BF16 and NVFP4 precision formats and has a context length of up to 256K tokens.
  14. Run Local AI Agents with Faster Models and Multi-Node Clustering on NVIDIA DGX Spark | NVIDIA Technical BlogThis post details the integration of NVIDIA NemoClaw for streamlined local AI agent deployment on DGX Spark, including a simplified installation process, support for Qwen3.6-35B with NVFP4 quantization for improved inference speed (2.6x throughput increase), and the introduction of NVIDIA Sync's cluster assistant for multi-node clustering (2-4 DGX Spark units) to provide unified memory for larger models and multi-agent pipelines. It also highlights OpenShell for secure, sandboxed execution and local Ollama integration.
  15. Add a Specialized Deep Research Skill to Agent Harnesses | NVIDIA Technical BlogThis post introduces the AI-Q skill as a specialized deep research capability that can be integrated into agent harnesses like Claude Code and Codex. This skill allows these harnesses to delegate complex research tasks, including multi-document synthesis and decision briefs, to a dedicated AI-Q server. The AI-Q server handles the entire research pipeline, from intent classification to citation generation, and can be deployed within an enterprise environment to maintain data sovereignty. The post also details how AI-Q integrates with authenticated MCP servers as data sources, supporting various authentication patterns (no per-user auth, backend/app credentials, forwarding user bearer tokens) through NeMo Agent Toolkit function groups. This enables agents to access enterprise data without rebuilding retrieval logic within the harness itself.