TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor | NVIDIA Technical Blog

This post details the successful application of TensorRT Edge-LLM on a Jetson AGX Thor platform to achieve a 6.4x speedup in the MLPerf Edge Agentic benchmark compared to a llama.cpp reference. Key technical contributions include the use of NVFP4 quantization for weights and activations, FP8 for the KV cache, tree-based multi-token prediction (MTP) for accelerated decoding, and KV cache reuse across agent turns to reduce prefill computation. The implementation achieved 52.33 tokens per second and completed the benchmark's performance workload in 24 minutes and 36 seconds, demonstrating efficient long-context processing and tool-call acceleration on edge devices.