Nvidia Jetson AGX Thor Speeds Up Qwen3.6 by 6.4x
Nvidia's TensorRT Edge-LLM ran the Qwen3.6-27B model 6.4 times faster than a standard reference on Jetson AGX Thor, showing that complex AI agents can run efficiently on edge hardware.
During the MLPerf Inference v6.1 Edge Agentic benchmark, Nvidia demonstrated that its TensorRT Edge-LLM software running on a single Jetson AGX Thor Developer Kit can execute the Qwen3.6-27B model at 52.33 tokens per second. The system completed all 1,007 turns of the performance workload in 24 minutes and 36 seconds. This represents a 6.4x speedup compared to the llama.cpp reference run, which took 2 hours and 37 minutes using Q4_K_M quantization. The benchmark evaluated the hardware in SingleStream mode with 128 GB of unified memory under the MAXN power mode.
To achieve these speeds, the setup utilized NVFP4 quantization for both weights and activations alongside an FP8 KV cache. NVFP4 is a 4-bit floating-point format natively supported by the Blackwell GPU inside the Jetson AGX Thor. This optimization significantly reduces the memory footprint, leaving more of the 128 GB unified memory available for other tasks. Additionally, the system reused the KV cache and recurrent states across agent turns, serving approximately 96 percent of prompt tokens from a hot cache. This meant the runtime only had to prefill 0.5 million of the 13.6 million total prompt tokens across the entire trajectory.
The implementation also introduced tree-based multi-token prediction (MTP) using an eight-step, top-two, 16-node verification tree. This approach yielded an extra 40 percent decoding performance boost over traditional linear MTP with three draft steps. In terms of responsiveness, the system recorded a median time to first token of 247.12 milliseconds and a median time per output token of 14.68 milliseconds. It also maintained high precision, achieving an overall accuracy of 87.94 percent on the Berkeley Function Calling Leaderboard (BFCL) v4 dataset.
For edge developers and robotics practitioners, these advancements make it highly practical to deploy large, multi-turn AI agents directly on local devices rather than relying on cloud data centers. By drastically reducing latency and memory bottlenecks, developers can build responsive autonomous systems, vehicles, and robots that can reason through complex tool-calling sequences in real time. The necessary configurations, model export settings, and engine build commands are currently available on the TensorRT Edge-LLM release/0.9.1-mlpinf branch.
This is our own summary of reporting by NVIDIA Developer Blog
