NVIDIA Details Dense vs. MoE Tradeoffs via Nemotron

NVIDIA has released an analysis comparing dense and Mixture-of-Experts (MoE) architectures, highlighting how models like Nemotron 3.5 Lightning decouple memory from compute. While a dense model activates all its parameters for every token, an MoE model routes tokens to a subset of specialized expert networks. For instance, the Nemotron 3.5 Lightning hybrid model has 30B total parameters but activates only 3B per token, utilizing Mamba-2 layers and speculative decoding to achieve high throughput.

The architectural differences translate directly to performance benchmarks. According to Artificial Analysis data, Nemotron 3.5 Lightning achieves an output speed of 235.7 to 494.2 tokens per second (t/s) at a cost of $0.22 per million tokens, scoring 24 on the Intelligence Index. In contrast, the dense Gemma 4 31B model, which activates all 31B of its parameters, reaches 36.9 to 222.4 t/s at $0.40 per million tokens, scoring 30. Meanwhile, the dense Qwen3.8-27B model runs at 46.8 t/s, costing $3.00 per million tokens with an Intelligence Index score of 52. For larger deployments, the MoE Mistral Small 4 has 119B total parameters, activates 6B (8B including embeddings), and delivers 147.3 t/s at $0.60 per million tokens, scoring 20.

Memory requirements also reflect these structural choices. Nemotron 3.5 Lightning requires ~60 GB of native BF16 VRAM on a single H100 GPU, which drops to ~20 GB with 4-bit quantization. Gemma 4 31B needs ~61 GB of BF16 VRAM and ~16 GB in 4-bit, while Qwen3.8-27B requires ~56 GB of BF16 VRAM and ~14 GB in 4-bit. The massive Mistral Small 4 requires ~121 GB of FP8 VRAM on four H100 GPUs, compressing to ~71 GB in 4-bit. Practitioners must note that quantization affects MoE routers and recurrent projections differently than dense components, sometimes requiring linear-attention blocks to remain in BF16 to prevent routing errors.

For developers, choosing between these architectures depends on deployment constraints. MoE models offer superior throughput for single requests, but their latency advantage narrows at high concurrency. Fine-tuning an MoE model also requires care, as full fine-tuning can unbalance the router; techniques like LoRA, parameter-efficient fine-tuning (PEFT), or freezing the router are recommended. These architectural considerations are increasingly vital for physical AI workloads, where platforms like AgiBot GO-1 and Tencent Hy-Embodied-VLM-1.0 are gaining popularity.