| Model | Avg. total throughput(tok/s/GPU) | p99 TTFT | p99 ITL | p50 user throughput(tok/s/user) | Run it* |
|---|---|---|---|---|---|
| 8× RTX PRO 4500 Blackwell | |||||
| Gemma-4-26B-A4B-IT | 4.0× | 3.2× | 7.5× | 7.9× | |
| Nemotron-3-Nano-30B-A3B | 3.2× | 3.0× | 5.9× | 5.5× | |
| Qwen3-32B | 3.3× | 2.5× | 9.8× | 8.1× | |
| 8× RTX PRO 6000 Blackwell | |||||
| Gemma-4-26B-A4B-IT | 2.5× | 3.7× | 8.9× | 9.0× | |
| Nemotron-3-Nano-30B-A3B | 1.8× | 3.6× | 10.2× | 9.1× | |
| Qwen3-32B | 3.4× | 6.4× | 7.0× | 9.1× | |
| 8× H200 | |||||
| Gemma-4-26B-A4B-IT | 1.3× | 1.9× | 7.6× | 7.3× | |
| Nemotron-3-Nano-30B-A3B | 1.4× | 2.7× | 7.3× | 6.3× | |
| Qwen3-32B | 1.6× | 5.7× | 7.5× | 5.7× | |