# Systalyze Pushes the LLM Inference Frontier: Up to 10× Faster Serving

Systalyze · September 25, 2026 · https://www.systalyze.com/blog/systalyze-achieves-10x-faster-serving

> How Systalyze composes serving optimizations across Gemma 4, Nemotron and Qwen3-32B, achieving up to 10× faster serving and cutting cost per output token by up to 73%.

## Optimizing Inference Performance

We took three open models with very different architectures—Gemma-4-26B-A4B-IT, Nemotron-3-Nano-30B-A3B, and Qwen3-32B—and optimized their serving on three hardware platforms. Against vLLM on the same hardware, we improved per-token latency (p99 inter-token latency, or ITL) by up to 8.9× for Gemma, 10.2× for Nemotron, and 9.8× for Qwen.

<iframe
  src="/embeds/systalyze-10x/table-1.html"
  title="Speedup with Systalyze over the vLLM baseline"
  className="post-embed post-embed--table"
  loading="lazy"
></iframe>

## Reproduce it

Everything behind these results is public. Start from the artifacts, then run any configuration end to end: each ships as a container with the benchmark built in. You will need Docker with the NVIDIA container runtime and approximately 100 GB of free disk.

<iframe
  src="/embeds/systalyze-10x/reproduce.html"
  title="Artifacts and commands for reproducing the serving benchmarks"
  className="post-embed post-embed--reproduce"
  loading="lazy"
></iframe>

### Baseline checkpoints and hardware notes

Download the BF16 checkpoint to the path on `baseline.sh`’s `# model` line:

<iframe
  src="/embeds/systalyze-10x/baseline.html"
  title="Baseline model checkpoint download commands"
  className="post-embed post-embed--baseline"
  loading="lazy"
></iframe>

As `qwen3-32b-g7` runs on two GPUs: `--gpus '"device=0,1"'`. On the 8× RTX PRO 4500 the BF16 baseline needs four 32 GB GPUs: add `--gpus-per-replica 4` to the baseline bench. The Gemma baseline image is `systalyze/vllm-openai:v0.27.1-tf5.14.1` (stock vLLM 0.27.1, `transformers==5.14.1`), built from `bench/Dockerfile` in the benchmarks repository.

The exact workload distribution, saturation sweep, and per-figure vLLM versions are listed below. Our Systalyze platform automatically finds the right set of optimizations and configurations for each workload and hardware combination.

<iframe
  src="/embeds/systalyze-10x/frontier.html"
  title="Interactive measured serving frontier across models and hardware"
  className="post-embed post-embed--frontier"
  loading="lazy"
></iframe>

## Speedup becomes token-cost savings

The same frontier changes cost per output token. Using the evaluation’s on-demand prices, the best observed cost point on the 8× RTX PRO 4500 cuts dollars per million output tokens by 69–73% across the three models.

| Model     | 8× RTX PRO 4500       | 8× RTX PRO 6000      | 8× H200              |
| --------- | --------------------- | -------------------- | -------------------- |
| Gemma-4   | $5.41 → $1.45 (−73%)  | $1.36 → $0.56 (−59%) | $1.11 → $0.94 (−16%) |
| Nemotron  | $4.10 → $1.28 (−69%)  | $0.98 → $0.54 (−44%) | $0.86 → $0.60 (−29%) |
| Qwen3-32B | $25.23 → $7.68 (−70%) | $8.44 → $2.79 (−67%) | $6.26 → $4.12 (−34%) |

Best observed cost point, baseline → optimized, in dollars per million output tokens. Cost uses the on-demand instance prices from this evaluation; exact hourly rates are listed in the appendix.

## The real optimization is composition

The gains above come from three families of optimizations: quantization, optimized kernels, and speculative decoding. It is tempting to think of them as independent multipliers, but the core argument of this post lives in one inequality:

> **Total speedup ≠ quantization speedup × kernel speedup × speculative-decoding speedup.**

Combined performance emerges from the interaction between optimizations. Each one changes the environment the next one runs in:

- **Quantization** uses lower-precision data to reduce memory traffic and speed up matrix multiplications (GEMMs). As GEMM operations get faster, other kernels, GPU communication, and scheduling become more significant bottlenecks.
- **Kernel fusion** combines multiple operations into a single GPU kernel, reducing launch overhead and repeated reads and writes to GPU memory. This addresses overheads that become more prominent after quantization.
- **Speculative decoding** changes the compute balance again, replacing many cheap-per-token decode steps with fewer expensive verification steps plus draft overhead.
- **Concurrency** decides whether that trade is worth making at all: extra speculative work that pays off with one request in flight can be pure waste when the batch is already large enough to saturate the GPU.

Gemma’s 4.0× on the 8× RTX PRO 4500 comes from the composition of NVFP4, the text-only model class, the tuned MoE tile ladder, the router GEMM fix, and DSpark on one specific box. The same model improves by 1.3× over its 8× H200 baseline. The configurations themselves also differ, including the MoE backend, which selects Triton on the RTX boxes and CUTLASS on H200. The winning composition changes with the hardware.

Each layer changes the environment the next one runs in:

1. **Quantization** shifts kernel and communication weight.
2. **Kernel fusion** shifts the draft/verify balance.
3. **Speculative decoding** shifts the compute balance.
4. **Concurrency** gates whether speculation pays.

## Appendix

### DSpark training details

We trained two DSpark variants. Our first [DSpark speculator](https://huggingface.co/systalyze/gemma4-26b-a4b-it-dspark) follows the upstream DSpark recipe, using roughly 1.3M Open-PerfectBlend examples over about 10 epochs [1]. Open-PerfectBlend is the public instruction mixture used by the reference recipe [3]. The second is our [multidomain DSpark](https://huggingface.co/systalyze/gemma4-26b-a4b-it-dspark-multidomain), trained on a separately curated dataset of 743,408 unique, long-context examples.

### Benchmark methodology

Every serving-performance number in this post comes from an end-to-end serving benchmark. The fixed workload is a synthetic coding-request stream with input length p50 3.5K / p90 10K tokens and output length p50 200 / p90 400 tokens, Poisson arrivals, temperature 0, and prefix caching disabled. We sweep load from 30% to 100% of measured saturation and report percentile latencies and per-GPU throughput at each point; every marker in the figures comes from one of those measured curves.

### On-demand pricing used for cost

All per-million-token figures use AWS EC2 On-Demand prices for US East (N. Virginia), `us-east-1`, as of September 2026:

- 8× RTX PRO 4500 (`g7.48xlarge`): $28.51/hour.
- 8× RTX PRO 6000 (`g7e.24xlarge`): $16.57/hour.
- 8× H200 (`p5en.48xlarge`): $63.296/hour.

We compute output-token cost as `hourly instance price / (output tokens/s × 3600) × 1,000,000`. Each cost figure uses that configuration’s saturation-throughput operating point, a higher-load point than the p99-ITL frontier in Figure 1. Source: [AWS EC2 On-Demand Pricing](https://aws.amazon.com/ec2/pricing/on-demand/). These are evaluation-time on-demand rates, so current cloud pricing may differ.

### Reproduction versions

For reproduction, match the vLLM version used by each figure. The serving-throughput evaluation runs on `v0.27.1-tf5.14.1`, with one Nemotron configuration on `v0.28.0`, while the multidomain DSpark results were measured on vLLM 0.28. Each figure therefore has its own versioned reproduction path. [6]

## References

1. Xin Cheng et al. “[DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation](https://arxiv.org/abs/2607.05147).” arXiv:2607.05147, 2026.
2. DeepSeek AI. [DeepSpec](https://github.com/deepseek-ai/DeepSpec), the full-stack training and evaluation codebase used for DSpark and related speculative-decoding methods.
3. Maxime Labonne. [Open-PerfectBlend](https://huggingface.co/datasets/mlabonne/open-perfectblend), the public instruction mixture used by the reference DSpark training recipe.
4. Systalyze. [Gemma-4-26B-A4B-IT ExpertsINT4/FP8 model card](https://huggingface.co/systalyze/gemma-4-26B-A4B-it-ExpertsINT4-FP8), including the BF16/FP8/INT4 accuracy evaluation and RULER context tests.
5. Systalyze. [Gemma-4 DSpark multidomain model card](https://huggingface.co/systalyze/gemma4-26b-a4b-it-dspark-multidomain), including base-versus-draft throughput, domain, concurrency, acceptance-length, and context-length evaluations.
6. vLLM Project. [vLLM](https://github.com/vllm-project/vllm), the serving engine used for the end-to-end evaluations in this post.
7. Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. “[GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://arxiv.org/abs/2210.17323).” arXiv:2210.17323, 2022.
8. Ji Lin et al. “[AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration](https://proceedings.mlsys.org/paper_files/paper/2024/hash/42a452cbafa9dd64e9ba4aa95cc1ef21-Abstract-Conference.html).” MLSys 2024.
