Up to 10× faster LLM serving
Systalyze found and applied the right combination of serving optimizations across Gemma 4, Nemotron and Qwen, cutting cost per output token by up to 73%.
Systalyze grew out of years of work at MIT CSAIL on making machine learning systems efficient. We keep publishing, and we turn what we learn into the platform.
Systalyze found and applied the right combination of serving optimizations across Gemma 4, Nemotron and Qwen, cutting cost per output token by up to 73%.
The standard utilization metric only says whether a GPU is busy. We explain what accurate measurement looks like and release Utilyze, a free tool that shows it live.
Checkpoints every training iteration with no slowdown, cutting repeated work after a failure by 80–97%.
Checkmate enables per-iteration checkpointing in DNN training without slowdown. Its key insight is that in data-parallel training, all information necessary to create a checkpoint already exists in the network as gradients. A multicast abstraction simultaneously delivers gradients to a separate CPU-based shadow cluster, which maintains a checkpoint by applying them to a copy of the model. Checkmate performs per-iteration checkpointing with throughput comparable to an ideal no-checkpoint baseline. It achieves 5 to 34.5× more frequent checkpointing than state-of-the-art systems, reducing repeated work per failure by 80% to 97.1%, and delivers 1.3× to 6.5× throughput at the same checkpointing frequency.
Interleaves the communication of jobs sharing a link, improving tail completion time by up to 2.5×.
CASSINI is a network-aware job scheduler for machine learning clusters. Its geometric abstraction and affinity graph find time-shift values that interleave the communication patterns of jobs sharing a network link. Experiments with 13 common ML models on a 24-server testbed show that CASSINI improves average and tail completion time by up to 1.6× and 2.5×, respectively, compared to state-of-the-art ML schedulers. It also reduces ECN-marked packets by up to 33×.
Matches conventional training performance while cutting network cost by 38–77%.
This paper presents a low-cost network architecture for training large language models at hyperscale. Because LLM training generates sparse communication patterns, it does not require an any-to-any full-bisection network. The rail-only design eliminates the spine layer in traditional GPU clusters and maintains training performance while reducing network cost by 38% to 77% and power consumption by 37% to 75%. It also supports Mixture-of-Expert models through forwarding, with only 8.2% to 11.2% completion-time overhead. The paper studies failure robustness and the impact of network and training parameters.
A few dozen lines added to standard congestion control speed up training iterations by up to 2×.
MLTCP augments congestion-control algorithms to accelerate DNN training jobs in shared GPU clusters by interleaving the communication phases of jobs competing for bandwidth. Its central principle is that DNN training flows should scale their congestion window based on bytes sent in each training iteration. Adding 30–60 lines of code to Reno, CUBIC, or DCQCN stabilizes flows into an interleaved state within a few iterations, regardless of flow counts or start times. MLTCP accelerates average and 99th-percentile training iteration time by up to 2× and 4×, respectively.
Queueing-theoretic scheduling that balances compute and network to cut inference completion time.
Nona combines a stochastic scheduler with an offline optimization formulation rooted in queueing theory to minimize the average completion time of heterogeneous inference queries. It incorporates compute and network tradeoffs, uses the Pollaczek–Khinchine formula to estimate queueing latency, and builds on Jackson networks to capture dependencies between computation and communication operations. A graph-contraction procedure lets cloud providers solve the optimization in practical settings. Evaluation with models including AlexNet, ResNet, DenseNet, VGG, and GPT-2 shows that Nona outperforms state-of-the-art schedulers by up to 350×.
Shows that deliberately unfair sharing can make every competing training job faster.
This paper argues that fair sharing is not necessarily desirable in machine learning training clusters. For compatible combinations of jobs, deliberately introducing unfairness improves training time for every competing job. A geometric abstraction rolls time around a circle and rotates communication phases to identify fully compatible jobs, demonstrating up to 1.3× improvement in average training iteration time. The paper proposes three directions for resource management: adaptively unfair congestion control, priority queues on switches, and precise flow scheduling.