Case Studies

GLM-5.3 on Two DGX Stations: 607.8 tok/s

NVIDIA deployed GLM-5.3 across two DGX Station systems with vLLM, reporting 607.8 tok/s throughput. The setup shows how multi-node inference expands capacity while leaving important production benchmarks unanswered.

Font size

Two NVIDIA DGX Station systems running vLLM delivered 607.8 tok/s for GLM-5.3 inference. This deployment shows how a multi-node architecture can increase serving capacity and identifies the benchmarks still needed before production use.

The Result: Two DGX Systems Deliver 607.8 tok/s

The test configuration was straightforward:

  • Model: GLM-5.3
  • Hardware: Two NVIDIA DGX Station systems
  • Inference framework: vLLM
  • Result: 607.8 tok/s

NVIDIA's technical article describes this setup as a response to cases where a single DGX Station cannot meet the target throughput. Both systems jointly serve one GLM-5.3 inference service rather than processing entirely separate request pools.

This is better understood as a multi-node inference deployment than another GPU benchmark. It addresses a specific engineering question: when an existing workstation cannot handle the target throughput, what options are available besides purchasing a larger single system?

Original technical article: https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/

Multi-Node Inference Scales Throughput, Not Every Latency Metric

LLM inference is often reduced to one number: how many tokens are generated per second.

But tok/s raises at least three questions:

  1. Does it represent the output speed of one request or the combined throughput of multiple requests?
  2. How long are the input and output for each request?
  3. How many requests were running concurrently during the test?

Without that context, 607.8 tok/s cannot be translated into a meaningful measure of perceived speed for each user.

Multi-node inference expansion typically relies on model parallelism. Model weights are divided across GPUs, each device computes some of the layers, and forward passes exchange intermediate states over high-speed interconnects. This approach uses more memory and compute without requiring every node to load a complete copy of the model.

The key is not running two independent service instances. The two DGX Station systems form a single inference cluster. vLLM handles request scheduling, continuous batching, and parallel execution, while NVIDIA's infrastructure provides communication between nodes.

Model parallelism also has clear costs:

  • Inter-node latency enters every inference path.
  • Load must remain balanced across both machines.
  • A failure on either node could affect the entire service.
  • Deployment, troubleshooting, and scaling are considerably more complex than with a single process.

This setup primarily expands throughput rather than improving every latency metric. It may increase the number of requests the system can serve at once, but it does not guarantee a faster time to first token.

Why vLLM Is a Good Fit

vLLM is not designed simply to make one model run faster. It is a high-throughput inference serving framework that addresses how dynamic requests share limited GPU resources.

Its capabilities have three main layers.

The first is PagedAttention, which manages the KV cache with paging to reduce memory waste. For long contexts and high-concurrency serving, this can be more effective than adding compute alone.

The second is continuous batching. Generative requests take different amounts of time to complete. When a new request arrives, the scheduler can adjust the batch instead of waiting for every request in the current batch to finish. This offers little advantage for offline workloads with fixed prompts, but it is essential for continuously changing production traffic.

The third is distributed execution. vLLM can divide a model across multiple machines and manage communication between nodes, extending model parallelism beyond a single multi-GPU host.

Together, these capabilities form the deployment path used in this case:

GLM-5.3
   ↓
vLLM distributed serving
   ↓
NVIDIA DGX Station #1 ←→ NVIDIA DGX Station #2

Peak hardware performance is only an upper bound. Weight partitioning, request queuing, KV cache reuse, and inter-node communication determine how much effective throughput the system ultimately delivers.

607.8 tok/s Does Not Establish Production Readiness

The 607.8 tok/s result is impressive, but it cannot serve as a standalone production acceptance metric.

At minimum, I would benchmark the following areas:

Dimension Measurements needed
Input and output lengths Separate tests for short questions and answers, long prompts, and code generation
Concurrency Single requests, typical production peaks, and stress-test peaks
Latency TTFT, TPOT, and end-to-end P50/P95/P99
Resources Memory usage and GPU utilization per node, along with power consumption
Stability Throughput and error rate after several hours of continuous operation
Failure behavior Single-node shutdown, restart, and recovery time

TTFT is particularly important. The time between a user submitting a request and seeing the first token often affects the interactive experience more than the speed of subsequent tokens. Batch offline workloads may tolerate high TTFT, but online Agent calls may not.

The distribution of request lengths matters as well. A test consisting entirely of short inputs and long outputs can produce attractive throughput numbers. With many long-context requests, KV cache usage and prefill time may change the results.

A more reliable approach is to replay real production traffic. Without production traffic, the evaluation should at least fix the input and output token lengths, concurrency, request arrival rate, and model parameters before comparing single-system and dual-system results. Otherwise, 607.8 tok/s is merely a snapshot from a test run.

What This Means for Deploying Chinese-Origin Models

For teams already using NVIDIA infrastructure, the value of this test is that the inference stack can operate as an independent layer rather than being tied to one model.

When GLM-5.3 no longer meets cost, quality, or concurrency requirements, teams can test another model without rebuilding the entire hardware platform. The target model must support vLLM and meet its requirements for distributed parallelism, quantization, and operator coverage.

This approach separates two decisions:

  • Model layer: Compare the output quality, price, and context capabilities of GLM-5.3 with alternatives such as Claude, Qwen, and DeepSeek.
  • System layer: Compare inference stacks such as vLLM and TensorRT-LLM, as well as the operating costs of single-node, multi-node, and different parallelism strategies.

Multi-node deployment also increases operational complexity. Teams must monitor node health, network interfaces, and interconnect status; synchronize software versions; and prepare degradation and recovery procedures for node failures. The higher the inter-node throughput, the greater the potential impact of a single-node failure.

For that reason, 607.8 tok/s is best treated as validation of a deployment path: existing NVIDIA infrastructure can host open-source models, while the model and inference stack can be replaced independently. Whether the approach merits production deployment requires complete benchmarks and failure exercises.

Engineering Takeaways

The most interesting aspect of this case is not a new throughput record. It reframes “one GPU is not enough” as a systems problem rather than solely a purchasing problem. Model parallelism, continuous batching, and cross-node scheduling can change how existing hardware is utilized. That is especially relevant to deployments of Chinese-origin models, where model quality may require frequent iteration but the underlying inference system does not need to be rewritten each time.

Teams evaluating models will increasingly use a two-part cost model: the price per token plus the cost of the effective throughput required to meet the target SLA. The latter will directly influence GPU selection, parallelism strategy, and vendor choice.

Unlike a conventional GPU benchmark, this test evaluates the complete service path rather than an individual GPU. However, the available summary does not include input and output lengths, concurrency, TTFT, or the full configuration. As a result, the production cost-effectiveness of 607.8 tok/s cannot yet be determined. The complete test data is worth waiting for before declaring that two machines are the answer.

Source

https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/

Source: https://developer.nvidia.com/blog/when-one-gpu-isnt-enough-serving-glm-5-3-across-two-nvidia-dgx-stations-with-vllm/