Skip to content

Benchmarking Guide

This guide describes a rigorous, reproducible method for measuring TensorNativeDriver (TND) acceleration on your model and GPU: serve the baseline engine, benchmark it, then serve the TND-accelerated engine and benchmark it again under identical load.

Because greedy output on the accelerated paths is token-identical to the baseline, the comparison measures speed and efficiency, not quality.

1. Method overview

Step Action
1 Install the driver (see Installation & Configuration).
2 Serve the baseline variant on the GPU; run the load; record results.
3 Stop it; serve the accelerated variant on the same GPU; run the identical load.
4 Confirm the accelerator actually engaged (§4) before trusting the numbers.
5 Compare throughput, p99 latency, and TTFT.

Hardware Invariant

Run variants one at a time on the same GPU so the A/B shares identical hardware. Redirect each server's log to a per-variant file to support the engagement check.

2. Serve baseline and accelerated

vLLM

# Baseline
vllm serve <model> --no-async-scheduling > vllm-baseline.log 2>&1

# Accelerated (TND scheduler)
TND_VLLM_SCHEDULER=1 vllm serve <model> --no-async-scheduling > vllm-accel.log 2>&1

# Baseline
vllm serve <model> --async-scheduling --no-enable-prefix-caching > vllm-baseline-async.log 2>&1

# Accelerated (TND scheduler)
TND_VLLM_SCHEDULER=1 TND_VLLM_ASYNC=1 vllm serve <model> \
    --async-scheduling --no-enable-prefix-caching > vllm-accel-async.log 2>&1

A100 Parity Run pending

Async benchmarks are experimental and unvalidated. The default A/B is sync-only.

TensorRT-LLM

# Baseline
PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --baseline -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000 > trt-baseline.log 2>&1

# Accelerated (fused-greedy sampler)
TND_FUSED_SAMPLER=1 PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --accel -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000 > trt-accel.log 2>&1

Native TND

python3.10 -m tnd.serve --model <model> --port 8000 --cascade > tnd.log 2>&1

3. Drive the load

Each server exposes a standard OpenAI-compatible API at http://localhost:8000/v1, so point your existing load generator at it. Common tools work unchanged, for example vllm bench serve --backend openai-chat, llmperf, or k6/oha with a chat body.

A minimal concurrency check needs only curl — fire N in-flight chat completions and time the batch:

# minimal concurrency check
curl http://localhost:8000/v1/chat/complet completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Explain quantum computing."}], "temperature": 0}'

Run the identical load against each variant and compare throughput, p99 latency, and time-to-first-token (TTFT).

4. Confirm the accelerator engaged

A flat A/B is only meaningful if the accelerated path actually ran. A silently-inert wedge records stock performance under the accelerated label. Verify against the per-variant log captured in §2:

  • vLLM (accel): the startup log must name the TND scheduler for the run. Absent → the run was stock; re-check TND_VLLM_SCHEDULER=1 and workload scope.
  • TensorRT-LLM (accel): the log must confirm that TND_FUSED_SAMPLER is armed (fused sampler LIVE in the log). Absent → set TND_FUSED_SAMPLER=1 (and TND_TRT_ACCEL_STRICT=1 to make a mismatch fatal rather than silent).
  • Native TND: check that Cascade shared-prefix decode: ON appears in the server log.

tnd-status also reports each path's wiring state ([ ACTIVE ] / [ OPT-IN ] / [ MISSING ]).

5. Interpreting results

  • vLLM / TensorRT-LLM sampler paths. The wedge removes kernel launches and scheduling overhead per step; the win shows up in throughput and tail latency, and scales with concurrency. Greedy output stays token-identical.
  • Native TND (cascade decode). The cascade decoder shares prefix KV-cache blocks across concurrent queries. The win is massive on long system prompts or multi-turn chats; on short, unrelated inputs, performance aligns with the baseline (or slightly below due to allocation checks).