Skip to content

Benchmarking Guide

This guide describes a rigorous, reproducible method for measuring TensorNativeDriver (TND) acceleration on your model and GPU: serve the baseline engine, benchmark it, then serve the TND-accelerated engine and benchmark it again under identical load.

Because greedy output on the accelerated paths is token-identical to the baseline, the comparison measures speed and efficiency, not quality.

1. Method overview

Step Action
1 Install the driver (see Installation & Configuration).
2 Serve the baseline variant on the GPU; run the load; record results.
3 Stop it; serve the accelerated variant on the same GPU; run the identical load.
4 Confirm the accelerator actually engaged (§4) before trusting the numbers.
5 Compare throughput, p99 latency, and TTFT.

Hardware Invariant

Run variants one at a time on the same GPU so the A/B shares identical hardware. Redirect each server's log to a per-variant file to support the engagement check.

2. Serve baseline and accelerated

vLLM

TND's scheduler is default-ON once vLLM shares the driver's system python3 (see Installation Pathway A) — the baseline run below has to explicitly opt out, or it will be accelerated too:

# Baseline (stock vLLM -- explicitly disable the default-ON scheduler)
TND_VLLM_SCHEDULER=0 vllm serve <model> --no-async-scheduling > vllm-baseline.log 2>&1

# Accelerated (TND scheduler -- default ON, no flag needed)
vllm serve <model> --no-async-scheduling > vllm-accel.log 2>&1

Offline parity only — no live server-workload A/B yet

Async scheduling with the TND scheduler has passed offline token-exact parity (Qwen2.5-7B/A100, natural EOS, 2026-07-16), but a live server-workload A/B is still pending. A/B it on your own hardware before relying on it.

# Baseline (stock vLLM, async on -- explicitly disable the default-ON scheduler)
TND_VLLM_SCHEDULER=0 vllm serve <model> --async-scheduling --no-enable-prefix-caching > vllm-baseline-async.log 2>&1

# Accelerated (TND scheduler, async on -- default ON, no flag needed)
vllm serve <model> --async-scheduling --no-enable-prefix-caching > vllm-accel-async.log 2>&1

Harness variants

The 05-inference-accel benchmark rig exposes vllm_async / vllm_accel_async engine labels for a controlled async A/B on one GPU.

TensorRT-LLM

# Baseline
PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --baseline -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000 > trt-baseline.log 2>&1

# Accelerated (fused-greedy sampler)
TND_FUSED_SAMPLER=1 PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --accel -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000 > trt-accel.log 2>&1

Native TND

python3.10 -m tnd.serve --model <model> --host 0.0.0.0 --port 8000 > tnd.log 2>&1

3. Drive the load

Each server exposes a standard OpenAI-compatible API at http://localhost:8000/v1, so point your existing load generator at it. Common tools work unchanged, for example vllm bench serve --backend openai-chat, llmperf, or k6/oha with a chat body.

A minimal concurrency check needs only curl — fire N in-flight chat completions and time the batch:

# minimal concurrency check
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen2.5-1.5B-Instruct", "messages": [{"role": "user", "content": "Explain quantum computing."}], "temperature": 0}'

Run the identical load against each variant and compare throughput, p99 latency, and time-to-first-token (TTFT).

4. Confirm the accelerator engaged

A flat A/B is only meaningful if the accelerated path actually ran. A silently-inert wedge records stock performance under the accelerated label. Verify against the per-variant log captured in §2:

  • vLLM (accel): the startup log must name the TND scheduler for the run. Absent → confirm TND_VLLM_SCHEDULER=0 isn't set, that vLLM shares the driver's system python3, and workload scope.
  • TensorRT-LLM (accel): the launcher log must show armed: int8-gemm,fused-sampler, followed by fused_greedy_sample LIVE on first inference. Absent → set TND_FUSED_SAMPLER=1; an ABI mismatch is already fatal (tnd-trtllm-launch exits with a FATAL error, no silent fallback). tnd-status also reports each path's wiring state ([ ACTIVE ] / [ OPT-IN ] / [ MISSING ]).

5. Interpreting results

  • vLLM / TensorRT-LLM sampler paths. The wedge removes kernel launches and scheduling overhead per step; the win shows up in throughput and tail latency, and scales with concurrency. Greedy output stays token-identical.