Skip to content

Engine Integration Reference

TensorNativeDriver (TND) accelerates four serving paths. You install and run the engine exactly as you do today; TND detects it and wires its acceleration behind it.

Confirm what TND detected on your box at any time:

tnd-status

Hugging Face Transformers

Automatic — no code change, no flag.

Once the driver is installed, existing Transformers code routes supported decoder models through TND's engine:

from transformers import AutoModelForCausalLM
# TND's decoder hook auto-activates on import — no changes needed
model = AutoModelForCausalLM.from_pretrained("<your-model>")
model.generate(...)
  • Supported decoder models run on the TND path — nothing to change.
  • Unsupported models fall back to stock — your code still runs, just unaccelerated.

Disable for a single process:

TRANSFORMERS_TND_DISABLE=1 python your_script.py

vLLM

Default-ON once vLLM shares the driver's system Python — TND's GIL-free scheduler. vLLM (>= 0.23.0) installs into the same system python3 as the core driver — no separate venv, no separate wheel. The driver patches vLLM's scheduler on import automatically; there is no flag to pass.

Install vLLM before the driver, not after

Install vLLM first (sudo python3 -m pip install vllm==<version>), then the core .deb. The driver's post-install detects whichever torch build is already present (vLLM's, if it's already installed) and activates the matching pre-built libtnd_paged_cuda.so variant. That detection runs once, at install time — installing the driver first and adding vLLM afterward leaves the activated variant orphaned. vLLM's own scheduler is unaffected either way (it's pure Python and never touches that library), but native tnd.serve and Transformers acceleration on the same box can silently lose their CUDA-graph/fused-sampler acceleration if the order is reversed — no crash, no error, just slower. See Installation Pathway A.

vllm serve <model> --no-async-scheduling

No --scheduler-cls flag needed. Confirm engagement in the startup log:

Using custom scheduler class tnd.vllm_adapter.TndScheduler

Opt out for a single process: TND_VLLM_SCHEDULER=0 vllm serve <model> (also accepts false/no/off).

vllm serve <model> --async-scheduling --no-enable-prefix-caching

Passing --async-scheduling directly is enough — the adapter inherits your choice and activates set_overlap_mode(True) in the C++ engine core, so the scheduler computes the next step ahead while the current step runs on the GPU.

Offline parity only — no live server-workload A/B yet

Async scheduling with the TND scheduler has passed offline token-exact parity (Qwen2.5-7B/A100, natural EOS, 2026-07-16), but a live server-workload A/B is still pending. A/B it on your own hardware before relying on it.

Benchmark on your own hardware

Async throughput gains are workload- and model-scale-dependent. A/B it against the sync path on your model and GPU (see the Benchmarking Guide) before rolling it out.

Scope. Generate-only, single full-attention KV group, prefix caching off.

No automatic fallback for out-of-scope workloads

Unlike Transformers' automatic path, the scheduler does not detect or reject LoRA, speculative decoding, structured output, multimodal, or beam requests — they are accepted and the unsupported behavior is silently a no-op, which is not the same as falling back to stock vLLM. "Default ON" only means nothing to enable — it does not mean safe for every workload. If your workload isn't confirmed in scope, set TND_VLLM_SCHEDULER=0 for that process.

Why install order matters

vLLM (>= 0.23.0) may pull a different torch/CUDA build than the core driver's default cu128 pin. Because vLLM shares the driver's system python3 rather than an isolated venv, the driver's post-install is what reconciles the two — it detects whichever torch is present and activates a matching pre-built variant, but only once, at .deb install time. See Installation §1 for the full ordering guidance.

TensorRT-LLM

Opt-in — TND's fused-greedy sampler, token-identical to stock greedy.

Isolated environment required

TensorRT-LLM is not run from the system Python environment. It must be installed in an isolated venv via sudo tnd-trtllm-setup --venv /opt/tnd/venv (shipped by tensor-native-driver-trtllm-accel_*.deb) and served with the virtual environment's bin on PATH. See Installation Pathway B.

To launch the accelerated server:

TND_FUSED_SAMPLER=1 PATH="/opt/tnd/venv/bin:$PATH" \
    tnd-trtllm-launch --accel -- serve <model> --host 0.0.0.0 --port 8000
  • TND_FUSED_SAMPLER=1 arms the fused-greedy sampler under your installed TensorRT-LLM.
  • tnd-trtllm-launch --accel wraps trtllm-serve with the correct op-library path.
  • Output is token-identical to stock greedy decoding.

Baseline. To serve stock TensorRT-LLM for comparison:

PATH="/opt/tnd/venv/bin:$PATH" \
    tnd-trtllm-launch --baseline -- serve <model> --host 0.0.0.0 --port 8000

ABI mismatches are always fatal, not a silent fallback

tnd-trtllm-launch resolves the ABI-matched libtnd_paged_cuda.so for the venv's torch build automatically and exits with a FATAL error (no serve attempt) if no matching variant is found or the resolved library fails to load — there is no silent stock fallback to guard against, and no flag is needed to make this behavior loud.

Native TND engine

When no external engine is present — or you prefer to serve directly on TND — use TND's own OpenAI-compatible server:

python3.10 -m tnd.serve --model <model> --host 0.0.0.0 --port 8000

See Installation §7 for the full option table.

Engine detection matrix

Engine Detected when Baseline serve Accelerated serve
tnd (native) No vLLM and no TensorRT-LLM present (native only) python3.10 -m tnd.serve --model <M> --host 0.0.0.0 --port 8000
Transformers transformers importable (stock .generate) Automatic on import
vLLM (generation) vllm present in the same system python3 as the driver TND_VLLM_SCHEDULER=0 vllm serve <M> --no-async-scheduling vllm serve <M> --no-async-scheduling (default — no flag needed)
TensorRT-LLM tensorrt_llm present PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --baseline -- serve <M> PATH="/opt/tnd/venv/bin:$PATH" TND_FUSED_SAMPLER=1 tnd-trtllm-launch --accel -- serve <M>

Which path am I on?

tnd-status shows every path as [ ACTIVE ], [ OPT-IN ], or [ MISSING ], plus the exact serve command for each detected engine:

tnd-status          # human-readable dashboard
tnd-status --json   # machine-readable for automation