Skip to content

Engine Integration Reference

TensorNativeDriver (TND) accelerates four serving paths. You install and run the engine exactly as you do today; TND detects it and wires its acceleration behind it.

Confirm what TND detected on your box at any time:

tnd-status

Hugging Face Transformers

Automatic — no code change, no flag.

Once the driver is installed, existing Transformers code routes supported decoder models through TND's engine:

from transformers import AutoModelForCausalLM
# TND's decoder hook auto-activates on import — no changes needed
model = AutoModelForCausalLM.from_pretrained("<your-model>")
model.generate(...)
  • Supported decoder models run on the TND path — nothing to change.
  • Unsupported models fall back to stock — your code still runs, just unaccelerated.

Disable for a single process:

TRANSFORMERS_TND_DISABLE=1 python your_script.py

vLLM

vLLM has two independent acceleration paths.

Embedding / BERT-family — automatic

When you serve an embedding or BERT-family model, TND's fused BERT-FFN kernels load automatically via vLLM's plugin system. Nothing to enable:

vllm serve <embedding-model>

Text generation — opt-in (TND scheduler)

Routing generation through TND's GIL-free scheduler changes scheduling behavior, so it is never forced on. Enable it per process.

TND_VLLM_SCHEDULER=1 vllm serve <model> --no-async-scheduling

TND_VLLM_SCHEDULER=1 routes vLLM onto the TND scheduler and forces async scheduling off for parity with the validated sync scope. Confirm the vLLM startup log names the TND scheduler for the run.

TND_VLLM_SCHEDULER=1 TND_VLLM_ASYNC=1 vllm serve <model> \
    --async-scheduling --no-enable-prefix-caching

Requires A100 parity validation

Set TND_VLLM_ASYNC=1 in addition to TND_VLLM_SCHEDULER=1 to preserve your --async-scheduling choice instead of forcing it off. This activates set_overlap_mode(True) in the C++ engine core so the scheduler computes ahead during GPU execution.

The async path compiles and passes unit tests, but no A100 parity run has been completed — end-to-end correctness and 7B performance are unmeasured. Do not use it in production until parity is confirmed.

Scope. Generate-only, single full-attention KV group, prefix caching off. Leave it off for LoRA, speculative decoding, structured output, multimodal, or beam workloads — vLLM then runs stock (the BERT plugin stays active).

Why vLLM runs in its own environment

vLLM (>= 0.23.0) ships a different torch/CUDA ABI than the core package's cu128 build. Place vLLM on its own box or venv; TND accelerates it there via the scheduler wedge and the BERT plugin. See Installation §1 for isolation guidance.

TensorRT-LLM

Opt-in — TND's fused-greedy sampler, token-identical to stock greedy.

Isolated environment required

TensorRT-LLM is not run from the system Python environment. It must be installed in an isolated venv via sudo tnd-trtllm-setup --venv /opt/tnd/venv (shipped by tensor-native-driver-trtllm-accel_*.deb) and served with the virtual environment's bin on PATH. See Installation Pathway B.

To launch the accelerated server:

TND_FUSED_SAMPLER=1 PATH="/opt/tnd/venv/bin:$PATH" \
    tnd-trtllm-launch --accel -- serve <model> --host 0.0.0.0 --port 8000
  • TND_FUSED_SAMPLER=1 arms the fused-greedy sampler under your installed TensorRT-LLM.
  • tnd-trtllm-launch --accel wraps trtllm-serve with the correct op-library path.
  • Output is token-identical to stock greedy decoding.

Baseline. To serve stock TensorRT-LLM for comparison:

PATH="/opt/tnd/venv/bin:$PATH" \
    tnd-trtllm-launch --baseline -- serve <model> --host 0.0.0.0 --port 8000

Make mismatches fatal

Use TND_TRT_ACCEL_STRICT=1 to make an op-library ABI mismatch fail loudly instead of silently falling back to stock.

Native TND engine

When no external engine is present — or you prefer to serve directly on TND — use TND's own OpenAI-compatible server:

python3.10 -m tnd.serve --model <model> --port 8000 --cascade

The --cascade shared-prefix decode path delivers a large p99-latency and TTFT win on long shared-prefix workloads (RAG, system prompts, agentic). It self-gates to a no-op below the 768-token shared-prefix floor and requires prefix caching (on by default). See Installation §7 for the full option table.

Engine detection matrix

Engine Detected when Baseline serve Accelerated serve
tnd (native) No vLLM and no TensorRT-LLM present (native only) python3.10 -m tnd.serve --model <M> --port 8000 [--cascade]
Transformers transformers importable (stock .generate) Automatic on import
vLLM (BERT / embedding) vllm present vllm serve <M> Automatic (plugin)
vLLM (generation) vllm present vllm serve <M> --no-async-scheduling TND_VLLM_SCHEDULER=1 vllm serve <M> --no-async-scheduling
TensorRT-LLM tensorrt_llm present PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --baseline -- serve <M> PATH="/opt/tnd/venv/bin:$PATH" TND_FUSED_SAMPLER=1 tnd-trtllm-launch --accel -- serve <M>

Which path am I on?

tnd-status shows every path as [ ACTIVE ], [ OPT-IN ], or [ MISSING ], plus the exact serve command for each detected engine:

tnd-status          # human-readable dashboard
tnd-status --json   # machine-readable for automation