Installation & Configuration¶

This guide takes a host from clean to a verified TensorNativeDriver (TND) install, then documents every configuration surface. TND accelerates the inference engine you already run — no code change is required.

1. System requirements¶

Requirement	Specification
Operating system	Linux x86_64, Ubuntu 22.04 LTS
GPU	NVIDIA GPU with a CUDA 12.8-capable driver (driver `>= 525`)
GPU driver note	A CUDA-13 host driver (`nvidia-driver-580`) is supported — the driver is forward-compatible — but the `torch` build must remain `+cu128` (below)
Python	The default `python3` must be CPython 3.10 or 3.12 (the two ABIs the multi-ABI package ships; PEP 3149 auto-select). Other defaults (3.11, 3.13+, < 3.10) are refused at apt dependency resolution — see the Note on Dependencies below.
Runtime	`torch 2.8.0+cu128`, pre-installed into system `python3` (see §2)
Engine (optional)	The inference engine you intend to accelerate — Transformers and/or TensorRT-LLM — installed into that same system `python3`

Note on Dependencies (supported Python)

The core package's Depends: field encodes a tight contract on the system-default python3:

Depends: python3 (>= 3.10),
         python3 (<< 3.11) | python3 (>= 3.12),
         python3 (<< 3.13)

The package installs only when the box's default python3 is CPython 3.10 or 3.12 (the two ABIs the multi-ABI package ships; PEP 3149 auto-select). On any other default python3 (3.11, 3.13+, or < 3.10), apt install ./tensor-native-driver_*.deb fails at dependency resolution with a clean apt error and never reaches state ii — no half-installed state. This is by design, not a defect.

Remediation: Switch the system-default python3 to 3.10 or 3.12 (e.g. via update-alternatives or your distro's python packages), then re-run apt install. Do NOT use --force-depends: it would install an unusable/mismatched binding.

Install-acceptance matrix

Default python3	`apt install ./tensor-native-driver_*.deb` result
3.10	SUCCESS → state `ii`; `python3 -c "from tnd import PagedAttentionEngine"` imports
3.12	SUCCESS → state `ii`; import works
3.11	FAIL at dependency resolution (clean apt error); NO half-installed state
3.13+	FAIL at dependency resolution (clean apt error); NO half-installed state

Why the system python3

The core package's post-install runs its torch.ops.load_library check against the box's system python3, not a virtualenv or conda env. Installing the engine into a venv leaves it where the post-install cannot see it, and the check can report a false success.

vLLM environment isolation

vLLM (>= 0.23.0) ships a different torch/CUDA ABI than the core package's cu128 build, so the two are not installed into the same Python. Place vLLM on its own box or venv; TND accelerates it there via the scheduler wedge and the BERT plugin (see the Engine Integration Reference).

Installing the core .deb on a box that already runs vLLM or TensorRT-LLM

This presumes the box's default python3 is already CPython 3.10 or 3.12 — without that, apt refuses at dependency resolution before reaching this stage (see the Note on Dependencies above).

Requires Inference bundle >= v4.3.10 (the release cut with the multi-ABI engine-detection fix, Inference PR #278).

From that bundle on, the core package's post-install detects an engine present under either shipped interpreter (python3.10 and python3.12) and downgrades its libtnd_paged_cuda.so load self-check from a fatal error to a warning. The install then stays clean — apt returns 0, dpkg -l shows ii — and the warning points you at the vLLM scheduler / TensorRT-LLM plugin path instead of aborting. A genuine ABI mismatch with no engine present still fails fatally (see §2).

Get the bundle¶

The install bundle is the token-free public GCS "latest" mirror that clients pull from:

gs://tnd-bundle-artifacts/releases/inference-engine/py3.10/tnd-install-linux.zip

Download with either:

gcloud storage cp gs://tnd-bundle-artifacts/releases/inference-engine/py3.10/tnd-install-linux.zip .
# or
gsutil cp gs://tnd-bundle-artifacts/releases/inference-engine/py3.10/tnd-install-linux.zip .

It extracts (single unzip) to tnd-install-linux/ containing three DEBs: * tensor-native-driver_*.deb — CORE driver. Installs into system python3; ABI-locked to torch 2.8.0+cu128. * tensor-native-driver-trtllm-accel_*.deb — TensorRT-LLM acceleration package. Ships tnd-trtllm-setup, tnd-trtllm-launch, and /opt/tnd/trtllm/libtnd_paged_cuda.so. REQUIRED for the TRT-LLM fused-sampler path. * tensor-native-driver-trt-plugin_*.deb — OPTIONAL custom TensorRT plugin layer. Depends on libnvinfer10 / libnvinfer-plugin10, which are NOT installable without NVIDIA's TensorRT apt repo. It is NOT required for fused-sampler acceleration.

2. Prerequisites¶

A clean box has neither pip, unzip, nor OpenMPI. Install these system dependencies before anything else:

sudo apt-get update && sudo apt-get install -y python3-pip unzip libopenmpi-dev openmpi-bin

python3-pip — The host uses python3 -m pip but pip is absent on a clean box, causing No module named pip errors.
unzip — Required to extract the installation bundle; otherwise fails with unzip: command not found.
libopenmpi-dev openmpi-bin — tensorrt_llm dynamically loads libmpi.so at import time. Without it, importing tensorrt_llm raises RuntimeError: cannot load MPI library.

After the system dependencies are installed, install the exact ABI-matched torch runtime in system python3 (required for the core driver's CUDA-core load check):

sudo python3 -m pip install "torch==2.8.0" --index-url https://download.pytorch.org/whl/cu128

3. Choose your pathway¶

Depending on the inference engine(s) you run on the host, choose one of the following deployment pathways:

Pathway	Target Workload	Key Components Installed
Pathway A — Standard	Transformers and/or vLLM	Core driver only. Runs on system Python + torch cu128.
Pathway B — TensorRT-LLM	TensorRT-LLM	Core driver + TRT-LLM acceleration. Isolated cu130 virtual environment for TensorRT-LLM.
Pathway C — Coexistence	Standard + TensorRT-LLM	Both paths on the same host, keeping the environments strictly isolated.

Pathway A — Standard (Transformers and/or vLLM)¶

Use this pathway when running Hugging Face Transformers and/or vLLM. The driver installs directly into your system python3.

# Install system prerequisites and staging runtime
sudo apt-get update && sudo apt-get install -y python3-pip unzip
sudo python3 -m pip install "torch==2.8.0" --index-url https://download.pytorch.org/whl/cu128

# Extract the bundle
unzip tnd-install-linux.zip

# Install the core driver
sudo apt install ./tnd-install-linux/tensor-native-driver_*.deb

# Verify the integration
tnd-status

Note: Transformers acceleration is automatic on import. vLLM generation is opt-in via setting the environment variable TND_VLLM_SCHEDULER=1 (vLLM itself runs in its own environment — see Engine Integration Reference for configuration detail).

Pathway B — TensorRT-LLM (isolated cu130 venv; system python UNTOUCHED)¶

Use this pathway for TensorRT-LLM acceleration. To maintain stability, TRT-LLM and its dependencies are isolated in a virtual environment (/opt/tnd/venv), keeping the system Python clean.

P0 WARNING: System-Python ABI Trap

NEVER pip install tensorrt_llm directly into the system python3. The PyPI wheel for tensorrt_llm will pull torch off 2.8.0+cu128 to a CUDA-13 build. This breaks the core driver's ABI lock and causes a segmentation fault on every subsequent python3 invocation on the box (due to the driver's sitecustomize/autowire bootstrap that runs on interpreter startup). This is silent and catastrophic. TRT-LLM must live in the isolated venv built by tnd-trtllm-setup below; the system python3 stays on cu128.

# 1. Install system prerequisites (including OpenMPI for TRT-LLM)
sudo apt-get update && sudo apt-get install -y python3-pip unzip libopenmpi-dev openmpi-bin

# 2. Stage the core driver runtime on system python (torch cu128)
sudo python3 -m pip install "torch==2.8.0" --index-url https://download.pytorch.org/whl/cu128

# 3. Extract the bundle
unzip tnd-install-linux.zip

# 4. Install the core driver
sudo apt install ./tnd-install-linux/tensor-native-driver_*.deb

# 5. Install the TRT-LLM acceleration package (ships tnd-trtllm-setup & tnd-trtllm-launch)
sudo apt install ./tnd-install-linux/tensor-native-driver-trtllm-accel_*.deb

# 6. Build the isolated virtual environment (installs torch 2.10.0+cu130 and tensorrt_llm 1.3.0rc17)
# This step takes ~10 minutes and downloads several gigabytes
sudo tnd-trtllm-setup --venv /opt/tnd/venv

Serve (trtllm-serve lives in the venv, so put the venv bin on PATH):

# baseline (stock TensorRT-LLM):
PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --baseline -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000

# accelerated (TND fused-greedy sampler; token-identical to stock greedy):
PATH="/opt/tnd/venv/bin:$PATH" TND_FUSED_SAMPLER=1 tnd-trtllm-launch --accel -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000

PATH & Binary Callout

After running tnd-trtllm-setup, the trtllm-serve binary resides inside the virtual environment at /opt/tnd/venv/bin/trtllm-serve and is not on the default system PATH. Because tnd-trtllm-launch executes trtllm-serve from the active environment PATH, you must prepend the virtual environment to your PATH (e.g. PATH="/opt/tnd/venv/bin:$PATH") or explicitly set the environment variable TRTLLM_SERVE_BIN=/opt/tnd/venv/bin/trtllm-serve.

Note: serve <MODEL> is positional under trtllm-serve 1.3.x (the old top-level --model was removed).

Pathway C — Coexistence (core + TRT-LLM on the same box)¶

To run Standard workloads (system python cu128) and TensorRT-LLM workloads on the same host, follow the installation steps for both Pathway A and Pathway B.

Because the system Python and the virtual environment at /opt/tnd/venv never share an interpreter, the ABI locks are preserved, preventing any conflicts or segfaults: * Running tnd-status under the system python3 reports the Transformers and vLLM wiring. * Running tnd-status under /opt/tnd/venv/bin/python reports the TensorRT-LLM path wiring.

4. What gets accelerated¶

Once installed, your existing code runs unchanged. The behavior depends on the engine detected on the box:

Your workload	After install	Action needed
Transformers (supported decoder)	Routed through TND's engine	None
vLLM (embedding / BERT)	Fused BERT-FFN kernels loaded	None
vLLM (generation)	Runs stock unless opted in	Set `TND_VLLM_SCHEDULER=1`
TensorRT-LLM	Runs stock unless opted in	Set `TND_FUSED_SAMPLER=1`

Anything unsupported falls back to the stock framework — your workload never breaks because TND is installed.

5. Verify: `tnd-status`¶

Run the dashboard at any time:

tnd-status

It reports, on one screen: * Detected engines — Transformers / vLLM / TensorRT-LLM found on this box. * Wiring state per path — [ ACTIVE ], [ OPT-IN ], or [ MISSING ]. * The serve command for each detected engine.

For automation and health checks, use the machine-readable form:

tnd-status --json

Wiring values: ok = active, opt = available but not enabled, miss = the component that path needs is not installed. If a path you expect as [ ACTIVE ] shows [ MISSING ], tnd-status names the missing component.

6. Configuration reference¶

6.1 Environment variables¶

Acceleration paths are controlled per process with environment variables.

Variable	Effect
`TND_DISABLE_AUTO_INTEGRATION=1`	Master off switch — disables all auto-integration for the process.
`TRANSFORMERS_TND_DISABLE=1`	Disable only Transformers acceleration (run stock Hugging Face).
`TND_VLLM_SCHEDULER=1`	Opt in to the vLLM TND scheduler (forces `--no-async-scheduling` unless `TND_VLLM_ASYNC=1` is also set).
`TND_VLLM_ASYNC=1`	Allow async scheduling with the TND scheduler. Must also pass `--async-scheduling` to vLLM. Experimental — requires A100 parity validation.
`TND_FUSED_SAMPLER=1`	Opt in to the TensorRT-LLM fused-greedy sampler.
`TND_PAGED_CUDA_LIB=<path>`	Path to the ABI-matched op library for the TensorRT-LLM path (default `/opt/tnd/trtllm/libtnd_paged_cuda.so`).
`TND_TRT_ACCEL_STRICT=1`	Make a TensorRT-LLM arming failure fatal instead of silently falling back to stock.

6.2 Opt-in paths¶

Two paths change engine behavior and are therefore never forced on.

vLLM on TND's scheduler (sync, default — validated)

TND_VLLM_SCHEDULER=1 vllm serve <model> --no-async-scheduling

Routes vLLM onto TND's GIL-free scheduler. Scope: generate-only, single full-attention KV group, prefix caching off. Leave it off for LoRA, speculative decoding, structured output, multimodal, or beam workloads — vLLM then runs stock (the BERT plugin stays active).

vLLM on TND's scheduler (async — experimental, unvalidated)

TND_VLLM_SCHEDULER=1 TND_VLLM_ASYNC=1 vllm serve <model> \
    --async-scheduling --no-enable-prefix-caching

When TND_VLLM_ASYNC=1 is set alongside TND_VLLM_SCHEDULER=1, the adapter preserves your --async-scheduling choice and activates set_overlap_mode(True) in the C++ engine core for schedule-ahead overlap. This path compiles and passes unit tests but no A100 parity run has been completed — do not use in production until parity is confirmed at 7B.

TensorRT-LLM fused sampler

TND_FUSED_SAMPLER=1 trtllm-serve <model>
# or the convenience wrapper:
tnd-trtllm-launch --accel -- serve <model>

Arms TND's fused-greedy sampler under your installed TensorRT-LLM. Output is token-identical to stock greedy. The op library must be ABI-matched to your TensorRT-LLM environment's torch; the shipped /opt/tnd/trtllm/libtnd_paged_cuda.so is built for that. Use TND_TRT_ACCEL_STRICT=1 to make a mismatch fatal instead of a silent fallback.

7. Native TND server¶

TND ships its own OpenAI-compatible server for when no external engine is present.

# Managed service — configure /etc/tnd-server/tnd-server.conf first, then:
sudo systemctl enable --now tnd-server

# Ad hoc:
python3.10 -m tnd.serve --model <model> --host 0.0.0.0 --port 8000

Sanity-check the endpoint and point any OpenAI-compatible client at http://<host>:8000/v1:

curl http://localhost:8000/health
curl http://localhost:8000/v1/models

Serve options¶

Option	Default	Description
`--model`	—	Hugging Face model id.
`--host` / `--port`	`0.0.0.0` / `8000`	Bind address.
`--cascade` / `--no-cascade`	auto	Cascade shared-prefix decode. Large p99-latency and TTFT win on long shared-prefix workloads (RAG, system prompts, agentic). Self-gated to a no-op below the 768-token shared-prefix floor. Requires prefix caching (on by default).

8. Troubleshooting¶

Symptom	Cause	Resolution
`apt install ./tensor-native-driver_*.deb` aborts with an unmet `python3` dependency	default `python3` is not CPython 3.10 or 3.12 (fail-clean by design)	Switch the system-default `python3` to 3.10 or 3.12 (e.g. via `update-alternatives` or your distro's python packages), then re-run `apt install`. Do NOT use `--force-depends` — it would install an unusable/mismatched binding.
Segfault on any `python3` after installing TensorRT-LLM	`pip install tensorrt_llm` pulled system `torch` off `2.8.0+cu128` (CUDA-13 wheel), breaking the core-driver ABI lock	Reinstall `torch==2.8.0 --index-url .../cu128` into system python; keep TRT-LLM in the `/opt/tnd/venv` venv built by `tnd-trtllm-setup` — never in system python
`No module named pip` / `unzip: command not found`	bare Ubuntu 22.04 lacks base tooling	`sudo apt-get install -y python3-pip unzip`
`RuntimeError: cannot load MPI library` on `import tensorrt_llm`	OpenMPI runtime absent (`tensorrt_llm` dlopens `libmpi.so`)	`sudo apt-get install -y libopenmpi-dev openmpi-bin`
`trtllm-serve: command not found` via `tnd-trtllm-launch`	serve binary is in the venv, not on PATH	prepend `PATH="/opt/tnd/venv/bin:$PATH"` or set `TRTLLM_SERVE_BIN=/opt/tnd/venv/bin/trtllm-serve`
A path expected as `[ ACTIVE ]` shows `[ MISSING ]`	The engine, or a TND component it needs, is not installed in this Python	`tnd-status` names the missing component; install it in the same system `python3`, then re-check
Transformers ran on stock Hugging Face	Model unsupported (falls back by design), or `TRANSFORMERS_TND_DISABLE=1` is set	Confirm with `tnd-status`; look for the `TND transformers decoder hook installed` log line
vLLM log does not name the TND scheduler	`TND_VLLM_SCHEDULER=1` not set, or the workload is out of scope (LoRA, spec-decode, multimodal)	Set `TND_VLLM_SCHEDULER=1` and re-serve with `--no-async-scheduling`
TensorRT-LLM sampler did not arm	`TND_FUSED_SAMPLER` unset, or the op library is not ABI-matched to your TensorRT-LLM `torch`	Set `TND_FUSED_SAMPLER=1` and `TND_PAGED_CUDA_LIB`; use `TND_TRT_ACCEL_STRICT=1` to surface the mismatch
`/health` refused right after start	Model still loading	Wait for the model to finish loading, then re-check

9. Support diagnostics¶

If an install does not land the way tnd-status reports, capture the following and include it with any support request:

echo $?; dpkg -l | grep tensor-native-driver
python3 -c 'import torch; print(torch.__version__, torch.version.cuda)'
ldd /usr/lib/x86_64-linux-gnu/libtnd_paged_cuda.so | grep 'not found'
tnd-status --json

Architecture — how auto-integration works.
Engine Integration Reference — per-engine detail and serve commands.
Benchmarking Guide — prove the win on your hardware.