Installation & Configuration¶

This guide takes a host from clean to a verified TensorNativeDriver (TND) install, then documents every configuration surface. TND accelerates the inference engine you already run — no code change is required.

1. System requirements¶

Requirement	Specification
Operating system	Linux x86_64, Ubuntu 22.04 LTS
GPU	NVIDIA GPU with a CUDA 12.8-capable driver (driver `>= 525`)
GPU driver note	A CUDA-13 host driver (`nvidia-driver-580`) is supported — the driver is forward-compatible — but the `torch` build must remain `+cu128` (below)
Python	The default `python3` must be CPython 3.10 or 3.12 (the two ABIs the multi-ABI package ships; PEP 3149 auto-select) — see the Note on Dependencies below.
Runtime	`torch 2.8.0+cu128`, pre-installed into system `python3` (see §2) — skip this if you're installing vLLM; vLLM's own `torch` requirement becomes the baseline instead (see Pathway A)
Engine (optional)	The inference engine you intend to accelerate — Transformers, vLLM, and/or TensorRT-LLM — installed into that same system `python3` (TensorRT-LLM is the one exception: isolated venv, see Pathway B)

Note on Dependencies (supported Python)

The core package's Depends: field encodes a contract on the OS-level python3 apt package's own version (what dpkg -s python3 reports) — not on whatever /usr/bin/python3 is currently symlinked to:

Depends: python3 (>= 3.10),
         python3 (<< 3.11) | python3 (>= 3.12),
         python3 (<< 3.13)

When your distro's python3 package itself is a version other than 3.10 or 3.12 (e.g. a fresh install where that's genuinely the OS default), apt install ./tensor-native-driver_*.deb fails at dependency resolution with a clean apt error and never reaches state ii — no half-installed state. This is by design, not a defect.

This does not protect you if you change the effective default afterward. update-alternatives repoints the /usr/bin/python3 symlink — it does not change the OS's own python3 package version that the Depends: line above actually checks. Confirmed live: on Ubuntu 22.04 (python3 package fixed at 3.10.6-1~22.04.1), switching the default to python3.11 via update-alternatives and then running apt install ./tensor-native-driver_*.deb succeeds — ii state, no refusal — because apt is still checking the unchanged OS package version. The break shows up later instead: the package only ships compiled _paged_core extensions tagged cpython-310/cpython-312 (PEP 3149), so from tnd import _paged_core raises ImportError: cannot import name '_paged_core' from 'tnd' the first time anything tries to use the engine — a silent runtime failure, not a clean apt refusal.

Remediation: Switch the system-default python3 to 3.10 or 3.12 (e.g. via update-alternatives or your distro's python packages), then re-run apt install. If you switch the default to anything else later, apt will not catch it for you — run python3 -c "from tnd import _paged_core" yourself to confirm the binding still resolves. Do NOT use --force-depends: it would install an unusable/mismatched binding.

Install-acceptance matrix

Default python3	`apt install ./tensor-native-driver_*.deb` result
3.10 (the OS's own `python3` package)	SUCCESS → state `ii`; `python3 -c "from tnd import PagedAttentionEngine"` imports
3.12 (the OS's own `python3` package)	SUCCESS → state `ii`; import works
3.11 / 3.13+ / < 3.10 as the OS's own `python3` package	FAIL at dependency resolution (clean apt error); NO half-installed state
3.11 / 3.13+ selected via `update-alternatives`, OS package still 3.10/3.12	apt still SUCCEEDS → state `ii` (OS package version unchanged) — but `from tnd import _paged_core` raises `ImportError` the first time anything uses the engine; confirmed live on Ubuntu 22.04 switched to `python3.11`

Why the system python3

The core package's post-install runs its torch.ops.load_library check against the box's system python3, not a virtualenv or conda env. Installing the engine into a venv leaves it where the post-install cannot see it, and the check can report a false success.

Install vLLM before the core driver, not after

vLLM shares the core package's system python3 (no separate venv) — install it first: sudo python3 -m pip install vllm==<version>, then install tensor-native-driver_*.deb afterward. Reversing the order leaves the driver's ABI-variant detection stale, silently degrading acceleration elsewhere on the box. See Engine Integration — vLLM for why this matters and what happens if you get the order wrong.

Installing the core .deb on a box that already runs vLLM or TensorRT-LLM

This presumes the box's default python3 is already CPython 3.10 or 3.12 — without that, apt refuses at dependency resolution before reaching this stage (see the Note on Dependencies above).

Requires Inference bundle >= v4.3.10 (the release cut with the multi-ABI engine-detection fix, Inference PR #278).

From that bundle on, the core package's post-install detects an engine present under either shipped interpreter (python3.10 and python3.12) and downgrades its libtnd_paged_cuda.so load self-check from a fatal error to a warning. The install then stays clean — apt returns 0, dpkg -l shows ii — and the warning points you at the vLLM scheduler / TensorRT-LLM plugin path instead of aborting. A genuine ABI mismatch with no engine present still fails fatally (see §2).

Get the bundle¶

The install bundle is not publicly downloadable. Your Iturintel contact will send you a time-limited download link (revocable, typically valid a few days). Use it directly — no GCP account or gcloud auth needed:

curl -fL "<the-link-you-were-sent>" -o tnd-install-linux.zip

It extracts (single unzip) to tnd-install-linux/ containing three DEBs: * tensor-native-driver_*.deb — CORE driver. Installs into system python3; ABI-locked to torch 2.8.0+cu128. * tensor-native-driver-trtllm-accel_*.deb — TensorRT-LLM acceleration package. Ships tnd-trtllm-setup and tnd-trtllm-launch (the ABI-matched libtnd_paged_cuda.so variants ship inside the CORE .deb, under /opt/tnd/abi/<variant>/, and are resolved automatically). REQUIRED for the TRT-LLM fused-sampler path. * tensor-native-driver-trt-plugin_*.deb — OPTIONAL custom TensorRT plugin layer. Depends on libnvinfer10 / libnvinfer-plugin10, which are NOT installable without NVIDIA's TensorRT apt repo. It is NOT required for fused-sampler acceleration.

2. Prerequisites¶

A clean box has neither pip, unzip, nor OpenMPI. Install these system dependencies before anything else:

sudo apt-get update && sudo apt-get install -y python3-pip unzip libopenmpi-dev openmpi-bin

python3-pip — The host uses python3 -m pip but pip is absent on a clean box, causing No module named pip errors.
unzip — Required to extract the installation bundle; otherwise fails with unzip: command not found.
libopenmpi-dev openmpi-bin — tensorrt_llm dynamically loads libmpi.so at import time. Without it, importing tensorrt_llm raises RuntimeError: cannot load MPI library.

After the system dependencies are installed, install the exact ABI-matched torch runtime in system python3 (required for the core driver's CUDA-core load check):

sudo python3 -m pip install "torch==2.8.0" --index-url https://download.pytorch.org/whl/cu128

Skip this step if you're installing vLLM on this box. Install vLLM first instead (sudo python3 -m pip install vllm==<version>) — its own torch requirement becomes the ABI baseline the driver detects and matches at install time. See Pathway A below.

3. Choose your pathway¶

Depending on the inference engine(s) you run on the host, choose one of the following deployment pathways:

Pathway	Target Workload	Key Components Installed
Pathway A — Standard	Transformers and/or vLLM	Core driver only. Runs on system Python; torch build is `cu128` (Transformers-only) or whatever vLLM requires (vLLM installed first).
Pathway B — TensorRT-LLM	TensorRT-LLM	Core driver + TRT-LLM acceleration. Isolated cu130 virtual environment for TensorRT-LLM.
Pathway C — Coexistence	Standard + TensorRT-LLM	Both paths on the same host, keeping the environments strictly isolated.

Pathway A — Standard (Transformers and/or vLLM)¶

Use this pathway when running Hugging Face Transformers and/or vLLM. The driver installs directly into your system python3 — vLLM shares that same system python3, no separate venv.

With vLLMTransformers only (no vLLM)

Install vLLM before the core driver — see the warning above for why.

# Install system prerequisites
sudo apt-get update && sudo apt-get install -y python3-pip unzip

# Install vLLM FIRST -- its torch requirement becomes the ABI baseline
sudo python3 -m pip install vllm==0.23.0

# Extract the bundle and install the core driver
# (auto-detects vLLM's torch build, activates the matching variant)
unzip tnd-install-linux.zip
sudo apt install ./tnd-install-linux/tensor-native-driver_*.deb

# Verify the integration
tnd-status

Then serve — TND's scheduler wires in automatically, no extra flag needed. See Engine Integration — vLLM for the serve command, how to confirm engagement, scope limits, and the opt-out flag.

# Install system prerequisites and staging runtime
sudo apt-get update && sudo apt-get install -y python3-pip unzip
sudo python3 -m pip install "torch==2.8.0" --index-url https://download.pytorch.org/whl/cu128

# Extract the bundle
unzip tnd-install-linux.zip

# Install the core driver
sudo apt install ./tnd-install-linux/tensor-native-driver_*.deb

# Verify the integration
tnd-status

Transformers acceleration is automatic on import — nothing further to do.

Pathway B — TensorRT-LLM (isolated cu130 venv; system python UNTOUCHED)¶

Use this pathway for TensorRT-LLM acceleration. To maintain stability, TRT-LLM and its dependencies are isolated in a virtual environment (/opt/tnd/venv), keeping the system Python clean.

P0 WARNING: System-Python ABI Trap

NEVER pip install tensorrt_llm directly into the system python3. The PyPI wheel for tensorrt_llm will pull torch off 2.8.0+cu128 to a CUDA-13 build. This breaks the core driver's ABI lock and causes a segmentation fault on every subsequent python3 invocation on the box (due to the driver's sitecustomize/autowire bootstrap that runs on interpreter startup). This is silent and catastrophic. TRT-LLM must live in the isolated venv built by tnd-trtllm-setup below; the system python3 stays on cu128.

# 1. Install system prerequisites (including OpenMPI for TRT-LLM)
sudo apt-get update && sudo apt-get install -y python3-pip unzip libopenmpi-dev openmpi-bin

# 2. Stage the core driver runtime on system python (torch cu128)
sudo python3 -m pip install "torch==2.8.0" --index-url https://download.pytorch.org/whl/cu128

# 3. Extract the bundle
unzip tnd-install-linux.zip

# 4. Install the core driver
sudo apt install ./tnd-install-linux/tensor-native-driver_*.deb

# 5. Install the TRT-LLM acceleration package (ships tnd-trtllm-setup & tnd-trtllm-launch)
sudo apt install ./tnd-install-linux/tensor-native-driver-trtllm-accel_*.deb

# 6. Build the isolated virtual environment (installs torch 2.10.0+cu130 and tensorrt_llm 1.3.0rc17)
# This step takes ~10 minutes and downloads several gigabytes
sudo tnd-trtllm-setup --venv /opt/tnd/venv

Serve (trtllm-serve lives in the venv, so put the venv bin on PATH):

# baseline (stock TensorRT-LLM):
PATH="/opt/tnd/venv/bin:$PATH" tnd-trtllm-launch --baseline -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000

# accelerated (TND fused-greedy sampler; token-identical to stock greedy):
PATH="/opt/tnd/venv/bin:$PATH" TND_FUSED_SAMPLER=1 tnd-trtllm-launch --accel -- \
  serve Qwen/Qwen2.5-1.5B-Instruct --host 0.0.0.0 --port 8000

PATH & Binary Callout

After running tnd-trtllm-setup, the trtllm-serve binary resides inside the virtual environment at /opt/tnd/venv/bin/trtllm-serve and is not on the default system PATH. Because tnd-trtllm-launch executes trtllm-serve from the active environment PATH, you must prepend the virtual environment to your PATH (e.g. PATH="/opt/tnd/venv/bin:$PATH") or explicitly set the environment variable TRTLLM_SERVE_BIN=/opt/tnd/venv/bin/trtllm-serve.

Note: serve <MODEL> is positional under trtllm-serve 1.3.x (the old top-level --model was removed).

Pathway C — Coexistence (core + TRT-LLM on the same box)¶

To run Standard workloads (system python3, Transformers and/or vLLM) and TensorRT-LLM workloads on the same host, follow the installation steps for both Pathway A and Pathway B.

Because the system Python and the virtual environment at /opt/tnd/venv never share an interpreter, the ABI locks are preserved: each keeps its own torch build (whatever Pathway A resolved on system python3 — 2.8.0+cu128 if Transformers-only, or vLLM's own build if vLLM is in use — versus 2.10.0+cu130 in /opt/tnd/venv), and neither install disturbs the other — no conflicts or segfaults.

tnd-status only sees the interpreter it runs under

tnd-status reports only the engines importable from its own interpreter — it does not know that another interpreter on the same box has other engines correctly wired, and it reports those as missing rather than "not applicable here":

Under system python3: reports Transformers and vLLM correctly (both share this interpreter).
Under /opt/tnd/venv/bin/python (run as /opt/tnd/venv/bin/python /usr/bin/tnd-status): reports TensorRT-LLM correctly, but shows Transformers and vLLM as [MISSING], including the line "Transformers acceleration NOT active — install incomplete... your Hugging Face code runs STOCK" — even when both are fully wired and accelerated on the system Python.

Neither omission means acceleration failed. To get the complete picture on a coexistence box, run tnd-status under each interpreter you installed into (system python3 for Transformers/vLLM, /opt/tnd/venv/bin/python for TensorRT-LLM) and read each one only for the engine(s) that live in that interpreter.

4. What gets accelerated¶

Once installed, your existing code runs unchanged. The behavior depends on the engine detected on the box:

Your workload	After install	Action needed
Transformers (supported decoder)	Routed through TND's engine	None
vLLM (generation)	Runs on TND's scheduler by default once vLLM shares the driver's system `python3`	None (install vLLM before the driver; opt out with `TND_VLLM_SCHEDULER=0` — see Engine Integration)
TensorRT-LLM	Runs stock unless opted in	Set `TND_FUSED_SAMPLER=1`

Anything unsupported falls back to the stock framework — your workload never breaks because TND is installed.

5. Verify: `tnd-status`¶

Run the dashboard at any time:

tnd-status

It reports, on one screen: * Detected engines — Transformers / vLLM / TensorRT-LLM found on this box. * Wiring state per path — [ ACTIVE ], [ OPT-IN ], or [ MISSING ]. * The serve command for each detected engine.

For automation and health checks, use the machine-readable form:

tnd-status --json

Wiring values: ok = active, opt = available but not enabled, miss = the component that path needs is not installed. If a path you expect as [ ACTIVE ] shows [ MISSING ], tnd-status names the missing component.

Restart required to pick up new wiring

Installing or upgrading TND (.deb or wheel) takes effect for new processes only — it does not affect a Python process, vllm serve, or trtllm-serve that was already running. Install first, then start (or restart) the server; a server already running when you install TND will not pick up the new wiring until it is stopped and started again.

6. Configuration reference¶

6.1 Environment variables¶

Acceleration paths are controlled per process with environment variables.

Variable	Effect
`TND_DISABLE_AUTO_INTEGRATION=1`	Master off switch — disables all auto-integration for the process.
`TRANSFORMERS_TND_DISABLE=1`	Disable only Transformers acceleration (run stock Hugging Face).
`TND_VLLM_SCHEDULER=0`	Opt out of the scheduler wedge for a process (default is ON once vLLM shares the driver's system `python3` — see Pathway A). Also accepts `false`/`no`/`off`.
`TND_FUSED_SAMPLER=1`	Opt in to the TensorRT-LLM fused-greedy sampler.
`TND_PAGED_CUDA_LIB=<path>`	Override the ABI-matched op library `tnd-trtllm-launch` resolves for the TensorRT-LLM path. Unset (the default), it auto-resolves the variant matching the venv's `torch` build from `/usr/share/tnd/abi_variants.json` (e.g. `/opt/tnd/abi/cu128-torch2.10/libtnd_paged_cuda.so`) — there is no single static default path.

6.2 Opt-in paths¶

One path changes engine behavior and is therefore never forced on: the TensorRT-LLM fused sampler (TND_FUSED_SAMPLER=1). See Engine Integration — TensorRT-LLM for the serve command and how to confirm engagement, or Pathway B above for the install steps. (vLLM's scheduler wedge is default-ON once vLLM shares the driver's system python3 — see Pathway A above — with a per-process opt-out via TND_VLLM_SCHEDULER=0 above; it still carries the same scope limits, it's just no longer a separate install step.)

7. Native TND server¶

TND ships its own OpenAI-compatible server for when no external engine is present.

Requires transformers, even with no external engine

The native server loads model configs/checkpoints through the same code path as Transformers acceleration, which hard-depends on transformers (pip install transformers) — it is required infrastructure here, not an optional acceleration target, despite tnd-status's "DETECTED SERVING ENGINES" line listing it that way. Installing only the core .deb (no transformers) installs cleanly and tnd-status reports OK, but tnd.serve / tnd-server will crash-loop on startup with ModuleNotFoundError: No module named 'transformers' until it's installed.

# Managed service — configure /etc/tnd-server/tnd-server.conf first, then:
sudo systemctl enable --now tnd-server

# Ad hoc:
python3.10 -m tnd.serve --model <model> --host 0.0.0.0 --port 8000

Sanity-check the endpoint and point any OpenAI-compatible client at http://<host>:8000/v1:

curl http://localhost:8000/health
curl http://localhost:8000/v1/models

Serve options¶

Option	Default	Description
`--model`	—	Hugging Face model id.
`--host` / `--port`	`127.0.0.1` / `8000`	Bind address. The default binds to localhost only — pass `--host 0.0.0.0` explicitly to accept connections from other hosts.

8. Troubleshooting¶

Symptom	Cause	Resolution
`apt install ./tensor-native-driver_*.deb` aborts with an unmet `python3` dependency	default `python3` is not CPython 3.10 or 3.12 (fail-clean by design)	Switch the system-default `python3` to 3.10 or 3.12 (e.g. via `update-alternatives` or your distro's python packages), then re-run `apt install`. Do NOT use `--force-depends` — it would install an unusable/mismatched binding.
Segfault on any `python3` after installing TensorRT-LLM	`pip install tensorrt_llm` pulled system `torch` off `2.8.0+cu128` (CUDA-13 wheel), breaking the core-driver ABI lock	Reinstall `torch==2.8.0 --index-url .../cu128` into system python; keep TRT-LLM in the `/opt/tnd/venv` venv built by `tnd-trtllm-setup` — never in system python
`No module named pip` / `unzip: command not found`	bare Ubuntu 22.04 lacks base tooling	`sudo apt-get install -y python3-pip unzip`
`RuntimeError: cannot load MPI library` on `import tensorrt_llm`	OpenMPI runtime absent (`tensorrt_llm` dlopens `libmpi.so`)	`sudo apt-get install -y libopenmpi-dev openmpi-bin`
`trtllm-serve: command not found` via `tnd-trtllm-launch`	serve binary is in the venv, not on PATH	prepend `PATH="/opt/tnd/venv/bin:$PATH"` or set `TRTLLM_SERVE_BIN=/opt/tnd/venv/bin/trtllm-serve`
A path expected as `[ ACTIVE ]` shows `[ MISSING ]`	The engine, or a TND component it needs, is not installed in this Python	`tnd-status` names the missing component; install it in the same system `python3`, then re-check
Transformers ran on stock Hugging Face	Model unsupported (falls back by design), or `TRANSFORMERS_TND_DISABLE=1` is set	Confirm with `tnd-status`; look for the `TND transformers decoder hook installed` log line
vLLM log does not name the TND scheduler	vLLM is not sharing the driver's system `python3` (e.g. installed into its own venv), `TND_VLLM_SCHEDULER=0` is set, the server wasn't restarted after installing the driver, or a custom `--scheduler-cls` was already passed	Confirm `vllm` and the driver import from the same `python3`; restart the server after install; check for `Using custom scheduler class tnd.vllm_adapter.TndScheduler` in the log
TensorRT-LLM sampler did not arm	`TND_FUSED_SAMPLER` unset, or no ABI-matched op-library variant was found for the venv's `torch` build	`tnd-trtllm-launch` already exits with a `FATAL` error and no serve attempt in this case — read that error, or set `TND_PAGED_CUDA_LIB` to an explicit matching variant
`/health` refused right after start	Model still loading	Wait for the model to finish loading, then re-check

9. Support diagnostics¶

If an install does not land the way tnd-status reports, capture the following and include it with any support request:

echo $?; dpkg -l | grep tensor-native-driver
python3 -c 'import torch; print(torch.__version__, torch.version.cuda)'
ldd /usr/lib/x86_64-linux-gnu/libtnd_paged_cuda.so | grep 'not found'
tnd-status --json

Architecture — how auto-integration works.
Engine Integration Reference — per-engine detail and serve commands.
Benchmarking Guide — prove the win on your hardware.