Architecture¶

This document describes how TensorNativeDriver (TND) integrates with your existing inference stack, what it installs, and the guarantees that make it safe to deploy into a production path.

1. Integration model¶

TND is an acceleration layer, not a serving framework you migrate to. It attaches behind the engine you already run and replaces hot-path kernels and scheduling with faster, numerically-equivalent implementations.

        Your application / OpenAI-compatible client
                          │
                          ▼
        ┌──────────────────────────────────────────┐
        │   Serving engine (unchanged)             │
        │   Transformers · vLLM · TensorRT-LLM     │
        └──────────────────────────────────────────┘
                          │
                 TND acceleration wedge
                          ▼
        ┌──────────────────────────────────────────┐
        │       TensorNativeDriver (TND)           │
        │   GIL-free scheduler · Fused kernels     │
        └──────────────────────────────────────────┘
                          │
                          ▼
                  NVIDIA GPU (CUDA)

The application above the engine and the client protocol below it are untouched. TND inserts itself at the engine's internal extension points.

2. Detection and wiring¶

Installation is detect-then-wire, and it is deterministic:

Detect. The installer inspects the target Python environment for supported engines (Transformers, vLLM, TensorRT-LLM).
Wire. For each engine found, TND registers the corresponding acceleration path — a decoder hook, a plugin, a scheduler, or a sampler.
Report. tnd-status prints the resulting state for every path: [ ACTIVE ], [ OPT-IN ], or [ MISSING ].

Engine Preservation Policy

TND never installs or pins an engine for you. An engine that is not present is reported as missing and no path is wired for it — installation still succeeds.

3. Acceleration surfaces¶

Each engine exposes a different extension point, so TND wires a different surface per engine.

Engine	Surface	Behavior
Transformers	Decoder hook	Supported decoder models are routed through TND's engine on `.generate`. Automatic on import.
vLLM (generation)	GIL-free scheduler	Generation runs on TND's scheduler. Default ON once vLLM shares the driver's system `python3` — changes scheduling behavior, opt out with `TND_VLLM_SCHEDULER=0`.
TensorRT-LLM	Fused-greedy sampler	Greedy sampling is replaced with a fused kernel. Opt-in — token-identical to stock greedy.
Native	TND C++ engine	TND's own OpenAI-compatible server, used when no external engine is present.

Automatic-and-safe vs. behavior-changing

Transformers' decoder hook doesn't alter output or scheduling semantics and auto-detects unsupported models, so it's wired automatically with no caveat. The vLLM scheduler and the TensorRT-LLM sampler both change engine behavior and do not auto-detect out-of-scope workloads — that's true whether or not there's a flag to pass. The TensorRT-LLM sampler additionally requires an explicit opt-in (TND_FUSED_SAMPLER=1); the vLLM scheduler is default-ON (opt-out via TND_VLLM_SCHEDULER=0) once vLLM shares the driver's system Python, which is a statement about installation, not about scope safety.

4. Numerical equivalence¶

The accelerated greedy paths are token-identical to the stock engine's greedy output. Acceleration removes kernel launches and scheduling overhead; it does not change the tokens produced.

A/B Validation

Because the outputs are identical, a baseline-vs-accelerated comparison is a pure measurement of speed and efficiency, not quality — see the Benchmarking Guide.

5. Runtime ABI contract¶

The driver ships a compiled CUDA extension (libtnd_paged_cuda.so) that is ABI-locked to a specific PyTorch build. The extension must load against the exact torch present in the environment it accelerates, or it will not initialize.

Path	Runtime environment
Native `tnd` / Transformers	System `python3` with the ABI-matched `torch` build (current release: `torch 2.8.0+cu128`, or whatever vLLM requires if vLLM is also on this box — see the vLLM row).
TensorRT-LLM	The TensorRT-LLM environment, with an op library ABI-matched to that environment's `torch`, auto-resolved by `tnd-trtllm-launch` from the variants the core `.deb` ships under `/opt/tnd/abi/<variant>/`.
vLLM	The same system `python3` as Native/Transformers — no separate environment. vLLM may pull a different `torch`/CUDA build than the default `cu128` pin; install vLLM before the driver so its post-install detects and activates the matching variant (see below).

Preflight ABI Check

Because the compiled extension is version-specific, the prerequisites require you to pre-install the ABI-matched torch before installing the driver. The installer verifies the extension loads as part of its post-install check.

6. Deployment topologies¶

Single-engine box (recommended)¶

Install one engine plus TND per box. This keeps one runtime ABI on the box and gives the cleanest tnd-status.

vLLM install order¶

vLLM shares the driver's system python3 — there is no separate environment. It may pull a different torch/CUDA build than the core package's default cu128 pin, so install vLLM before the driver: the driver's post-install detects whichever torch is already present and activates the matching pre-built libtnd_paged_cuda.so variant. That detection runs once, at install time — reversing the order (driver first, vLLM after) leaves the activated variant orphaned. vLLM's own scheduler is unaffected either way (it never touches that library), but native tnd.serve and Transformers acceleration on the same box can silently lose their CUDA-graph/fused-sampler acceleration if the order is reversed.

Detection and Multi-ABI (Bundle >= v4.3.10)

As of Inference bundle >= v4.3.10 (PR #278), detection runs under both shipped interpreters (python3.10 and python3.12) and downgrades the core's CUDA-load self-check to a warning when an engine is present, so the .deb installs cleanly (ii) instead of aborting. A genuine ABI break with no engine present still fails fatally.

Independently, the core package's apt Depends: restricts the OS's own python3 package to CPython 3.10 or 3.12 — when that package itself is a different version, apt refuses at dependency resolution before configure ever runs, fail-clean by design. This check does not re-trigger if you later repoint the default python3 via update-alternatives: apt still sees the unchanged OS package version and installs successfully, but the engine silently fails to import under the new default (see Installation §1).

Air-gapped install¶

The release bundle is self-contained: extract the nested zip and install the .deb with a local apt install ./…. The only external dependency is the ABI-matched torch wheel, which you stage into your internal mirror.

7. Components installed¶

Component	Purpose
`tensor-native-driver_*.deb`	Core package: the CUDA engine, decoder hook, and native server.
`tensor-native-driver-trt-plugin_.deb` (optional)*	TensorRT-LLM plugin layer (custom TRT ops).
`libtnd_paged_cuda.so`	The ABI-locked CUDA extension.
`tnd-status`	Detection and wiring dashboard (human + `--json`).
`tnd.serve`	The native OpenAI-compatible server entry point.
`tnd-server` (systemd unit)	Managed native-server service.

See the Installation & Configuration guide for the install procedure and the Engine Integration Reference for per-engine detail.