Architecture¶

This document describes how TensorNativeDriver (TND) integrates with your existing inference stack, what it installs, and the guarantees that make it safe to deploy into a production path.

1. Integration model¶

TND is an acceleration layer, not a serving framework you migrate to. It attaches behind the engine you already run and replaces hot-path kernels and scheduling with faster, numerically-equivalent implementations.

        Your application / OpenAI-compatible client
                          │
                          ▼
        ┌──────────────────────────────────────────┐
        │   Serving engine (unchanged)             │
        │   Transformers · vLLM · TensorRT-LLM     │
        └──────────────────────────────────────────┘
                          │
                 TND acceleration wedge
                          ▼
        ┌──────────────────────────────────────────┐
        │       TensorNativeDriver (TND)           │
        │   GIL-free scheduler · Fused kernels     │
        └──────────────────────────────────────────┘
                          │
                          ▼
                  NVIDIA GPU (CUDA)

The application above the engine and the client protocol below it are untouched. TND inserts itself at the engine's internal extension points.

2. Detection and wiring¶

Installation is detect-then-wire, and it is deterministic:

Detect. The installer inspects the target Python environment for supported engines (Transformers, vLLM, TensorRT-LLM).
Wire. For each engine found, TND registers the corresponding acceleration path — a decoder hook, a plugin, a scheduler, or a sampler.
Report. tnd-status prints the resulting state for every path: [ ACTIVE ], [ OPT-IN ], or [ MISSING ].

Engine Preservation Policy

TND never installs or pins an engine for you. An engine that is not present is reported as missing and no path is wired for it — installation still succeeds.

3. Acceleration surfaces¶

Each engine exposes a different extension point, so TND wires a different surface per engine.

Engine	Surface	Behavior
Transformers	Decoder hook	Supported decoder models are routed through TND's engine on `.generate`. Automatic on import.
vLLM (embedding / BERT)	Fused-kernel plugin	Fused BERT-FFN kernels load via vLLM's plugin system. Automatic.
vLLM (generation)	GIL-free scheduler	Generation runs on TND's scheduler. Opt-in — changes scheduling behavior.
TensorRT-LLM	Fused-greedy sampler	Greedy sampling is replaced with a fused kernel. Opt-in — token-identical to stock greedy.
Native	TND C++ engine	TND's own OpenAI-compatible server, used when no external engine is present.

Automatic vs. Opt-in

Surfaces that do not alter output or scheduling semantics are wired automatically. Surfaces that change engine behavior (the vLLM scheduler, the TensorRT-LLM sampler) are opt-in and enabled per process with an environment variable, so they are never forced onto a workload they were not validated for.

4. Numerical equivalence¶

The accelerated greedy paths are token-identical to the stock engine's greedy output. Acceleration removes kernel launches and scheduling overhead; it does not change the tokens produced.

A/B Validation

Because the outputs are identical, a baseline-vs-accelerated comparison is a pure measurement of speed and efficiency, not quality — see the Benchmarking Guide.

5. Runtime ABI contract¶

The driver ships a compiled CUDA extension (libtnd_paged_cuda.so) that is ABI-locked to a specific PyTorch build. The extension must load against the exact torch present in the environment it accelerates, or it will not initialize.

Path	Runtime environment
Native `tnd` / Transformers	System `python3` with the ABI-matched `torch` build (current release: `torch 2.8.0+cu128`).
TensorRT-LLM	The TensorRT-LLM environment, with an op library ABI-matched to that environment's `torch` (`/opt/tnd/trtllm/libtnd_paged_cuda.so`).
vLLM	vLLM's own environment (it ships a different `torch`/CUDA ABI). Isolated from the system-Python path — see below.

Preflight ABI Check

Because the compiled extension is version-specific, the prerequisites require you to pre-install the ABI-matched torch before installing the driver. The installer verifies the extension loads as part of its post-install check.

6. Deployment topologies¶

Single-engine box (recommended)¶

Install one engine plus TND per box. This keeps one runtime ABI on the box and gives the cleanest tnd-status.

vLLM isolation¶

vLLM ships its own torch/CUDA build, a different ABI than the system-Python core package. Run vLLM in its own environment (its own box or venv); TND accelerates it there via the scheduler wedge and the BERT plugin. On a box that already has vLLM, the core package detects it and does not disturb vLLM's runtime.

Detection and Multi-ABI (Bundle >= v4.3.10)

As of Inference bundle >= v4.3.10 (PR #278), detection runs under both shipped interpreters (python3.10 and python3.12) and downgrades the core's CUDA-load self-check to a warning when an engine is present, so the .deb installs cleanly (ii) instead of aborting. A genuine ABI break with no engine present still fails fatally.

Independently, the core package's apt Depends: restricts the default python3 to CPython 3.10 or 3.12, so on any other default python3 (3.11, 3.13+, or < 3.10) apt refuses at dependency resolution before configure ever runs — fail-clean by design.

Air-gapped install¶

The release bundle is self-contained: extract the nested zip and install the .deb with a local apt install ./…. The only external dependency is the ABI-matched torch wheel, which you stage into your internal mirror.

7. Components installed¶

Component	Purpose
`tensor-native-driver_*.deb`	Core package: the CUDA engine, decoder hook, and native server.
`tensor-native-driver-trt-plugin_.deb` (optional)*	TensorRT-LLM plugin layer (custom TRT ops).
`libtnd_paged_cuda.so`	The ABI-locked CUDA extension.
`tnd-status`	Detection and wiring dashboard (human + `--json`).
`tnd.serve`	The native OpenAI-compatible server entry point.
`tnd-server` (systemd unit)	Managed native-server service.

See the Installation & Configuration guide for the install procedure and the Engine Integration Reference for per-engine detail.