Skip to content

TensorNativeDriver

Accelerate the inference stack you already run — without changing a line of your code.

TensorNativeDriver (TND) is a drop-in acceleration layer for GPU inference. You install it once; your existing Hugging Face Transformers, vLLM, and TensorRT-LLM services transparently begin running on TND's accelerated kernels and scheduler. No model re-export, no API change, no rewrite.

What TND does

On installation, TND detects which inference engines are present on the box and wires its acceleration behind each one. It never installs, upgrades, or pins Transformers, vLLM, or TensorRT-LLM on your behalf — anything absent is simply reported as not found.

Your workload TND acceleration Activation
Hugging Face Transformers (.from_pretrained → .generate) Routes supported decoder models through TND's engine Automatic
vLLM — embedding / BERT-family (vllm serve) Loads fused BERT-FFN kernels via the vLLM plugin Automatic
vLLM — text generation (vllm serve) Runs vLLM on TND's GIL-free scheduler Opt-in (TND_VLLM_SCHEDULER=1)
TensorRT-LLM (trtllm-serve) Replaces greedy sampling with a fused-greedy kernel (token-identical) Opt-in (TND_FUSED_SAMPLER=1)
No engine installed Serves directly on TND's native C++ engine tnd.serve

Safety guarantee

Unsupported models and workloads always fall back to the stock framework. Your workload never breaks because TND is installed.

Explore the docs

  • Installation & Configuration

    Requirements, the three deployment pathways (Standard / TensorRT-LLM / Coexistence), verification, and the full configuration reference.

    Install TND →

  • Engine Integration

    Exactly what changes per engine — Transformers, vLLM, TensorRT-LLM, and the native server — with serve commands and scope limits.

    Per-engine reference →

  • Benchmarking

    A rigorous baseline-vs-accelerated A/B method, how to prove the accelerator engaged, and how to read the results honestly.

    Measure the win →

  • Architecture

    How auto-integration works, the components installed, and the safety and fallback model that make TND safe in a production path.

    Understand the design →

Quick start

On a supported host (see requirements):

# 1. Download the installation bundle
gcloud storage cp gs://tnd-bundle-artifacts/releases/inference-engine/py3.10/tnd-install-linux.zip .

# 2. Install base system prerequisites
sudo apt-get update && sudo apt-get install -y python3-pip unzip

# 3. Stage the ABI-matched runtime into system python3
sudo python3 -m pip install "torch==2.8.0" --index-url https://download.pytorch.org/whl/cu128

# 4. Extract the bundle and install the core driver
unzip tnd-install-linux.zip
sudo apt install ./tnd-install-linux/tensor-native-driver_*.deb

# 5. Verify what was detected and wired
tnd-status

tnd-status — not the apt exit code — is the authoritative confirmation that acceleration is wired. See Installation & Configuration for the full procedure, the TensorRT-LLM pathway, and the configuration reference.