vLLM vs SGLang vs TensorRT-LLM: how to choose a self-hosted inference engine in 2026

AI infrastructure & software

You have already decided to run your own GPUs. Now someone is asking you to pick a serving engine, and every deck in your inbox quotes a single throughput number. Those numbers are real, and they are nearly useless to you: the right engine for your platform is decided by the shape of your traffic and the strictness of your latency SLO, not by a leaderboard. This post gives you the decision table, the operational costs nobody quotes, and a benchmark you can run in a day to settle the question on your own hardware.

The short answer: vLLM vs SGLang vs TensorRT-LLM

If you want the decision in one table before the reasoning, here it is.

Everything below explains why that table holds, where the published numbers come from, and how to verify it yourself on your own infrastructure in one working day.

What the audited numbers actually say

Start with the only cross-engine numbers that are externally audited: MLPerf Inference v6.0, released by MLCommons on 1 April 2026 (announcement, benchmark definitions). This round added a GPT-OSS 120B benchmark with three scenarios, including a latency-bounded Server scenario with demanding constraints: p99 time to first token under 3 seconds and p99 time per output token under 80 milliseconds, with input lengths ranging from 600 to 15,000 tokens.

On GPT-OSS 120B running on 8×B200 systems, the audited spread is instructive:

• A vLLM stack (Red Hat OpenShift AI with the llm-d scheduler and vLLM) submitted 93,071 output tokens/s offline and 71,588 tokens/s server, which Red Hat reports as about 8% ahead of the nearest comparable B200 submission in the throughput-oriented Offline scenario (Red Hat's submission write-up).
• TensorRT-LLM based submissions led the latency-bounded Server scenario on the same model and hardware class by roughly 22%, while trailing on raw offline throughput.
• NVIDIA reported that TensorRT-LLM and Dynamo optimisations delivered up to 2.7x higher token throughput on the same GB300 NVL72 hardware compared with six months earlier, and notably used vLLM for its Qwen3-VL multimodal submission (NVIDIA's write-up).

Two operational facts matter far more than who won a given bar chart. First, the audited spread on identical hardware is single-digit to low-double-digit percentages, not the 2x to 5x figures routinely claimed in vendor marketing blogs. Second, 20 submitters used vLLM in this round, up from 5 submitters in v5.1. That surge illustrates where open-source ecosystem gravity has settled.

Compare that audited reality with un-audited benchmarks. An independent cross-engine harness published in June 2026 (methodology and results) reports SGLang's RadixAttention advantage as "up to 5x" on extreme synthetic prefix sharing, yet measures a much more modest 1.3 to 1.4x on a realistic enterprise RAG workload with a 4K shared system prompt. It also notes that repeated runs carry a variance band of plus or minus 15 to 20%. Both realities coexist: the caching mechanism is genuinely powerful, but the marketing multiplier depends on synthetic assumptions that rarely reflect production traffic.

The takeaway is not to ignore benchmarks. It is to benchmark your own traffic shape, at your concurrency, under your strict latency service level objectives (SLOs).

Where the engines actually differ

All three engines implement continuous batching, paged or blocked KV cache management, chunked prefill, and FP8 execution. The architectural differences that alter your production throughput and latency are much narrower than vendor marketing suggests.

Consider the final row carefully against your specific workload. In Spheron's single-H100 benchmark using fully unique prompts (Llama 3.3 70B FP8, vLLM 0.18.0, SGLang 0.5.9, TensorRT-LLM 1.2.0), SGLang and vLLM finished within 4% of each other on raw throughput because RadixAttention had no shared tokens to reuse. In the alternative harness where incoming requests shared a 4K system prompt, SGLang pulled 30 to 40% ahead. Identical serving engines reached opposing conclusions purely due to differences in prompt structure.

Choose by traffic shape, not by leaderboard

Three distinct traffic profiles cover the vast majority of enterprise deployments in the UAE and wider GCC region.

1. Prefix-heavy traffic

This pattern includes retrieval-augmented generation (RAG) platforms with lengthy system instructions, legal or regulatory compliance assistants carrying mandatory policy blocks, multi-turn conversational agents, and agentic workflows running structured evaluation loops.

This is where SGLang's radix tree structure excels. Instead of discarding KV cache entries or relying solely on block hashing, SGLang maintains a tree structure in memory. When a request arrives, the engine finds the longest matching token prefix and reuses the computed KV states immediately. The prefill phase for those tokens is skipped entirely.

Empirical field testing indicates an operational threshold of around 60% prefix overlap. When shared prefix tokens exceed 60% of total input tokens across incoming requests, SGLang delivers a substantial throughput and latency advantage. Below 60%, vLLM with --enable-prefix-caching active closes most of the performance difference. Multi-turn chat agents represent an ideal workload for SGLang, as every consecutive turn resends the entire conversational history as an extended prefix.

2. Strict tail-latency traffic

This profile covers interactive voice agents, real-time code completion, customer-facing search copilot integrations, and any deployment governed by strict contractual latency SLOs.

TensorRT-LLM led across every measured latency percentile in both independent benchmarks. In the Spheron evaluation at a concurrency of 50 requests:

• Time to first token (TTFT) p50 reached 340 ms for TensorRT-LLM, compared to 360 ms for SGLang and 380 ms for vLLM.
• The TTFT p95 divergence was wider: 620 ms for TensorRT-LLM versus 680 ms for SGLang and 720 ms for vLLM.
• On a 4×H100 cluster running conversational chat at concurrency 64, TensorRT-LLM clocked a TTFT p99 of 265 ms, against 340 ms for SGLang and 410 ms for vLLM.

If your technical team has committed to an internal SLO requiring p99 TTFT under 300 ms on a 70B parameter model, that 145 ms margin is the difference between compliant production operations and an SLA breach. TensorRT-LLM achieves this tail compression through custom fused kernels, specialised memory scheduling, and CUDA graph execution that avoids CPU launch overheads.

3. High model churn and broad catalogue requirements

This profile describes platforms supporting multiple business units, regular model fine-tuning cycles, experimental agent pilots, LoRA adapter routing, and mixed GPU fleets combining older Hopper nodes with newer hardware.

In this environment, vLLM remains the premier operational choice. Supporting over 200 Hugging Face model architectures, vLLM allows platform engineers to onboard a new model checkpoint simply by updating a Kubernetes manifest or configuration argument. Quantisation formats like FP8, AWQ, or GPTQ are applied dynamically with command-line flags such as --quantization fp8, without requiring offline compilation scripts.

Furthermore, vLLM provides the most mature ecosystem support, the deepest Prometheus metrics instrumentation, and the broadest developer community. If your platform engineering team is lean, vLLM minimises operational maintenance overhead.

Worked example: what a million output tokens costs on one H100

Consider a realistic deployment scenario: a bilingual banking copilot built for a financial institution in Dubai, serving English and Arabic inquiries. The workload runs Llama 3.3 70B Instruct at FP8 precision on a single NVIDIA H100 SXM 80GB GPU. Model weights occupy approximately 70GB in FP8, leaving the remaining memory for KV cache allocation using --max-model-len 8192 and --gpu-memory-utilization 0.92.

Each incoming request carries a 3,000-token system prompt containing regulatory policies, formatting guidelines, and retrieved documentation blocks. The end user provides 200 tokens of input, and the model generates 250 output tokens. Steady peak operating volume is 50 concurrent requests.

Step 1: Measure prefix overlap

Before selecting an engine or tuning hardware flags, measure the exact prefix reuse of your production traffic. Launch an SGLang instance configured with --enable-cache-report, dispatch 200 production sample requests, and read the usage.prompt_tokens_details.cached_tokens field in the API response.

If roughly 3,000 of your 3,200 input tokens return as cached on warm requests, your workload exhibits 94% prefix overlap. RadixAttention will yield immediate gains.

Step 2: Benchmark at operating concurrency

Benchmark the engines at your realistic concurrency level of 50, rather than at saturation limits. Drawing from published single-H100 measurements with unique prompts:

Step 3: Translate throughput into infrastructure costs

The conversion formula from raw output tokens per second to cost per million output tokens is:

# cost per 1M output tokens = node $/hour / (output tok/s * 3600) * 1e6
rate_usd_per_hour = 2.64  # Spheron H100 SXM on-demand rate, October 2026
engines = {
    "TensorRT-LLM": 2100,
    "SGLang": 1920,
    "vLLM": 1850,
}

for name, tps in engines.items():
    cost_per_million = (rate_usd_per_hour / (tps * 3600)) * 1e6
    print(f"{name}: ${cost_per_million:.2f} per 1M output tokens")
# TensorRT-LLM: $0.35 per 1M output tokens
# SGLang: $0.38 per 1M output tokens
# vLLM: $0.40 per 1M output tokens

Under these baseline parameters, generating 250 output tokens costs less than $0.0001 in compute time per answered query across all three engines.

Step 4: Apply the traffic shape correction

The baseline figures above reflect unique, non-overlapping prompts. Once we evaluate the actual workload structure (a 4K shared system prompt with 8K total context across concurrency 256 on a 4×H100 cluster billed at $3.50 per GPU-hour), independent measurements demonstrate how traffic shape alters unit economics:

• SGLang: $0.49 per million output tokens
• TensorRT-LLM: $0.64 per million output tokens
• vLLM: $0.70 per million output tokens

For an enterprise cluster serving 750 million output tokens daily, SGLang requires roughly $367 per day, compared to $480 per day on TensorRT-LLM and $525 per day on vLLM. Over an annual operational cycle, choosing an engine aligned with your prefix pattern yields over $57,000 in direct compute savings on a single cluster.

Step 5: Formulate the decision

Because this specific copilot workload demonstrates 94% prefix overlap and carries no contractual sub-300 ms p99 TTFT requirement, SGLang represents the optimal, cost-efficient deployment choice. vLLM with --enable-prefix-caching serves as a viable fallback. If product managers later impose a strict p99 TTFT threshold under 250 ms, the engineering team should pivot the deployment to TensorRT-LLM to safeguard tail latency.

A benchmark you can run in a day

To avoid vendor bias, establish an internal reproducible benchmark. Fix the technical parameters that benchmark marketing routinely obscures: exact Hugging Face checkpoint revision hash, precision configuration (FP8 versus INT4), physical GPU model, tensor-parallel degree, maximum context window (max_model_len), host OS kernel, NVIDIA driver, and CUDA toolkit version.

Evaluate four key operational metrics under your target production concurrency:

1. TTFT p50 and p99: The critical indicator of perceived interactive responsiveness.
2. Output token throughput: Total generation speed at target concurrency.
3. KV cache utilisation and prefix hit rate: Memory efficiency during sustained load.
4. Cost per million output tokens: True operational expenditure calculated at your realistic traffic rate.

Both vLLM and SGLang bundle native client benchmarking utilities designed to capture end-to-end metrics at the network edge:

# vLLM: Measure throughput and latency percentiles against a running instance
vllm bench serve \
  --backend vllm \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --dataset-name sharegpt \
  --dataset-path ./sharegpt.json \
  --request-rate 8 \
  --num-prompts 400 \
  --percentile-metrics ttft,tpot,itl \
  --metric-percentiles 50,99 \
  --output-json results_vllm.json

# SGLang: Run steady-state benchmark with custom operational traces
python3 -m sglang.bench_serving \
  --backend sglang \
  --max-concurrency 50 \
  --num-prompts 400 \
  --dataset-name custom \
  --dataset-path ./production_trace.jsonl

For TensorRT-LLM, run the identical load generator against the trtllm-serve OpenAI-compatible endpoint. Standardising the test harness across all engines ensures parity. As vLLM's official documentation highlights, metric definitions vary significantly across distinct benchmarking packages (benchmark CLI documentation).

Follow three rules to ensure reliable data:

• Replay authentic traces: Avoid synthetic pseudo-random token streams. Random strings suppress prefix cache hits and distort memory allocators. Collect 500 to 1,000 anonymised production requests and replay them sequentially.
• Warm up before recording: Execute 60 seconds of unrecorded traffic to prime memory allocations, CUDA graphs, and caching trees. Record three subsequent five-minute test passes and record median values.
• Sweep concurrency bounds: Test load at 0.5x, 1x, and 2x expected operational concurrency. Engine performance degrades along non-linear curves as KV cache memory saturates.

The operational costs nobody puts in the slide deck

Engine bake-offs frequently disregard platform operational overhead. Factor these practical engineering costs into your architectural plan.

TensorRT-LLM compilation overhead

Historically, compiled TensorRT-LLM engines required substantial compilation time: approximately 28 minutes for Llama 3.3 70B FP8 on a single H100 (5 minutes for weight quantisation and 23 minutes for engine graph generation). Building artefacts across multiple GPU counts and quantisation profiles easily demands 1 to 5 hours of engineering effort before serving the first request. Spheron's cold-start benchmarks recorded 28 minutes to readiness for compiled paths, compared to approximately 60 seconds for vLLM and SGLang. For Kubernetes clusters autoscaling from zero, compiled engines introduce unacceptable pod startup delays.

The legacy build path has been removed

Platform engineers must note a major architectural change: NVIDIA's TensorRT backend removal guide confirms that the legacy TensorRT engine backend has been removed. PyTorch is now the sole execution backend. Legacy utilities including trtllm-build, trtllm-refit, trtllm-prune, and standalone convert_checkpoint.py scripts have been deprecated and retired.

Current versions of TensorRT-LLM load Hugging Face checkpoints directly through the PyTorch runtime backend. This transition eliminates prolonged compilation delays, but historical automation scripts and CI/CD pipelines calling trtllm-build will break on modern releases. Any architectural assessment must be verified against current PyTorch-backend documentation.

Silent performance regressions during upgrades

High-velocity open-source engines release frequent updates. A minor version bump in vLLM or SGLang can introduce double-digit shifts in p99 latency or alter default memory allocators. Engineering teams should pin container image tags strictly in production and run automated benchmark verification scripts in staging environments prior to promoting updates.

Memory preallocation trade-offs

Serving engines allocate KV cache pools upon process startup. Engineering teams that proactively calibrate parameters such as gpu-memory-utilization in vLLM and --mem-fraction-static in SGLang consistently recover 20 to 30% in throughput. Conversely, improper configurations trigger out-of-memory crashes during unexpected context spikes. Memory settings must be tuned systematically per model checkpoint and context length.

Multi-replica tail latency

In scaled Kubernetes deployments, tail latency is frequently dictated by cluster ingress and pod scheduling rather than kernel execution. As demonstrated in the Red Hat MLPerf submission, integrating the llm-d scheduler with queue-aware routing compressed p99 TTFT from 3.4 seconds down to 2.1 seconds purely through intelligent dispatching based on KV cache state and pod queue depth.

Keep the serving engine replaceable

Because all three engines expose OpenAI-compatible REST endpoints at /v1/chat/completions, treating the inference engine as an immutable, tightly coupled dependency is an architectural anti-pattern. The engine should function as a swappable component sitting behind an enterprise routing gateway.

This decoupling allows organisations to operate a hybrid strategy:

• Deploy vLLM for internal innovation sandboxes and long-tail departmental models.
• Deploy SGLang for customer-facing RAG pipelines and multi-turn autonomous agents.
• Deploy TensorRT-LLM for mission-critical core models bound by rigid latency service level agreements.

Adopting this architecture requires three capabilities at the gateway layer: unified OpenAI-compatible routing across both self-hosted nodes and external providers, fine-grained access control, and dynamic usage quota management to protect GPU resources from runaway query loops.

At Azrty, this architectural pattern is standard practice. We deploy FastLLM Proxy as an OpenAI-compatible gateway in front of self-hosted LLM clusters and hosted APIs, managing routing, user access, and team budgets from a single plane. When an enterprise requires an end-to-end platform to orchestrate agents, models, and GPU hardware as a managed internal service, we implement Kuvryn AI. For organisations planning their cluster architecture, our team provides comprehensive AI infrastructure engineering. Azrty is a UAE company based in Dubai. Our own team in the UAE does the consulting, the deployment, and the support for our clients across the region.

What to do next

1. Quantify your prefix overlap: Deploy SGLang with --enable-cache-report or monitor vLLM's prefix cache hit metrics against an active trace. If shared prefixes exceed 60%, prioritise SGLang.
2. Formalise concrete latency SLOs: Define target thresholds explicitly (for example: "p99 TTFT under 250 ms at 64 concurrent requests"). Without explicit figures, engine trade-offs remain ambiguous.
3. Execute the one-day benchmark: Deploy your candidate model on target hardware, sweep concurrency levels, and calculate unit economics using the cost formula provided above.
4. Audit operational friction: Include model onboarding time, cold-start latency, memory tuning, and container upgrade maintenance within your total cost of ownership.
5. Decouple applications with an inference gateway: Shield upstream applications behind an abstraction layer. When your inference engine sits behind a unified API gateway, changing serving engines requires a routing adjustment rather than an application refactoring.

Serving engines will continue to iterate rapidly. Maintaining a measurement-driven evaluation process and decoupling your infrastructure ensures your platform captures performance advances without incurring technical debt.

https://www.azrty.com/blog/vllm-vs-sglang-vs-tensorrt-llm-how-to-choose-a-self-hosted-inference-engine-in-2

<a href="https://www.azrty.com/blog/vllm-vs-sglang-vs-tensorrt-llm-how-to-choose-a-self-hosted-inference-engine-in-2">vLLM vs SGLang vs TensorRT-LLM: how to choose a self-hosted inference engine in 2026</a> (Azrty)