๐ Production LLM Deployment & Quantization
vLLM High-Throughput Engine & PagedAttention
PagedAttention, Continuous Batching, and OpenAI-compatible API serving โ production vLLM from scratch
Standard Transformer inference allocates KV Cache in contiguous GPU memory blocks, wasting 60โ80% on internal and external fragmentation. vLLM's PagedAttention solves this with OS-paging-inspired non-contiguous block allocation โ achieving 14ร higher throughput on identical hardware. Combined with Continuous Batching, a single A100 can serve 200+ concurrent users at GPT-3.5 quality.
Key Concepts
PagedAttention: OS Paging for KV Cache
KV Cache is split into fixed-size blocks (e.g., 16 tokens/block). A block table maps logical sequence positions to physical VRAM pages โ identical to OS virtual memory paging. Fragmentation drops from 60% to under 4%.
Continuous Batching vs Static Batching
Static batching waits for ALL sequences in a batch to finish before starting the next batch โ leaving GPU idle for short sequences. Continuous batching inserts new requests the moment any sequence completes, keeping GPU utilization near 100%.
Tensor Parallelism & Pipeline Parallelism
Tensor Parallelism splits individual weight matrices across GPUs (e.g., Attention heads distributed across 4 GPUs). Pipeline Parallelism splits model layers across GPUs. vLLM supports both via --tensor-parallel-size and --pipeline-parallel-size.
Speculative Decoding (Draft Model)
A tiny draft model (e.g., 68M params) proposes k tokens in parallel. The large target model verifies all k tokens in a single forward pass. Accepted tokens are free โ dramatically reducing target model calls by 2โ3ร.
PagedAttention: Memory Allocation Diagram
Throughput & Latency Benchmarks
| Serving System | Throughput (req/s) | P99 Latency | GPU Util. | Notes |
|---|---|---|---|---|
| HuggingFace generate() | ~3โ5 | ~8,000ms | ~40% | Static batching, max_length pre-allocated |
| HF Text Generation Inference (TGI) | ~18โ25 | ~2,000ms | ~65% | Continuous batching, Flash Attention |
| vLLM (PagedAttention) | ~60โ85 | ~500ms | ~92% | PagedAttention + Continuous Batching |
| vLLM + Speculative Decoding | ~140โ200 | ~280ms | ~95% | Draft model reduces target model calls 2โ3ร |
Production Code 1: OpenAI-Compatible API Server
start_vllm_server.shbash
1# โโ Launch vLLM as OpenAI-compatible REST API server โโโโโโโโโโโโโโโโโโโโโ
2# Drop-in replacement for OpenAI API โ change base_url, keep same client code
3
4python -m vllm.entrypoints.openai.api_server \
5 --model meta-llama/Meta-Llama-3-70B-Instruct \
6 --tensor-parallel-size 4 \ # 4 ร A100 GPUs for 70B model
7 --pipeline-parallel-size 1 \
8 --gpu-memory-utilization 0.92 \ # 92% VRAM for KV cache paging
9 --max-model-len 8192 \ # Max context window
10 --max-num-seqs 256 \ # Max concurrent sequences in KV cache
11 --enable-prefix-caching \ # Cache common system prompt prefixes
12 --enable-chunked-prefill \ # Prevent long prefill from blocking decode
13 --speculative-model meta-llama/Llama-3.2-1B-Instruct \ # Draft model for speculative decoding
14 --num-speculative-tokens 5 \ # k=5 tokens proposed per draft step
15 --host 0.0.0.0 \
16 --port 8000
17
18# โโ Verify server health โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
19curl http://localhost:8000/health
20# {"status":"ok"}
21
22curl http://localhost:8000/v1/models
23# Lists available model endpoints
24Production Code 2: Async Streaming Client (FastAPI Gateway)
vllm_async_streaming_gateway.pypython
1import asyncio
2from openai import AsyncOpenAI
3from fastapi import FastAPI
4from fastapi.responses import StreamingResponse
5import json
6
7app = FastAPI()
8
9# โโ vLLM server is OpenAI API-compatible โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
10client = AsyncOpenAI(
11 base_url="http://localhost:8000/v1",
12 api_key="EMPTY" # vLLM doesn't require real API key
13)
14
15async def stream_llm_response(user_prompt: str, system_prompt: str):
16 """Stream tokens as Server-Sent Events (SSE) directly from vLLM."""
17 stream = await client.chat.completions.create(
18 model="meta-llama/Meta-Llama-3-70B-Instruct",
19 messages=[
20 {"role": "system", "content": system_prompt},
21 {"role": "user", "content": user_prompt}
22 ],
23 temperature=0.7,
24 max_tokens=1024,
25 stream=True, # Enable token-by-token streaming
26 extra_body={
27 "repetition_penalty": 1.05,
28 "min_tokens": 10,
29 }
30 )
31
32 # Stream each token chunk as SSE
33 async for chunk in stream:
34 delta = chunk.choices[0].delta.content
35 if delta:
36 yield f"data: {json.dumps({'token': delta})}\n\n"
37
38 yield "data: [DONE]\n\n"
39
40@app.post("/chat/stream")
41async def chat_endpoint(prompt: str, system: str = "You are a helpful AI assistant."):
42 return StreamingResponse(
43 stream_llm_response(prompt, system),
44 media_type="text/event-stream",
45 headers={
46 "Cache-Control": "no-cache",
47 "X-Accel-Buffering": "no", # Disable nginx buffering for real-time stream
48 }
49 )
50
51# โโ Production Usage โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
52# uvicorn vllm_async_streaming_gateway:app --host 0.0.0.0 --port 9000 --workers 4
53Production Code 3: Kubernetes Deployment for vLLM
vllm-k8s-deployment.yamlyaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: vllm-server
5 namespace: ai-serving
6spec:
7 replicas: 2 # 2 replicas for HA
8 selector:
9 matchLabels:
10 app: vllm-server
11 template:
12 metadata:
13 labels:
14 app: vllm-server
15 spec:
16 containers:
17 - name: vllm
18 image: vllm/vllm-openai:latest
19 command:
20 - python
21 - -m
22 - vllm.entrypoints.openai.api_server
23 args:
24 - "--model"
25 - "meta-llama/Meta-Llama-3-8B-Instruct"
26 - "--tensor-parallel-size"
27 - "1"
28 - "--gpu-memory-utilization"
29 - "0.90"
30 - "--max-num-seqs"
31 - "128"
32 - "--enable-prefix-caching"
33 - "--port"
34 - "8000"
35 resources:
36 limits:
37 nvidia.com/gpu: 1 # 1ร A100 per pod
38 memory: "120Gi"
39 requests:
40 nvidia.com/gpu: 1
41 memory: "80Gi"
42 ports:
43 - containerPort: 8000
44 livenessProbe:
45 httpGet:
46 path: /health
47 port: 8000
48 initialDelaySeconds: 120 # Model loading takes 60-90s
49 periodSeconds: 30
50 env:
51 - name: HUGGING_FACE_HUB_TOKEN
52 valueFrom:
53 secretKeyRef:
54 name: hf-token
55 key: token
56---
57apiVersion: v1
58kind: Service
59metadata:
60 name: vllm-service
61 namespace: ai-serving
62spec:
63 selector:
64 app: vllm-server
65 ports:
66 - protocol: TCP
67 port: 80
68 targetPort: 8000
69 type: ClusterIP
70---
71# Horizontal scaling based on GPU utilization metric
72apiVersion: autoscaling/v2
73kind: HorizontalPodAutoscaler
74metadata:
75 name: vllm-hpa
76 namespace: ai-serving
77spec:
78 scaleTargetRef:
79 apiVersion: apps/v1
80 kind: Deployment
81 name: vllm-server
82 minReplicas: 2
83 maxReplicas: 8
84 metrics:
85 - type: Resource
86 resource:
87 name: memory
88 target:
89 type: Utilization
90 averageUtilization: 80
91Production Code 4: Offline Batch Throughput Benchmark
vllm_benchmark.pypython
1"""
2vLLM offline throughput benchmark โ measure tokens/second before deployment.
3Compare against HuggingFace baseline to verify vLLM speedup.
4"""
5import time
6import torch
7from vllm import LLM, SamplingParams
8
9def run_vllm_benchmark(model_path: str, num_prompts: int = 200, max_tokens: int = 256):
10 sampling_params = SamplingParams(
11 temperature=0.0, # Greedy for deterministic benchmark
12 max_tokens=max_tokens,
13 ignore_eos=True # Always generate full max_tokens
14 )
15
16 llm = LLM(
17 model=model_path,
18 tensor_parallel_size=torch.cuda.device_count(),
19 gpu_memory_utilization=0.90,
20 enforce_eager=False, # Allow CUDA graph capture for faster decode
21 enable_prefix_caching=True,
22 )
23
24 # Synthetic prompts with varying lengths (simulates real traffic distribution)
25 import random
26 words = ["system", "design", "architecture", "inference", "optimization", "latency"]
27 prompts = [
28 " ".join(random.choices(words, k=random.randint(10, 50)))
29 for _ in range(num_prompts)
30 ]
31
32 # Warmup pass
33 llm.generate(prompts[:5], sampling_params)
34
35 start = time.perf_counter()
36 outputs = llm.generate(prompts, sampling_params)
37 elapsed = time.perf_counter() - start
38
39 total_tokens = sum(len(o.outputs[0].token_ids) for o in outputs)
40 throughput = total_tokens / elapsed
41
42 print(f"{'='*50}")
43 print(f"Model: {model_path}")
44 print(f"Prompts: {num_prompts}")
45 print(f"Max tokens: {max_tokens}")
46 print(f"Total elapsed: {elapsed:.2f}s")
47 print(f"Total tokens out: {total_tokens:,}")
48 print(f"Throughput: {throughput:.1f} tokens/sec")
49 print(f"Req/sec: {num_prompts/elapsed:.1f}")
50 print(f"{'='*50}")
51 return throughput
52
53if __name__ == "__main__":
54 run_vllm_benchmark("meta-llama/Meta-Llama-3-8B-Instruct")
55๐ก
Senior Architect Insight: The single most impactful vLLM production config is --enable-prefix-caching. If all your requests share a system prompt (common in chatbots), prefix caching reuses the KV cache for that prefix across ALL requests โ eliminating prefill cost entirely for the shared portion and cutting first-token latency by 50โ80% on long system prompts.