sglang

Use SGLang when:

• Need structured outputs (JSON, regex, grammar)
• Building agents with repeated prefixes (system prompts, tools)
• Agentic workflows with function calling
• Multi-turn conversations with shared context
• Need faster JSON decoding (3× vs standard)

Use vLLM instead when:

• Simple text generation without structure
• Don't need prefix caching
• Want mature, widely-tested production system

Use TensorRT-LLM instead when:

• Maximum single-request latency (no batching needed)
• NVIDIA-only deployment
• Need FP8/INT4 quantization on H100

Installation

```bash

# pip install (recommended)

pip install "sglang[all]"

# With FlashInfer (faster, CUDA 11.8/12.1)

pip install sglang[all] flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/

# From source

git clone https://github.com/sgl-project/sglang.git

cd sglang

pip install -e "python[all]"

```

Launch server

```bash

# Basic server (Llama 3-8B)

python -m sglang.launch_server \

--model-path meta-llama/Meta-Llama-3-8B-Instruct \

--port 30000

# With RadixAttention (automatic prefix caching)

python -m sglang.launch_server \

--model-path meta-llama/Meta-Llama-3-8B-Instruct \

--port 30000 \

--enable-radix-cache # Default: enabled

# Multi-GPU (tensor parallelism)

python -m sglang.launch_server \

--model-path meta-llama/Meta-Llama-3-70B-Instruct \

--tp 4 \

--port 30000

```

Basic inference

```python

import sglang as sgl

# Set backend

sgl.set_default_backend(sgl.OpenAI("http://localhost:30000/v1"))

# Simple generation

@sgl.function

def simple_gen(s, question):

s += "Q: " + question + "\n"

s += "A:" + sgl.gen("answer", max_tokens=100)

# Run

state = simple_gen.run(question="What is the capital of France?")

print(state["answer"])

# Output: "The capital of France is Paris."

```

Structured JSON output

```python

import sglang as sgl

@sgl.function

def extract_person(s, text):

s += f"Extract person information from: {text}\n"

s += "Output JSON:\n"

# Constrained JSON generation

s += sgl.gen(

"json_output",

max_tokens=200,

regex=r'\{"name": "[^"]+", "age": \d+, "occupation": "[^"]+"\}'

)

# Run

state = extract_person.run(

text="John Smith is a 35-year-old software engineer."

)

print(state["json_output"])

# Output: {"name": "John Smith", "age": 35, "occupation": "software engineer"}

```

What it does: Automatically caches and reuses common prefixes across requests.

Performance:

• 5× faster for agentic workloads with shared system prompts
• 10× faster for few-shot prompting with repeated examples
• Zero configuration - works automatically

How it works:

1. Builds radix tree of all processed tokens
2. Automatically detects shared prefixes
3. Reuses KV cache for matching prefixes
4. Only computes new tokens

Example (Agent with system prompt):

```

Request 1: [SYSTEM_PROMPT] + "What's the weather?"

→ Computes full prompt (1000 tokens)

Request 2: [SAME_SYSTEM_PROMPT] + "Book a flight"

→ Reuses system prompt KV cache (998 tokens)

→ Only computes 2 new tokens

→ 5× faster!

```

JSON with schema

```python

@sgl.function

def structured_extraction(s, article):

s += f"Article: {article}\n\n"

s += "Extract key information as JSON:\n"

# JSON schema constraint

schema = {

"type": "object",

"properties": {

"title": {"type": "string"},

"author": {"type": "string"},

"summary": {"type": "string"},

"sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]}

},

"required": ["title", "author", "summary", "sentiment"]

}

s += sgl.gen("info", max_tokens=300, json_schema=schema)

state = structured_extraction.run(article="...")

print(state["info"])

# Output: Valid JSON matching schema

```

Regex-constrained generation

```python

@sgl.function

def extract_email(s, text):

s += f"Extract email from: {text}\n"

s += "Email: "

# Email regex pattern

s += sgl.gen(

"email",

max_tokens=50,

regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}'

)

state = extract_email.run(text="Contact john.doe@example.com for details")

print(state["email"])

# Output: "john.doe@example.com"

```

Grammar-based generation

```python

@sgl.function

def generate_code(s, description):

s += f"Generate Python code for: {description}\n"

s += "```python\n"

# EBNF grammar for Python

python_grammar = """

?start: function_def

function_def: "def" NAME "(" [parameters] "):" suite

parameters: parameter ("," parameter)*

parameter: NAME

suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT

"""

s += sgl.gen("code", max_tokens=200, grammar=python_grammar)

s += "\n```"

```

```python

import sglang as sgl

# Define tools

tools = [

{

"name": "get_weather",

"description": "Get weather for a location",

"parameters": {

"type": "object",

"properties": {

"location": {"type": "string"}

}

}

},

{

"name": "book_flight",

"description": "Book a flight",

"parameters": {

"type": "object",

"properties": {

"from": {"type": "string"},

"to": {"type": "string"},

"date": {"type": "string"}

}

}

}

]

@sgl.function

def agent_workflow(s, user_query, tools):

# System prompt (cached with RadixAttention)

s += "You are a helpful assistant with access to tools.\n"

s += f"Available tools: {tools}\n\n"

# User query

s += f"User: {user_query}\n"

s += "Assistant: "

# Generate with function calling

s += sgl.gen(

"response",

max_tokens=200,

tools=tools, # SGLang handles tool call format

stop=["User:", "\n\n"]

)

# Multiple queries reuse system prompt

state1 = agent_workflow.run(

user_query="What's the weather in NYC?",

tools=tools

)

# First call: Computes full system prompt

state2 = agent_workflow.run(

user_query="Book a flight to LA",

tools=tools

)

# Second call: Reuses system prompt (5× faster)

```

RadixAttention speedup

Few-shot prompting (10 examples in prompt):

• vLLM: 2.5 sec/request
• SGLang: 0.25 sec/request (10× faster)
• Throughput: 4× higher

Agent workflows (1000-token system prompt):

• vLLM: 1.8 sec/request
• SGLang: 0.35 sec/request (5× faster)

JSON decoding:

• Standard: 45 tok/s
• SGLang: 135 tok/s (3× faster)

Throughput (Llama 3-8B, A100)

| Workload | vLLM | SGLang | Speedup |

|----------|------|--------|---------|

| Simple generation | 2500 tok/s | 2800 tok/s | 1.12× |

| Few-shot (10 examples) | 500 tok/s | 5000 tok/s | 10× |

| Agent (tool calls) | 800 tok/s | 4000 tok/s | 5× |

| JSON output | 600 tok/s | 2400 tok/s | 4× |

```python

@sgl.function

def multi_turn_chat(s, history, new_message):

# System prompt (always cached)

s += "You are a helpful AI assistant.\n\n"

# Conversation history (cached as it grows)

for msg in history:

s += f"{msg['role']}: {msg['content']}\n"

# New user message (only new part)

s += f"User: {new_message}\n"

s += "Assistant: "

s += sgl.gen("response", max_tokens=200)

# Turn 1

history = []

state = multi_turn_chat.run(history=history, new_message="Hi there!")

history.append({"role": "User", "content": "Hi there!"})

history.append({"role": "Assistant", "content": state["response"]})

# Turn 2 (reuses Turn 1 KV cache)

state = multi_turn_chat.run(history=history, new_message="What's 2+2?")

# Only computes new message (much faster!)

# Turn 3 (reuses Turn 1 + Turn 2 KV cache)

state = multi_turn_chat.run(history=history, new_message="Tell me a joke")

# Progressively faster as history grows

```

Speculative decoding

```bash

# Launch with draft model (2-3× faster)

python -m sglang.launch_server \

--model-path meta-llama/Meta-Llama-3-70B-Instruct \

--speculative-model meta-llama/Meta-Llama-3-8B-Instruct \

--speculative-num-steps 5

```

Multi-modal (vision models)

```python

@sgl.function

def describe_image(s, image_path):

s += sgl.image(image_path)

s += "Describe this image in detail: "

s += sgl.gen("description", max_tokens=200)

state = describe_image.run(image_path="photo.jpg")

print(state["description"])

```

Batching and parallel requests

```python

# Automatic batching (continuous batching)

states = sgl.run_batch(

[

simple_gen.bind(question="What is AI?"),

simple_gen.bind(question="What is ML?"),

simple_gen.bind(question="What is DL?"),

]

)

# All 3 processed in single batch (efficient)

```

```bash

# Start server with OpenAI API

python -m sglang.launch_server \

--model-path meta-llama/Meta-Llama-3-8B-Instruct \

--port 30000

# Use with OpenAI client

curl http://localhost:30000/v1/chat/completions \

-H "Content-Type: application/json" \

-d '{

"model": "default",

"messages": [

{"role": "system", "content": "You are helpful"},

{"role": "user", "content": "Hello"}

],

"temperature": 0.7,

"max_tokens": 100

}'

# Works with OpenAI Python SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")

response = client.chat.completions.create(

model="default",

messages=[{"role": "user", "content": "Hello"}]

)

```

Text models:

• Llama 2, Llama 3, Llama 3.1, Llama 3.2
• Mistral, Mixtral
• Qwen, Qwen2, QwQ
• DeepSeek-V2, DeepSeek-V3
• Gemma, Phi-3

Vision models:

• LLaVA, LLaVA-OneVision
• Phi-3-Vision
• Qwen2-VL

100+ models from HuggingFace

NVIDIA: A100, H100, L4, T4 (CUDA 11.8+)

AMD: MI300, MI250 (ROCm 6.0+)

Intel: Xeon with GPU (coming soon)

Apple: M1/M2/M3 via MPS (experimental)

• [Structured Generation Guide](references/structured-generation.md) - JSON schemas, regex, grammars, validation
• [RadixAttention Deep Dive](references/radix-attention.md) - How it works, optimization, benchmarks
• [Production Deployment](references/deployment.md) - Multi-GPU, monitoring, autoscaling

• GitHub: https://github.com/sgl-project/sglang
• Docs: https://sgl-project.github.io/
• Paper: RadixAttention (arXiv:2312.07104)
• Discord: https://discord.gg/sglang