LLM Systems
Multi-Tenant Serving and Isolation
Serving many tenants from one model is cheap and easy; giving each tenant their own fine-tune is expensive and hard. S-LoRA and per-request LoRA serving collapse the trade-off, but only for tenants who can share a base model.
advanced · 10 min read · Premium
Phase 1 of any LLM product: one model, one prompt template, one API key, every tenant gets the same behaviour. Cheap, simple, the right starting point. Phase 2: enterprise customers demand their own behaviour - their tone, their vocabulary, their refusal rules, their internal knowledge. You have three options - prompt-engineer per tenant (cheap, low ceiling), fine-tune a model per tenant (expensive, high ceiling), or serve thousands of LoRA adapters on top of one shared base model (the middle path that S-LoRA and vLLM made practical). The choice is fundamentally about where you put the isolation boundary.
The simple case - shared model, prompt-level isolation
One base model. Every tenant's request goes through the same vLLM instance. Tenant identity is carried in the prompt (system message: "You are an assistant for ACME Corp") and in metadata (for routing, rate limits, audit).
| Property | Shared-model multi-tenancy |
|---|---|
| Throughput | Maximum - one model fully utilised |
| Per-tenant cost | Minimum - amortised across the whole fleet |
| Per-tenant customisation | Prompt only |
| Data isolation | Logical (in-prompt), not cryptographic |
| Noisy neighbour risk | High - one tenant's huge prompt blocks decode for others |
| Cold-start latency | None - model is always warm |
This works for 80% of multi-tenant products and almost all SaaS chat applications. Make it your default and only move off it when a specific tenant pays you enough to justify it.
Per-tenant LoRA adapters
LoRA (Hu et al. 2021) fine-tunes a model by training low-rank update matrices A and B such that the new weight is W' = W + alpha * B @ A where A and B are tiny (rank 8-64 instead of full d-by-d). A LoRA adapter for a 7B base model might be 20-100 MB instead of 14 GB for the full model.
The serving question: can you swap LoRA adapters per request without reloading the base model? Naively, no - you would have to merge the LoRA into the base weights, which takes seconds. With dedicated infrastructure, yes - you keep the base weights in HBM, keep N LoRA adapters in HBM (or paged from CPU), and dispatch each request through the appropriate adapter at the matmul level.
S-LoRA - the paper that made it work at scale
S-LoRA (Sheng et al., arXiv:2311.03285) demonstrated serving thousands of LoRA adapters on a single GPU with small overhead. The core contributions:
- Unified paging. Both the KV cache and the LoRA adapter weights live in one unified memory pool, allocated in fixed-size blocks. This solves the fragmentation that destroys naive LoRA serving when adapters vary in rank.
- Heterogeneous batching. A single batch can mix requests against different LoRA adapters. Custom CUDA kernels apply the right adapter per request without breaking batched matmul.
- Tensor-parallel sharding of both base weights and adapter weights across GPUs.
Keep reading with Pro.
You're reading the preview. Unlock the full concept plus the library, study plans, the AI mentor, and daily emails.