QUARTZ: Quantile-Aware Routing and Queueing for TTFT SLOs in LLM Serving

Large Language Model (LLM) serving systems increasingly face strict time-to-first-token (TTFT) service-level objectives (SLOs), yet TTFT remains highly sensitive to router-side queueing effects. Prefill costs scale with prompt length, decode lengths are uncertain, and prefix locality creates strong performance skew across requests. Despite major advances in continuous batching and KV-cache management, today's routers are often agnostic to request size, which makes them vulnerable to head-of-line blocking and tail-latency amplification under mixed workloads. We propose QUARTZ, a quantile-aware routing and queueing layer for LLM serving that predicts servicetime quantiles, rather than point estimates, using lightweight, router-visible features such as prompt length, cache-hit signals, and decoding parameters. QUARTZ uses these quantiles to route each request to the worker that minimizes predicted tail completion, and to prioritize admission and queueing decisions to satisfy TTFT SLOs while preserving fairness. We implement QUARTZ as a router upgrade for SGLang and evaluate it on representative interactive and retrieval-augmented workloads. The results show consistent reductions in TTFT tail latency and SLO violations across heterogeneous workloads.