LLM-as-Scheduler: Agentic Workflow Dynamic Scheduling

As large language models (LLMs) become more capable, many applications are shifting from a single LLM call to multi-agent systems. Manually designed or automatically optimized workflows often include multiple verification and testing stages. These stages can improve accuracy but also introduce substantial latency and increased token consumption. We find that many requests do not require such heavyweight processing and are solvable by a single strong agent. To address this inefficiency, we propose LLM-as-Scheduler (LAS), a system that dynamically routes each query through a workflow. LAS uses a two-stage cascade: a lightweight gate that quickly checks each agent's output, and an LLM-based scheduler that makes fine-grained routing decisions using query features and gate signals. Experiments show that LAS reduces token usage by 50.5% and end-to-end latency by over 36% on average, with at most a 1.4 percentage-point drop in accuracy compared with a strong fixed workflow. The code will be public at https://gi thub.com/YoshuaDavy/LLM-as-Scheduler