SRAG: A Lightweight and Specialized Retrieval-augmented Generation System at the Edge

Retrieval-augmented generation (RAG) has shown strong potential for deploying large language models at the edge, yet existing designs largely rely on generic and monolithic knowledge bases that are poorly matched to the heterogeneous queries and resource-constrained edge computing environments. Through extensive empirical analysis, we find that domain-specialized knowledge bases, when deployed on individual edge servers, deliver substantially higher retrieval accuracy and generation quality than generic knowledge bases under identical resource budgets. Based on this, we propose SRAG, a distributed RAG system that enforces knowledge specialization at the edge. Each edge server maintains a domain-aware specialized knowledge base by retaining domain-aligned knowledge and decoupling out-of-domain content. SRAG uses a buffer-based knowledge migration mechanism to redistribute out-of-domain content to better-matched edge servers, enabling efficient global knowledge utilization without central coordination. To handle domain-mismatched queries, SRAG employs lightweight cross-node routing guided by compact metadata summaries, avoiding full knowledge replication. Together, these mechanisms form an end-to-end workflow for decentralized edge RAG. Experiments show that SRAG improves retrieval relevance, generation quality, and storage efficiency, while reducing end-to-end latency.