BOND: A Co-Designed Framework for LLM-Powered Analytics Over Relational Data

The integration of Large Language Models (LLMs) into database systems through SQL has made the data analytics workflow elegant. However, this integration introduces significant computational overhead, as conventional LLM inference is not optimized for relational workloads. Existing optimization techniques are often designed for unpredictable online request streams and treat the LLM engine as a black box, failing to leverage the prior knowledge of the entire data batch available in database operations. To bridge this gap, we propose BOND, a co-designed framework that deeply integrates batch-aware optimizations on both the inference engine and data organization sides. Specifically, BOND consists of two main components. First, a batch-efficient LLM inference engine indexes incoming prompts using a radix tree to identify shared prefix. Then, it models batch creation as a bin-packing problem, grouping computations to align with GPU hardware characteristics. As a result, it effectively mitigates tile-quantization effects. Second, a prefix-oriented data re-organizer optimizes the relational data before inference. This process uses a tree-based framework to find a data layout that maximizes prefix sharing and cache localities. Finally, these two components are bridged by a bubble-free task scheduler that ensures continuous GPU utilization in distributed environments. Experimental results show that our approach could reduce query execution time by up to 71.8%.