InferLog: Accelerating LLM Inference for Online Log Parsing via ICL-oriented Prefix Caching

Modern software systems generate massive volumes of runtime logs, necessitating efficient and accurate log parsing to enable critical downstream tasks such as anomaly detection and root cause analysis. Recently, large language models (LLMs) have achieved advanced accuracy on log parsing, but their deployment in production environments faces two major limitations: First, the privacy risks associated with commercial LLMs, driving the adoption of local deployment. Second, online log parsing poses stringent challenges for latency and throughput. While recent methods reduce the number of LLM queries, they overlook the inherent overhead of LLM inference where concurrent log parsing requests can lead to performance degradation in the LLM inference system.

In this study, we present InferLog, the first LLM inference optimization method for online log parsing. Our key insight is that the inference efficiency emerges as the vital bottleneck in LLMbased online log parsing, rather than parsing accuracy. InferLog accelerates inference by designing (i) A prefix-aware ICL refinement strategy to refine the examples and permutation of in-context learning to improve the prefix caching efficiency. (ii) A rapid and task-specific configuration tuning pipeline based on meta-learning to find the optimal LLM inference system configuration. The experimental results based on Loghub-2k dataset and vLLM demonstrate that InferLog significantly outperforms existing inference optimization methods and markedly accelerates the state-of-the-art LLM-based log parsers without compromising parsing accuracy.