SLGParser: Practical and Efficient Label-Free Log Parsing Using Large Language Models

Logs serve as crucial artifacts for recording runtime system states and play a vital role in system diagnostics, security analysis, and performance optimization. Log parsing, the process of converting semi-structured logs into structured formats, is a pivotal preliminary step for enabling automated log analysis and utilization. Recently, semantic-based parsers leverage labeled data, which is labor-intensive. Unsupervised LLM-based log parsers perform comparative analysis on a set of logs that share similar static text, and we refer to this set of logs as a log group. Through a comprehensive analysis of unsupervised methods, we identify three limitations. 1) Validity: existing parsers fail to adequately capture the variable parts of log entries in selecting log groups. 2) Generalizability: the absence of log groups often leads to suboptimal performance when relying solely on groupbased reasoning strategy. 3) Practicality: handling large volumes of log entries incurs substantial computational overhead and poses a great challenge in parsing efficiency. To overcome above limitations, we introduce SLGParser, a universal and efficient unsupervised log parsing approach. Specifically, we propose a Suitable Log Group (SLG) selector that maximizes variable differences among logs to enhance the validity of SLGParser. Furthermore, we develop a hint-enhanced strategy to address the performance degradation caused by the absence of log groups. Additionally, we employ an efficient Log Matcher mechanism to reduce LLM query and time overhead. SLGParser reveals several significant inaccuracies in the manually labeled ground truth of the Loghub-2.0 benchmark. To validate the effectiveness of SLGParser, we perform extensive experiments on the revised datasets. The experimental results demonstrate that SLGParser significantly outperforms state-of-the-art log parsers in terms of accuracy and efficiency, providing an efficient and practical solution for real-world deployment.