LogDelta: Differential Encoding for Log Data

System logs are critical for understanding the performance of various software systems. Compression is essential to efficient storage or transmission of system log data. Generalpurpose compressors like LZMA treat logs as binary data, ignoring their structure, and thus fail to achieve the best compression. The semi-structured nature of log data leads to parser-based methods, which however require extra space for storing templates. We observe that although log data exhibit high similarity, the repetitions are extremely scattered. Rather than storing the vast positions or complicated templates for short repetitions, we propose to merge them as one large difference between log records. Inspired by the differential encoding for numerical data, we propose LogDelta, a novel differential encoding method for log data. The compression task is formulated as an optimization problem to minimize the cost of differential operations from one log record to another. By replaying such differential operations, each log record can be reconstructed in decompression. While a dynamic programming algorithm is developed, we further propose a linear time approximation, delivering comparable compression ratio as the optimal solution. Experimental results on real world system logs demonstrate that LogDelta achieves superior compression ratios while maintaining comparable compression speeds.