IncreQueryFusion: On-demand Data Fusion Framework in Dynamic Data Lakes

In data lakes, efficient data fusion is essential yet challenging in dynamic scenarios due to heterogeneous multi-source data. Batch-based methods ensure high accuracy but suffer from high latency, while emerging on-demand fusion approaches, despite offering real-time responsiveness, lack native support for incremental scenarios and often struggle with query evidence constraints. To overcome these issues, we propose IncreQueryFusion, an end-to-end on-demand fusion framework tailored for dynamic data lakes. First, it introduces a Temporal Index Evidence Retrieval module to incrementally retrieve and selectively expand evidence for subsequent fusion. Then, an adaptive aligned hierarchical fusion mechanism organizes data values and performs evidence-aware truth inference. Finally, a dynamic fusion truth maintenance algorithm refreshes fusion results efficiently without full recomputation. A comprehensive evaluation of established benchmarks and our newly constructed dynamic datasets demonstrates the effectiveness of IncreQueryFusion. It outperforms the state-of-the-art baseline in accuracy by 5%-30% on static workloads, while achieving speedups of over 10× against on-demand methods and over 100× against batch methods in dynamic scenarios.