Lune

ICDE2024Top-tier venue

Gen-T: Table Reclamation in Data Lakes

Grace Fan, Roee Shraga, Renée J. Miller

Abstract

We introduce the problem of Table Reclamation. Given a Source Table and a large table repository, reclamation finds a set of tables that, when integrated, reproduce the source table as closely as possible. Unlike query discovery problems like Query-by-Example or by-Target, Table Reclamation focuses on reclaiming the data in the Source Table as fully as possible using real tables that may be incomplete or inconsistent. To do this, we define a new measure of table similarity, called error-aware instance similarity, to measure how close a reclaimed table is to a Source Table, a measure grounded in instance similarity used in data exchange. Our search covers not only Select-project-join queries, but integration queries with unions, outerjoins, and the unary operators subsumption and complementation that have been shown to be important in data integration and fusion. Using reclamation, a data scientist can understand if any tables in a repository can be used to exactly reclaim a tuple in the Source. If not, one can understand if this is due to differences in values or to incompleteness in the data. Our solution, Gen-T, performs table discovery to retrieve a set of candidate tables from the table repository, filters these down to a set of originating tables, then integrates these tables to reclaim the Source as closely as possible. We show that our solution, while approximate, is accurate, efficient and scalable in the size of the table repository with experiments on real data lakes containing up to 15K tables, where the average number of tuples varies from small (web tables) to extremely large (open data tables) up to 1M tuples.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • How Large Language Models Will Disrupt Data Management cites Deep Entity Matching with Pre-Trained Language Models
  • Table Overlap Estimation through Graph Embeddings cites Finding Related Tables in Data Lakes for Interactive Data Science
  • Table Overlap Estimation through Graph Embeddings cites Creating Embeddings of Heterogeneous Relational Datasets for Data Integration Tasks
  • Table Overlap Estimation through Graph Embeddings cites Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning
  • LakeVisage: Towards Scalable, Flexible and Interactive Visualization Recommendation for Data Discovery over Data Lakes cites Finding Related Tables in Data Lakes for Interactive Data Science
  • LakeVisage: Towards Scalable, Flexible and Interactive Visualization Recommendation for Data Discovery over Data Lakes cites Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning
  • Qualitative Join Discovery in Data Lakes using Examples cites Deep Entity Matching with Pre-Trained Language Models
  • Qualitative Join Discovery in Data Lakes using Examples cites Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning
  • Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning cites Deep Entity Matching with Pre-Trained Language Models
  • Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning cites Finding Related Tables in Data Lakes for Interactive Data Science
  • Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning cites Creating Embeddings of Heterogeneous Relational Datasets for Data Integration Tasks
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Pneuma: Leveraging LLMs for Tabular Data Representation and Retrieval in an End-to-End SystemMuhammad Imam Luthfi Balaka, David Alexander, Qiming Wang, Yue Gong et al.SIGMOD 2025 · 12 citations
  2. Qualitative Join Discovery in Data Lakes using ExamplesMir Mahathir Mohammad, El Kindi RezigSIGMOD 2026 · 6 citations
  3. LakeVisage: Towards Scalable, Flexible and Interactive Visualization Recommendation for Data Discovery over Data LakesYihao Hu, Jin Wang, Sajjadur RahmanVLDB 2025 · 3 citations
  4. Table Overlap Estimation through Graph EmbeddingsFrancesco Pugnaloni, Luca Zecchini, Matteo Paganelli, Matteo Lissandrini et al.SIGMOD 2025 · 1 citation

Builds on17

  1. Deep Entity Matching with Pre-Trained Language ModelsYuliang Li, Jinfeng Li, Yoshihiko Suhara, AnHai Doan et al.VLDB 2021 · 484 citations
  2. Creating Embeddings of Heterogeneous Relational Datasets for Data Integration TasksRiccardo Cappuzzo, Paolo Papotti, Saravanan ThirumuruganathanSIGMOD 2020 · 139 citations
  3. Semantics-aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation LearningGrace Fan, Jin Wang, Yuliang Li, Dan Zhang et al.VLDB 2023 · 139 citations
  4. How Large Language Models Will Disrupt Data ManagementRaul Castro Fernandez, Aaron J. Elmore, Michael J. Franklin, Sanjay Krishnan et al.VLDB 2023 · 127 citations
  5. Finding Related Tables in Data Lakes for Interactive Data ScienceYi Zhang, Zachary G. IvesSIGMOD 2020 · 98 citations

Related papers

  1. Integrating Data Lake TablesAamod Khatiwada, Roee Shraga, Wolfgang Gatterbauer, Renée J. MillerVLDB 2023 · 59 citations
  2. Automatic Table Union Search with Tabular Representation LearningXuming Hu, Shen Wang, Xiao Qin, Chuan Lei et al.ACL Findings 2023 · 14 citations
  3. Dataset Discovery in Data LakesAlex Bogatu, Alvaro A. A. Fernandes, Norman W. Paton, Nikolaos KonstantinouICDE 2020 · 118 citations
  4. Novel Table SearchBesat Kassaie, Renée J. MillerICDE 2026
  5. SANTOS: Relationship-based Semantic Table Union SearchAamod Khatiwada, Grace Fan, Roee Shraga, Zixuan Chen et al.SIGMOD 2023 · 61 citations
Dusk over the sea between two cliffs drawn in fine vertical lines