Lune

ACL Findings2026

Generalization or Memorization? Multi-Agent vs. Baseline LLMs and AutoML Models for Tabular Classification

Aida Sanatizadeh, Sorouralsadat Fatemi, Reza Mousavi, Ahmed Abbasi

Abstract

Large Language Models (LLMs) are increasingly used for structured tabular data, yet it remains unclear whether their performance reflects genuine reasoning or memorization of pre-training corpora. We investigate this question through a rigorous, contaminationaware evaluation of a representative modular Multi-Agent LLM (MALLM) framework against state-of-the-art AutoML systems and established baselines (TABLET, TABLLM). We evaluate eleven binary classification tasks: five pre-cutoff benchmarks likely seen during LLM pre-training and six post-cutoff datasets released after the LLM knowledge cutoff. Results show a sharp performance dichotomy: MALLM achieves competitive or superior performance on pre-cutoff datasets but substantially underperforms AutoML on post-cutoff data, exhibiting poor calibration and high variance, especially on hard-to-classify instances. By contrast, AutoML models generalize consistently and align confidence more closely with instance hardness. These findings suggest that, despite agentic scaffolding, current LLMs cannot yet replace production-grade discriminative models for tabular classification, underscoring the need for contamination-free benchmarks to accurately assess tabular reasoning capabilities.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Related papers

  1. TabularMath: Understanding Math Reasoning over Tables with Large Language ModelsShi-Yu Tian, Zhi Zhou, Wei Dong, Kun-Yang Yu et al.ACL Findings 2026
  2. The Illusion of Generalization in Tabular Language ModelsAditya Gorla, Ratish Surendran PuduppullyICML 2026 · 2 citations
  3. Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMDingjie Song, Sicheng Lai, Mingxuan Wang, Shunian Chen et al.EMNLP Findings 2025 · 1 citation
  4. TABARD: A Novel Benchmark for Tabular Anomaly Analysis, Reasoning and DetectionManan Roy Choudhury, Anirudh Iyengar Kaniyar Narayana Iyengar, Shikhhar Siingh, Sugeeth Puranam et al.EMNLP Findings 2025
  5. How Far Can LLM Agents Reason with Tables? Benchmarking Multi-Turn Agentic Table Question Answering in the WildJingwang Huang, Jie Zhang, Haoyang Zeng, Changzai Pan et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines