Lune

ICLR2026Top-tier venue

Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMs

Zhuowen Liang, Xiaotian Lin, Zhengxuan Zhang, Yuyu Luo, Haixun Wang, Nan Tang

Abstract

Large language models (LLMs) are widely applied to data analytics over documents, yet direct reasoning over long, noisy documents remains brittle and error-prone. Hence, we study document question answering (QA) that consolidates dispersed evidence into a structured output (e.g., a table, graph, or chunks) to support reliable, verifiable QA. We propose a two-pillar framework, LiteCoST, to achieve both high accuracy and low latency with small language models (SLMs). Pillar 1: Chain-of-Structured-Thought (CoST). We introduce a CoST template, a schema-aware instruction that guides a strong LLM to produce both a step-wise CoST trace and the corresponding structured output. The process induces a minimal structure, normalizes entities/units, aligns records, serializes the output, and verifies/refines it, yielding auditable supervision. Pillar 2: SLM fine-tuning. The compact models are trained on LLM-generated CoST data in two stages: Supervised Fine-Tuning for structural alignment, followed by Group Relative Policy Optimization (GRPO) incorporating triple rewards for answer/format quality and process consistency. By distilling structure-first behavior into SLMs, this approach achieves LLM-comparable quality on multi-domain long-document QA using 3B/7B SLMs, while delivering 2–4× lower latency than GPT-4o and DeepSeek-R1 (671B). The code is available at https://github.com/HKUSTDial/LiteCoST.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations cites The Dawn of Natural Language to SQL: Are We Fully Ready? [Experiment, Analysis & Benchmark ]
  • VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations cites DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework
  • DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework cites VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
  • Document-to-Database: Extraction Meets Relational Semantics cites VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
  • DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL Framework cites The Dawn of Natural Language to SQL: Are We Fully Ready? [Experiment, Analysis & Benchmark ]
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. VisJudge-Bench: Aesthetics and Quality Assessment of VisualizationsYupeng Xie, Zhiyang Zhang, Yifan Wu, Sirong Lu et al.ICLR 2026 · 23 citations
  2. Document-to-Database: Extraction Meets Relational SemanticsZhengxuan Zhang, Zhuowen Liang, Jiazhuo Chen, Haixun Wang et al.VLDB 2026 · 3 citations

Builds on13

  1. Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni et al.NeurIPS 2020 · 19,162 citations
  2. The Dawn of Natural Language to SQL: Are We Fully Ready? [Experiment, Analysis & Benchmark ]Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li et al.VLDB 2024 · 137 citations
  3. LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations
  4. DeepEye-SQL: A Software-Engineering-Inspired Text-to-SQL FrameworkBoyan Li, Chong Chen, Zhujun Xue, Yinan Mei et al.SIGMOD 2026 · 40 citations
  5. MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical NotesAsma Ben Abacha, Wen-Wai Yim, Yujuan Fu, Zhaoyi Sun et al.ACL Findings 2025 · 29 citations

Related papers

  1. Drilling Down into the Discourse Structure with LLMs for Long Document Question AnsweringInderjeet Nair, Shwetha Somasundaram, Apoorv Saxena, Koustava GoswamiEMNLP Findings 2023 · 10 citations
  2. Small Encoders Can Rival Large Decoders in Detecting GroundednessIstabrak Abbes, Gabriele Prato, Quentin Fournier, Fernando Rodriguez et al.ACL Findings 2025
  3. Replacing Multi-Step Assembly of Data Preparation Pipelines with One-Step LLM Pipeline Generation for Table QAFengyu Li, Junhao Zhu, Kaishi Song, Lu Chen et al.VLDB 2026 · 1 citation
  4. David vs. Goliath: Cost-Efficient Financial QA via Cascaded Multi-Agent ReasoningChenghao Liu, Qian Liu, Ziqin Zhu, Hao Fei et al.EMNLP Findings 2025
  5. Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step EntropyZeju Li, Jianyuan Zhong, Ziyang Zheng, Xiangyu Wen et al.ICLR 2026 · 35 citations
Dusk over the sea between two cliffs drawn in fine vertical lines