Optimizing Deeper Transformers on Small Datasets
Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J. D. Prince, Yanshuai Cao
Abstract
It is a common belief that training deep transformers from scratch requires large datasets. Consequently, for small datasets, people usually use shallow and simple additional layers on top of pre-trained models during finetuning. This work shows that this does not always need to be the case: with proper initialization and optimization, the benefits of very deep transformers can carry over to challenging tasks with small datasets, including Textto-SQL semantic parsing and logical reading comprehension. In particular, we successfully train 48 layers of transformers, comprising 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers trained from scratch. With fewer training steps and no task-specific pre-training, we obtain the state-of-the-art performance on the challenging cross-domain Text-to-SQL parsing benchmark Spider 1 . We achieve this by deriving a novel Data-dependent Transformer Fixedupdate initialization scheme (DT-Fixup), inspired by the prior T-Fixup work (Huang et al., 2020) . Further error analysis shows that increasing depth can help improve generalization on small datasets for hard cases that require reasoning and structural understanding.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
10 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL cites RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
- RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQL cites SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites On the Variance of the Adaptive Learning Rate and Beyond
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models cites Understanding the Difficulty of Training Transformers
- SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQL cites RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
- Few-shot Text-to-SQL Translation using Structure and Content Prompt Learning cites RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers
- Understanding the Difficulty of Training Transformers cites On the Variance of the Adaptive Learning Rate and Beyond
- Improving Transformer Optimization Through Better Initialization cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Improving Transformer Optimization Through Better Initialization cites On the Variance of the Adaptive Learning Rate and Beyond
Cited by top-tier papers10
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language ModelsZhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen et al.NeurIPS 2023 · 1,003 citations
- RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SQLHaoyang Li, Jing Zhang, Cuiping Li, Hong ChenAAAI 2023 · 343 citations
- SADGA: Structure-Aware Dual Graph Aggregation Network for Text-to-SQLRuichu Cai, Jinjie Yuan, Boyan Xu, Zhifeng HaoNeurIPS 2021 · 90 citations
- Few-shot Text-to-SQL Translation using Structure and Content Prompt LearningZihui Gu, Ju Fan, Nan Tang, Lei Cao et al.SIGMOD 2023 · 60 citations
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is WastefulMartin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson et al.NeurIPS 2025 · 46 citations
Builds on5
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- On the Variance of the Adaptive Learning Rate and BeyondLiyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen et al.ICLR 2020 · 2,210 citations
- Improving Transformer Optimization Through Better InitializationXiao Shi Huang, Felipe Pérez, Jimmy Ba, Maksims VolkovsICML 2020 · 181 citations
- Understanding the Difficulty of Training TransformersLiyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen et al.EMNLP 2020 · 158 citations
- RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL ParsersBailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov et al.ACL 2020 · 39 citations
Related papers
- Graphix-T5: Mixing Pre-trained Transformers with Graph-Aware Layers for Text-to-SQL ParsingJinyang Li, Binyuan Hui, Reynold Cheng, Bowen Qin et al.AAAI 2023 · 164 citations
- Laziness Is a Virtue When It Comes to Compositionality in Neural Semantic ParsingMaxwell Crouse, Pavan Kapanipathi, Subhajit Chaudhury, Tahira Naseem et al.ACL 2023 · 1 citation
- Rose-SQL: Role-State Evolution Guided Structured Reasoning for Multi-Turn Text-to-SQLLe Zhou, Feng Yao, Fengcai Qiao, Bo Xu et al.ACL 2026
- Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic ParsingXi Victoria Lin, Richard Socher, Caiming XiongEMNLP Findings 2020 · 186 citations
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular DataPengcheng Yin, Graham Neubig, Wen-tau Yih, Sebastian RiedelACL 2020 · 417 citations
