Chain-of-Thought Reasoning in Tabular Language Models

Tabular mathematical reasoning task requires models to perform multi-step operations including information look-up and numerical calculations, based on heterogeneous data from tables and questions. Existing solutions tend to extend chain-of-thought (CoT) reasoning into powerful large language models (LLMs) to promote multi-hop mathematical reasoning. However, it can be extremely difficult to apply such LLMbased approaches under scenarios of privatization deployment or limited resources. To address this problem, we revisit small-scale tabular language models (TaLMs) and extend chainof-thought reasoning into TaLMs for the first time. Specifically, we propose a novel framework, TaCo, which coordinates two TaLMs responsible for CoT generation and answer inference, respectively. Besides, our framework can be combined with an external calculator to enhance accurate numerical calculations. On the TABMWP dataset, TaCo outperforms the state-of-the-art ChatGPT by 9.55% (82.60%→92.15% in accuracy) with much less parameters (0.8B). 1