Lune

ACL Findings2025

Memory or Reasoning? Explore How LLMs Compute Mixed Arithmetic Expressions

Chengzhi Li, Heyan Huang, Ping Jian, Zhen Yang, Chenxu Wang, Yifan Wang

Abstract

Large language models (LLMs) can solve complex multi-step math reasoning problems, but little is known about how these computations are implemented internally. Many recent studies have investigated the mechanisms of LLMs on simple arithmetic tasks (e.g., 𝑎 + 𝑏, 𝑎 × 𝑏), but how LLMs solve mixed arithmetic tasks still remains unexplored. This gap highlights the limitation of these findings in reflecting realworld scenarios. In this work, we take a step further to explore how LLMs compute mixed arithmetic expressions. We find that LLMs follow a similar workflow to mixed arithmetic calculations: first parsing the complete expression, then using attention heads to aggregate information to the last token position for result generation, without step-by-step reasoning at the token dimension. However, for some specific expressions, the model generates the final result depends on the generation of intermediate results at the last token position, which is similar to human thinking. Furthermore, we propose a Causal Effect Driven Fine-tuning method (CEDF) to adaptively enhance the identified key components used to execute mixed arithmetic calculations to improve LLMs' reasoning ability.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models cites Causal Abstractions of Neural Networks
  • Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models cites How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model
  • How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model cites Causal Abstractions of Neural Networks
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on14

  1. Causal Abstractions of Neural NetworksAtticus Geiger, Hanson Lu, Thomas Icard, Christopher PottsNeurIPS 2021 · 516 citations
  2. Physics of Language Models: Part 3.1, Knowledge Storage and ExtractionZeyuan Allen-Zhu, Yuanzhi LiICML 2024 · 258 citations
  3. How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelMichael Hanna, Ollie Liu, Alexandre VariengienNeurIPS 2023 · 251 citations
  4. Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language ModelsAsma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon et al.ICML 2024 · 197 citations
  5. A Toy Model of Universality: Reverse Engineering how Networks Learn Group OperationsBilal Chughtai, Lawrence Chan, Neel NandaICML 2023 · 144 citations

Related papers

  1. Interpreting and Improving Large Language Models in Arithmetic CalculationWei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu-ming Cheung et al.ICML 2024 · 47 citations
  2. A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation AnalysisAlessandro Stolfo, Yonatan Belinkov, Mrinmaya SachanEMNLP 2023 · 11 citations
  3. How do autoregressive transformers solve full addition?Wang Peixu, Chen Yu, Yu Ming, Cheng XiangEMNLP 2025
  4. Interpreting Arithmetic Reasoning in Large Language Models using Game-Theoretic InteractionsLeilei Wen, Liwei Zheng, Hongda Li, Lijun Sun et al.NeurIPS 2025 · 1 citation
  5. Can LLMs Reason Abstractly Over Math Word Problems Without CoT? Disentangling Abstract Formulation From Arithmetic ComputationZiling Cheng, Meng Cao, Leila Pishdad, Yanshuai Cao et al.EMNLP 2025 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines