Lune

ICLR2025Top-tier venue

ReCogLab: a framework testing relational reasoning & cognitive hypotheses on LLMs

Andrew Liu, Henry Prior, Gargi Balasubramaniam, Rivka Moroshko, Amir Zait, Ilia Labzovsky, Danny Karmon, Ishita Dasgupta, Kim Stachenfeld, Kenneth Marino

Abstract

A fundamental part of human cognition is the ability to not only recall previous memories, but also reason across them to draw conclusions. In cognitive science and psychology, this is termed relational reasoning and a number of effects and biases have been observed in human cognition. Designing experiments to measure these reasoning effects is effortful, and does not transfer easily to analyzing language model reasoning patterns. To make exploring language models on relational reasoning easier, we introduce ReCogLab -a generative framework for constructing reasoning examples. Unlike static datasets, our framework has a number of benefits that help us in our goal of flexible evaluation of LLMs. First, our framework allows us to control the difficulty and context-length of the problem, allowing us to scale with model capability and evaluate LLMs at a variety of scales. Second, the ability to change the configuration of a dataset dynamically allows us to probe models on different aspects and capabilities. Finally, the flexibility of our approach enables the recreation of classic cognitive science experiments and the systematic study of relational reasoning biases in language models. We demonstrate several such experiments and present our findings on a wide variety of open and closed-source language models. We release all data and code at https://github.com/google-deepmind/recoglab.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Evaluating Relational Reasoning in LLMs with REL cites Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Evaluating Relational Reasoning in LLMs with RELLukas Fesser, Yasha Ektefaie, Ada Fang, Sham Kakade et al.ICML 2026

Builds on8

  1. Solving Quantitative Reasoning Problems with Language ModelsAitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer et al.NeurIPS 2022 · 2,039 citations
  2. Reasoning on Graphs: Faithful and Interpretable Large Language Model ReasoningLinhao Luo, Yuan-Fang Li, Gholamreza Haffari, Shirui PanICLR 2024 · 499 citations
  3. The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"Lukas Berglund, Meg Tong, Maximilian Kaufmann, Mikita Balesni et al.ICLR 2024 · 462 citations
  4. Capturing Failures of Large Language Models via Human Cognitive BiasesErik Jones, Jacob SteinhardtNeurIPS 2022 · 154 citations
  5. LongBench: A Bilingual, Multitask Benchmark for Long Context UnderstandingYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu et al.ACL 2024 · 94 citations

Related papers

  1. Pre-training Language Models for Comparative ReasoningMengxia Yu, Zhihan Zhang, Wenhao Yu, Meng JiangEMNLP 2023 · 1 citation
  2. Microstructures and Accuracy of Graph Recall by Large Language ModelsYanbang Wang, Hejie Cui, Jon M. KleinbergNeurIPS 2024 · 3 citations
  3. Counterfactual-based Cognitive Alignment In-Context Learning for Relation ExtractionQibin Li, Shengyuan Bai, Nai Zhou, Nianmin YaoAAAI 2026
  4. RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning EvaluationXinnuo Xu, Rachel Lawrence, Kshitij Dubey, Atharva Pandey et al.ICML 2025
  5. MME-Reasoning: A Broad-Spectrum Benchmark for Evaluating Logical Reasoning in MLLMsJiakang Yuan, Tianshuo Peng, Yilei Jiang, Yiting Lu et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines