Lune

ACL Findings2026

Investigating Human and LLMs' Decisions in Unverifiable Environments: A Case Study with GitHub Activity Overview

Zheng Jiang, Wei Wang, Gaowei Zhang, Yang Feng, Yi Wang

Abstract

The behaviors of Large Language Models (LLMs) as artificial social actors are largely underexplored, particularly in unverifiable scenarios where conventional benchmarking has little to help improve their abilities. Thus, examining their behaviors in such scenarios can help understand and improve LLMs' capabilities of simulating real-world social actors in many tasks such as LLM-empowered social agents. We draw a typical unverifiable scenario-a simplified pull request scenario on GITHUB focusing on decision-making based on Activity Overview signal-to investigate how human and LLMs behave. We introduce a systematic method to collect, compare, and reason about human and LLMs' decisions. Our results reveal that there are both similarities and differences between human and LLMs' decisions, and proprietary LLMs generally behave more like human than open-source LLMs do. We further find that human and LLMs may rely on different information and reasoning mechanisms in decision-making. Our study thus urges more future work on human and LLMs decision-making in unverifiable environments.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Builds on15

  1. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMsXumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye et al.ICLR 2026 · 279 citations
  2. MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue ResolutionWei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang et al.NeurIPS 2024 · 210 citations
  3. Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM AgentsGiorgio Piatti, Zhijing Jin, Max Kleiman-Weiner, Bernhard Schölkopf et al.NeurIPS 2024 · 151 citations
  4. Don't Trust: Verify - Grounding LLM Quantitative Reasoning with AutoformalizationJin Peng Zhou, Charles Staats, Wenda Li, Christian Szegedy et al.ICLR 2024 · 72 citations
  5. Decision-Making Behavior Evaluation Framework for LLMs under Uncertain ContextJingru Jia, Zehua Yuan, Junhao Pan, Paul McNamara et al.NeurIPS 2024 · 71 citations

Related papers

  1. Evaluating the Moral Beliefs Encoded in LLMsNino Scherrer, Claudia Shi, Amir Feder, David M. BleiNeurIPS 2023 · 316 citations
  2. Awes, Laws, and Flaws From Today's LLM ResearchAdrian de WynterACL Findings 2025 · 1 citation
  3. MetaAgents: Large Language Model Based Agents for Decision-Making on TeamingYuan Li, Lichao Sun, Yixuan ZhangCSCW 2025 · 35 citations
  4. VirtualEnv: A Platform for Embodied AI ResearchKabir Swain, Sijie Han, Ayush Raina, Jin Zhang et al.AAAI 2026
  5. HumanEvo: An Evolution-Aware Benchmark for More Realistic Evaluation of Repository-Level Code GenerationDewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang et al.ICSE 2025 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines