A Law Reasoning Benchmark for LLM with Tree-Organized Structures including Factum Probandum, Evidence and Experiences

While progress has been made in legal applications, law reasoning, crucial for fair adjudication, remains unexplored. We propose a transparent law reasoning schema enriched with hierarchical factum probandum, evidence, and implicit experience, enabling public scrutiny and preventing bias. Inspired by this schema, we introduce the challenging task, which takes a textual case description and outputs a hierarchical structure justifying the final decision. We also create the first crowd-sourced dataset for this task, enabling comprehensive evaluation. Simultaneously, we propose an agent framework that employs a comprehensive suite of legal analysis tools to address the challenge task. This benchmark paves the way for transparent and accountable AI-assisted law reasoning in the "Intelligent Court" 1 . * Wenjuan Han is the corresponding author Case Description: it is necessary, as shortly as possible, to review some of the facts of this essentially commonplace and unedifying case. The appellant, Edith Jessie Thompson, is twenty-nine years of age. She is the daughter of a Mr. Graydon, and seven years ago she married Mr. Percy Thompson, the man now dead, the only person who in this case excites any sympathy… Input Case Description: it is necessary, as shortly as possible, to review some of the facts of this essentially commonplace and unedifying case. The appellant, Edith Jessie Thompson, is twenty-nine years of age. She is the daughter of a Mr. Graydon, and seven years ago she married Mr. Percy Thompson, the man now dead, the only person who in this case excites any sympathy… Input Interim Factum Probandum: The appellant, Edith Jessie Thompson, is twenty-nine years of age. She is the daughter of a Mr. Graydon, and seven years ago she married Mr. Percy Thompson, the man now dead, the only person who in this case excites any sympathy. Location in Description： [15, 16] Output