Lune

EMNLP Findings2025

GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations

Odysseas S. Chlapanis, Dimitris Galanis, Nikolaos Aletras, Ion Androutsopoulos

Abstract

We introduce GREEKBARBENCH, a benchmark that evaluates LLMs on legal questions across five different legal areas from the Greek Bar exams, requiring citations to statutory articles and case facts. To tackle the challenges of free-text evaluation, we propose a threedimensional scoring system combined with an LLM-as-a-judge approach. We also develop a meta-evaluation benchmark to assess the correlation between LLM-judges and human expert evaluations, revealing that simple, span-based rubrics improve their alignment. Our extensive evaluation of 13 proprietary and open-weight LLMs shows that even though the top models exhibit impressive performance, they remain susceptible to critical errors, most notably a failure to identify the correct statutory articles.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Pub-LawBench: Public-Oriented Benchmarking for LegalAI cites LawBench: Benchmarking Legal Knowledge of Large Language Models
  • Monotonic Scaffolding as a Diagnostic Lens for Legal Reasoning in LLMs cites LawBench: Benchmarking Legal Knowledge of Large Language Models
  • LEXam: Benchmarking Legal Reasoning on 340 Law Exams cites LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
  • LEXam: Benchmarking Legal Reasoning on 340 Law Exams cites Efficient Memory Management for Large Language Model Serving with PagedAttention
  • LEXam: Benchmarking Legal Reasoning on 340 Law Exams cites LawBench: Benchmarking Legal Knowledge of Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Pub-LawBench: Public-Oriented Benchmarking for LegalAIQiaoyu Zheng, Zehan Ma, Yijing Zhang, Qiqi Wang et al.ACL 2026
  2. Monotonic Scaffolding as a Diagnostic Lens for Legal Reasoning in LLMsPedro Calais, Janderson Santos, Anísio Lacerda, Wagner Meira Jr.ACL 2026

Builds on12

  1. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  2. Dense Passage Retrieval for Open-Domain Question AnsweringVladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis et al.EMNLP 2020 · 142 citations
  3. LawBench: Benchmarking Legal Knowledge of Large Language ModelsZhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou et al.EMNLP 2024 · 59 citations
  4. LEXam: Benchmarking Legal Reasoning on 340 Law ExamsYu Fan, Jingwei Ni, Jakob Merane, Yang Tian et al.ICLR 2026 · 56 citations
  5. LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal DomainJoel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi et al.EMNLP Findings 2023 · 50 citations

Related papers

  1. CitaLaw: Enhancing LLM with Citations in Legal DomainKepu Zhang, Weijie Yu, Sunhao Dai, Jun XuACL Findings 2025 · 6 citations
  2. PLAWBENCH: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal PracticeYuzhen Shi, Huanghai Liu, Yiran Hu, Gaojie Song et al.ACL 2026 · 7 citations
  3. SedarEval: Automated Evaluation using Self-Adaptive RubricsZhiyuan Fan, Weinong Wang, Xing Wu, Debing ZhangEMNLP Findings 2024 · 9 citations
  4. GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in GreekYang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis et al.ACL Findings 2026 · 1 citation
  5. LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language TextsHelia Hashemi, Jason Eisner, Corby Rosset, Benjamin Van Durme et al.ACL 2024 · 27 citations
Dusk over the sea between two cliffs drawn in fine vertical lines