Can LLMs Reason Soundly in Law? Auditing Inference Patterns for Legal Judgment

This paper presents a method to analyze the inference patterns used by Large Language Models (LLMs) for judgment in a case study on legal LLMs, so as to identify potential incorrect representations of the LLM, according to human domain knowledge. Unlike traditional evaluations on language generation results, we propose to evaluate the correctness of the detailed inference patterns of an LLM behind its seemingly correct outputs. To this end, we quantify the interactions between input phrases used by the LLM as primitive inference patterns, because recent theoretical achievements (26; 42) have proven several mathematical guarantees of the faithfulness of the interaction-based explanation. We design a set of metrics to evaluate the detailed inference patterns of LLMs. Experiments show that even when the language generation results appear correct, a significant portion of the inference patterns used by the LLM for the legal judgment may represent misleading or irrelevant logic 1 . * Quanshi Zhang is the corresponding author. He is with the Department of Computer Science and Engineering, the John Hopcroft Center, at the Shanghai Jiao Tong University, China. 1 The names used in the legal cases follow an alphabetical convention, e.g., Andy, Bob, Charlie, etc., which do not represent any bias against actual individuals. On the (A) morning of December 22, 2013, the defendants Andy and Bob deceived Charlie and the three of them (B) had an argument. Andy (C) chased Charlie (D) with an axe and (E) bit Charlie, causing Charlie to be (F) slightly injured. Bob (G) hit Charlie (H) with a shovel, (I) injuring Charlie and causing Charlie's (J) death. Irrelevant phrases Relevant phrases Forbidden phrases Input legal case 𝒙 LLM's output 𝒗("𝐴𝑠𝑠𝑎𝑢𝑙𝑡"|𝒙) Input legal case 𝒙 Equivalently modeling Rel iab le inte rac tion effe cts Un re lia bl e in te ra ct io n ef fe ct s 𝑅 !