Monotonic Scaffolding as a Diagnostic Lens for Legal Reasoning in LLMs

Modern evaluation of Legal QA systems is shifting from terminal accuracy toward processaware analyses of model reasoning. We propose a diagnostic framework grounded in monotonic pedagogical scaffolding, where language models receive gold-standard, caserelevant information across stages aligned with the canonical legal framework FIRAC -Facts, Issue, Rules, Application, Conclusion. By strictly adding solution-relevant content at each step, we introduce a controlled monotonic intervention that allows for the evaluation of reasoning trajectories rather than isolated outcomes. This longitudinal design enables the introduction of two transition-based diagnostics: Errorsto-Success (E2S) quantifies the guidance required to reach correctness, while Success-to-Errors (S2E) measures the fragility of that correctness under additional structure. These local patterns define a global robustness criterion termed Stable Accuracy, which credits a response only if the model maintains correctness throughout all scaffolding stages and enforces a higher bar for correctness by distinguishing sustained reasoning from transient patterns. We instantiate the framework on 3,123 Brazilian Bar Exam questions paired with expertannotated explanations. Our findings reveal model instability patterns hidden from accuracy-only metrics and demonstrate that terminal accuracy systematically overestimates legal reasoning competence. To test the robustness of our diagnostics, we also evaluate a majority-vote aggregation across multiple reasoning samples, finding that the observed instability patterns persist under this stronger inference setting. Furthermore, principal component analysis indicates that legal domains cluster into distinct regions, suggesting systematic differences in reasoning demands across domains. While focused on the legal domain, our evaluation protocol is generalizable to any task with a staged reasoning structure. Input Legal question (Tax Law) João and José inherited a residential property located in the municipality of Alfa from their parents. In January 2017, with authorization from José (a minor), his brother and guardian João (an adult), signed a rental contract for the property with Joaquim as the sole landlord. The contract was for a fixed term of three (3) years and included an express clause stating that the tenant would be solely responsible for paying all taxes and fees related to the rented property, thereby exempting the landlord from such obligations. In December 2021, João and José were surprised by a tax enforcement action filed against both of them by the municipality of Alfa for the collection of property tax (IPTU) for the entire fiscal year of 2018. Given this scenario and in light of the National Tax Code, the tax enforcement action a) could only have been filed against Joaquim. b) could only have been filed against João. c) was correctly filed, since João and José are responsible. d) could not have been filed because the tax credit was time-barred.