An LLM Agentic Approach for Legal-Critical Software: A Case Study for Tax Prep Software

As Large Language Models (LLMs) continue to advance in their ability to process both natural and programming languages, they show promise for translation tasks in domains with strict compliance requirements. Yet ensuring consistency in legally critical settings remains challenging due to inherent limitations such as natural language ambiguity and the tendency to hallucinate. This paper explores an agentic approach that leverages LLMs for the development of legal-critical software. We use U.S. federal tax software as a representative case study, where natural language tax code must be translated precisely into executable logic.

A central challenge in developing legal-critical software from specifications lies in test case generation, which suffers from the oracle problem: determining the correct output for a given scenario often requires interpreting legal statutes. Prior work has proposed metamorphic testing as a solution by evaluating equivalence across similarly situated individuals. A key innovation of our work is a higher-order generalization of metamorphic tests, motivated by our tax preparation case study, in which system outputs are compared across structured shifts among similar individuals. Since manually generating such higher-order relations is tedious and error-prone, our agentic paradigm is well suited to automate test case generation.

We design and implement Synedrion, an assembly of LLMbased agents simulating roles in real-world software development teams handling legal documents. The framework includes a metamorphic testing agent that produces counterexamples while translating tax code into executable software. Our findings indicate that Synedrion, when employing smaller language models (e.g., GPT-4o-mini), can outperform frontier models (e.g., GPT-4o and Claude-3.5) in complex tax code generation tasks, achieving a worst-case pass rate of 45% compared with 9%-15%. We thus make the case for LLM-driven agentic methodologies as a pathway for generating robust, trustworthy legal-critical software from natural language specifications.