Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise: A Case Study on Chinese Legal Domain

While large language models (LLMs) like GPT-4 have recently demonstrated astonishing zeroshot capabilities in general domain tasks, they often generate content with hallucinations in specific domains such as Chinese law, hindering their application in these areas. This is typically due to the absence of training data that encompasses such a specific domain, preventing GPT-4 from acquiring in-domain knowledge. A pressing challenge is that it's not plausible to continue training LLMs of the GPT-4's scale on in-domain data. This paper introduces a simple yet effective domain adaptation framework for GPT-4 by reformulating generation as an adapt-retrieverevise process. The initial step is to adapt an affordable 7B LLM to the Chinese legal domain by continuing learning in-domain data. When solving an in-domain task, we leverage the adapted LLM to generate a draft answer given a task query. Then, the draft answer will be used to retrieve supporting evidence candidates from an external in-domain knowledge base. Finally, the draft answer and retrieved evidence are concatenated into a whole prompt to let GPT-4 assess the evidence and revise the draft answer to generate the final answer. Our proposal combines the advantages of the efficiency of adapting a smaller 7B model with the evidence-assessing capability of GPT-4 and effectively prevents GPT-4 from generating hallucinatory content. In the zero-shot setting of four Chinese legal tasks, our method improves the average score by +33.6 points, compared to GPT-4 direct generation. When compared to two stronger retrieval-based baselines, our method outperforms them by +17.0 and +23.5. The code for training our model is here: 1 . 12.0 33.1 71.1 41.9 GPT-4 Direct Generation GPT-4 Retrieval-based Generation Domain-Adapted 7B LLM Ours Query: What are the penalties for parking violations under the traffic code? Answer: According to [article 93] clause index of [the Road Traffic Safety Law of the People's Republic of China] law name, [to anyone who violates the road traffic safety laws or regulations on parking or temporarily parks motor vehicles, his illegal acts may be pointed out, meanwhile he shall be imposed upon the penalty of oral warning, and be ordered to drive away immediately...... ] rationale Figure 1: Left: A real translated example of Chinese LegalQA. The square brackets and subscripts are offered for the purpose of clear demonstration, not actually exist in the ground-truth answer or generation. Right: Models' F1 scores on the LegalQA dataset.