FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

Evaluating the factuality of long-form text generated by large language models (LMs) is nontrivial because (1) generations often contain a mixture of supported and unsupported pieces of information, making binary judgments of quality inadequate, and (2) human evaluation is time-consuming and costly. In this paper, we introduce FACTSCORE, a new evaluation that breaks a generation into a series of atomic facts and computes the percentage of atomic facts supported by a reliable knowledge source. We conduct an extensive human evaluation to obtain FACTSCOREs of people biographies generated by several state-of-the-art commercial LMs-InstructGPT, ChatGPT, and the retrievalaugmented PerplexityAI-and report new analysis demonstrating the need for such a finegrained score (e.g., ChatGPT only achieves 58%). Since human evaluation is costly, we also introduce an automated model that estimates FACTSCORE using retrieval and a strong language model, with less than a 2% error rate. Finally, we use this automated metric to evaluate 6,500 generations from a new set of 13 recent LMs that would have cost $26K if evaluated by humans, with various findings: GPT-4 and ChatGPT are more factual than public models, and Vicuna and Alpaca are some of the best public models. FACTSCORE is available for public use via pip install factscore. 1 † Core contributors. 1 Source code and guidelines are available at https:// github.com/shmsw25/FActScore . -Bridget Moynahan is American. -Bridget Moynahan is an actress. -Bridget Moynahan is a model. -Bridget Moynahan is a producer. -She is best known for her roles in Grey's Anatomy. -She is best known for her roles in I, Robot. -She is best known for her roles in Blue Bloods. -She studied acting. -She studied at the American Academy of Dramatic Arts. -… Tell me a bio of Bridget Moynahan. Bridget Moynahan is an American actress, model and producer. She is best known for her roles in Grey's Anatomy, I, Robot and Blue Bloods. She studied acting at the American Academy of Dramatic Arts, and … 66.7% Bridget Moynahan is an American filmmaker and writer. She is best known for her work on the soap opera General Hospital, which she co-created with husband Charles Kelly. Moynahan was raised in a middle-class family in Los Angeles, … -Bridget Moynahan is American. -Bridget Moynahan is a filmmaker. -Bridget Moynahan is a writer. -She is best known for her work on General Hospital. -General Hospital is the soap opera. -She co-created General Hospital. -She co-created General Hospital with her husband. -Her husband is Charles Kelly. -Moynahan was raised in a middle-class family. -Moynahan was raised in Los Angeles. -… 10.0% Stable LM