Evaluating Factuality in Cross-lingual Summarization

Cross-lingual summarization aims to help people efficiently grasp the core idea of the document written in a foreign language. Modern text summarization models generate highly fluent but often factually inconsistent outputs, which has received heightened attention in recent research. However, the factual consistency of cross-lingual summarization has not been investigated yet. In this paper, we propose a crosslingual factuality dataset by collecting human annotations of reference summaries as well as generated summaries from models at both summary level and sentence level. Furthermore, we perform the fine-grained analysis and observe that over 50% of generated summaries and over 27% of reference summaries contain factual errors with characteristics different from monolingual summarization. Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of crosslingual summarization and perform differently at different tasks and levels. Finally, we adapt the monolingual factuality metrics as an initial step towards the automatic evaluation of summarization factuality in cross-lingual settings. Our dataset and code are available at https: //github.com/kite99520/Fact_CLS . * Equal contribution. by the General Administration of Customs on the 10th show that the total value of China's imports and exports in February was 260.43 billion U.S. dollars, up by 29.4%. Among them, the export was U.S. dollars 114.47 billion, up by 18.4%; Imports reached 145.96 billion U.S. dollars, up by 39.6 %. The trade deficit of 31.49 billion U.S. dollars was the largest in nearly a decade. ) Summaries TNCLS: China's exports exceeded 100 billion US dollars in February, the biggest trade deficit in nearly 10 years. CLSMS: China's trade deficit in the past 10 years is the largest in nearly 10 years. CLSMT: In February, the total import and export value of China's foreign trade increased by 29.4 % compared with the same period last year. ATS: China's foreign trade deficit in February was 26.4 billion US dollars, the biggest in 10 years.