Is Human-Like Text Liked by Humans? Multilingual Human Detection and Preference Against AI

Prior studies have shown that distinguishing text generated by Large Language Models (LLMs) from human-written one is highly challenging for humans, and often no better than random guessing. To verify the generalizability of this finding across languages and domains, we perform an extensive case study to identify the upper bound of human detection accuracy. Across 16 datasets covering 9 languages and 9 domains, 19 annotators achieved an average detection accuracy of 87.6%, thus challenging previous conclusions. We find that major gaps between human and machine text lie in concreteness, cultural nuances, and diversity. Prompting by explicitly explaining the distinctions in the prompts can partially bridge the gaps in over 50% of the cases. However, we also find that humans do not always prefer human-written text, particularly when they cannot clearly identify its source. We release our dataset, the human labels, and the annotator metadata at https://github.com/ xnlp-lab/HumanEval-MGT . Setting ID Input Task Description Outputs Applicable Scenarios I. Single-Binary hwt or mgt Given a piece of text, identify whether it is written by human? A. Yes, human; B. No, machine parallel data is not necessary. II. Pair-Binary (hwt, mgt) or (mgt, hwt) Given a pair of (text1, text2), identify which one is humanwritten? Either text1 or text2 must be hwt, and another is mgt randomly sampled from mgti. A. text1; B. text2 parallel data is available. III. Triplet-Three-Class (hwt, mgt1, mgt2) Given a set of texts (text1, text2, text3), identify which one is human-written? One of the text1, text2 and text3 must be hwt, and others are mgt randomly sampled from mgti. A. text1; B. text2; C. text3 parallel human and multiple LLM generations are collected to make comparisons. IV. Pair-Four-Class (hwt, mgt) or (mgt, hwt) or (hwt, hwt) or (mgt, mgt) Given a pair of (text1, text2), identify which one is humanwritten? Both text1 and text2 can be hwt, and can be mgt. A. text1; B. text2; C. none; D. both parallel data is not necessary.