ACL Findings2025
Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing
Abstract
The growing use of large language models (LLMs) for text generation has led to widespread concerns about AI-generated content detection. However, an overlooked challenge is AI-polished text, where human-written content undergoes subtle refinements using AI tools. This raises a critical question: should minimally polished text be classified as AIgenerated? Such classification can lead to false plagiarism accusations and misleading claims about AI prevalence in online content. In this study, we systematically evaluate twelve state-of-the-art AI-text detectors using our AI-Polished-Text Evaluation (APT-Eval) dataset, which contains 15K samples refined at varying AI-involvement levels. Our findings reveal that detectors frequently flag even minimally polished text as AI-generated, struggle to differentiate between degrees of AI involvement, and exhibit biases against older and smaller models. These limitations highlight the urgent need for more nuanced detection methodologies.
github.com/ShoumikSaha/ai-polished-text huggingface.co/datasets/smksaha/apt-eval
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection cites RADAR: Robust AI-Text Detection via Adversarial Learning
- CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection cites Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection cites Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
- CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection cites Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature cites Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- Frankentext: Stitching random text fragments into long-form narratives cites Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- Frankentext: Stitching random text fragments into long-form narratives cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- Frankentext: Stitching random text fragments into long-form narratives cites Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning cites RADAR: Robust AI-Text Detection via Adversarial Learning
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning cites Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning cites Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning cites Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
- EditLens: Quantifying the Extent of AI Editing in Text cites Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- EditLens: Quantifying the Extent of AI Editing in Text cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- EditLens: Quantifying the Extent of AI Editing in Text cites Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
- EditLens: Quantifying the Extent of AI Editing in Text cites Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
- Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable cites RADAR: Robust AI-Text Detection via Adversarial Learning
- Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable cites EditLens: Quantifying the Extent of AI Editing in Text
- Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable cites Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
- Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not Enforceable cites Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
- Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text cites RADAR: Robust AI-Text Detection via Adversarial Learning
- Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text cites Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text cites Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature
- Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
Cited by top-tier papers6
- EditLens: Quantifying the Extent of AI Editing in TextKatherine Thai, Bradley Emi, Elyas Masrour, Mohit IyyerICLR 2026 · 30 citations
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation LearningYongxin He, Shan Zhang, Yixuan Cao, Lei Ma et al.NeurIPS 2025 · 13 citations
- Frankentext: Stitching random text fragments into long-form narrativesChau Minh Pham, Jenna Russell, Dzung Pham, Mohit IyyerACL 2026 · 7 citations
- Policies Permitting LLM Use for Polishing Peer Reviews Are Currently Not EnforceableRounak Saha, Gurusha Juneja, Dayita Chaudhuri, Naveeja Sajeevan et al.ICML 2026 · 3 citations
- CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review DetectionYihan Chen, Jiawei Chen, Guozhao Mo, Xuanang Chen et al.ACL 2026 · 1 citation
Builds on8
- DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability CurvatureEric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning et al.ICML 2023 · 988 citations
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defenseKalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting et al.NeurIPS 2023 · 657 citations
- RADAR: Robust AI-Text Detection via Adversarial LearningXiaomeng Hu, Pin-Yu Chen, Tsung-Yi HoNeurIPS 2023 · 315 citations
- Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability CurvatureGuangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang et al.ICLR 2024 · 311 citations
- Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated TextAbhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova, Hamid Kazemi et al.ICML 2024 · 262 citations
Related papers
- Navigating the Shadows: Unveiling Effective Disturbances for Modern AI Content DetectorsYing Zhou, Ben He, Le SunACL 2024
- Benchmarking the Detection of LLMs-Generated Modern Chinese PoetryShanshan Wang, Junchao Wu, Fengying Ye, Derek F. Wong et al.EMNLP Findings 2025 · 1 citation
- An Empirical Study to Evaluate AIGC Detectors on Code ContentJian Wang, Shangqing Liu, Xiaofei Xie, Yi LiASE 2024 · 4 citations
- HACo-Det: A Study Towards Fine-Grained Machine-Generated Text Detection under Human-AI CoauthoringZhixiong Su, Yichen Wang, Herun Wan, Zhaohan Zhang et al.ACL 2025 · 10 citations
- Multi-level Style Preference Optimization: An Adaptive Detection Framework for Human-Machine Hybrid TextZehao Wang, Lianwei Wu, Wenbo An, Hang Zhang et al.AAAI 2026
