EMNLP Findings2024
Counter Turing Test (CT²): Investigating AI-Generated Text Detection for Hindi - Ranking LLMs based on Hindi AI Detectability Index (ADI_hi)
Ishan Kavathekar, Anku Rani, Ashmit Chamoli, Ponnurangam Kumaraguru, Amit P. Sheth, Amitava Das
Abstract
The widespread adoption of Large Language Models (LLMs) and awareness around multilingual LLMs have raised concerns regarding the potential risks and repercussions linked to the misapplication of AI-generated text, necessitating increased vigilance. While these models are primarily trained for English, their extensive training on vast datasets covering almost the entire web, equips them with capabilities to perform well in numerous other languages. AI-Generated Text Detection (AGTD) has emerged as a topic that has already received immediate attention in research, with some initial methods having been proposed, soon followed by the emergence of techniques to bypass detection. In this paper, we report our investigation on AGTD for an indic language Hindi. Our major contributions are in four folds: i) examined 26 LLMs to evaluate their proficiency in generating Hindi text, ii) introducing the AI-generated news article in Hindi (AG hi ) dataset, iii) evaluated the effectiveness of five recently proposed AGTD techniques: ConDA, J-Guard, RADAR, RAIDAR and Intrinsic Dimension Estimation for detecting AI-generated Hindi text, iv) proposed Hindi AI Detectability Index (ADI hi ) which shows a spectrum to understand the evolving landscape of eloquence of AI-generated text in Hindi. The code and dataset is available at https://github.com/ ishank31/Counter_Turing_Test
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense cites DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
- DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature cites Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense cites A Watermark for Large Language Models
- DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature cites A Watermark for Large Language Models
- A Watermark for Large Language Models cites Crosslingual Generalization through Multitask Finetuning
Builds on9
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability CurvatureEric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning et al.ICML 2023 · 988 citations
- A Watermark for Large Language ModelsJohn Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz et al.ICML 2023 · 854 citations
- Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defenseKalpesh Krishna, Yixiao Song, Marzena Karpinska, John Wieting et al.NeurIPS 2023 · 657 citations
- Crosslingual Generalization through Multitask FinetuningNiklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts et al.ACL 2023 · 319 citations
Related papers
- RADAR: Robust AI-Text Detection via Adversarial LearningXiaomeng Hu, Pin-Yu Chen, Tsung-Yi HoNeurIPS 2023 · 315 citations
- Raidar: geneRative AI Detection viA RewritingChengzhi Mao, Carl Vondrick, Hao Wang, Junfeng YangICLR 2024 · 66 citations
- MAGE: Machine-generated Text Detection in the WildYafu Li, Qintong Li, Leyang Cui, Wei Bi et al.ACL 2024 · 44 citations
- Profiler: Black-box AI-generated Text Origin Detection via Context-aware Inference Pattern AnalysisHanxi Guo, Siyuan Cheng, Xiaolong Jin, Zhuo Zhang et al.EMNLP 2025
- C-ReD: A Comprehensive Chinese Benchmark for AI-Generated Text Detection Derived from Real-World PromptsChenxi Qing, Junxi Wu, Zheng Liu, Yixiang Qiu et al.ACL Findings 2026
