Lune

EMNLP Findings2024

AutoDetect: Towards a Unified Framework for Automated Weakness Detection in Large Language Models

Jiale Cheng, Yida Lu, Xiaotao Gu, Pei Ke, Xiao Liu, Yuxiao Dong, Hongning Wang, Jie Tang, Minlie Huang

Abstract

Although Large Language Models (LLMs) are becoming increasingly powerful, they still exhibit significant but subtle weaknesses, such as mistakes in instruction-following or coding tasks. As these unexpected errors could lead to severe consequences in practical deployments, it is crucial to investigate the limitations within LLMs systematically. Traditional benchmarking approaches cannot thoroughly pinpoint specific model deficiencies, while manual inspections are costly and not scalable. In this paper, we introduce a unified framework, AutoDetect, to automatically expose weaknesses in LLMs across various tasks. Inspired by the educational assessment process that measures students' learning outcomes, AutoDetect consists of three LLM-powered agents: Examiner, Questioner, and Assessor. The collaboration among these three agents is designed to realize comprehensive and in-depth weakness identification. Our framework demonstrates significant success in uncovering flaws, with an identification success rate exceeding 30% in prominent models such as ChatGPT and Claude. More importantly, these identified weaknesses can guide specific model improvements, proving more effective than untargeted data augmentation methods like Self-Instruct. Our approach has led to substantial enhancements in popular LLMs, including the Llama series and Mistral-7b, boosting their performance by over 10% across several benchmarks. Code and data are publicly available at https://github.com/thu-coai/AutoDetect.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference cites Measuring Massive Multitask Language Understanding
  • ProbeLLM: Automating Principled Diagnosis of LLM Failures cites FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language Models
  • ProbeLLM: Automating Principled Diagnosis of LLM Failures cites Measuring Massive Multitask Language Understanding
  • Uncovering Competency Gaps in Large Language Models and Their Benchmarks cites Measuring Massive Multitask Language Understanding
  • SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. ProbeLLM: Automating Principled Diagnosis of LLM FailuresYue Huang, Zhengzhe Jiang, Yuchen Ma, Yu Jiang et al.ICML 2026 · 4 citations
  2. FACT-AUDIT: An Adaptive Multi-Agent Framework for Dynamic Fact-Checking Evaluation of Large Language ModelsHongzhan Lin, Yang Deng, Yuxuan Gu, Wenxuan Zhang et al.ACL 2025
  3. Uncovering Competency Gaps in Large Language Models and Their BenchmarksMaty Bohacek, Nino Scherrer, Nicholas Dufour, Thomas Leung et al.ICML 2026
  4. SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language ModelsJiale Cheng, Xiao Liu, Cunxiang Wang, Xiaotao Gu et al.ICLR 2025

Builds on17

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  3. Chatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceWei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos et al.ICML 2024 · 1,212 citations
  4. Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng et al.SOSP 2023 · 1,016 citations
  5. LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi FengNeurIPS 2024 · 865 citations

Related papers

  1. CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain SettingsDaniil Orel, Dilshod Azizov, Preslav NakovACL Findings 2025 · 5 citations
  2. MdEval: Massively Multilingual Code DebuggingShukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi et al.ACL Findings 2026 · 1 citation
  3. Do Large Language Models Know What They Don't Know?Zhangyue Yin, Qiushi Sun, Qipeng Guo, Jiawen Wu et al.ACL Findings 2023 · 75 citations
  4. Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation BenchmarkDewu Zheng, Yanlin Wang, Ensheng Shi, Xilin Liu et al.ICSE 2026
  5. MGTBench: Benchmarking Machine-Generated Text DetectionXinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes et al.CCS 2024 · 30 citations
Dusk over the sea between two cliffs drawn in fine vertical lines