Lune

EMNLP Findings2024

PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion

Zekai Zhang, Yiduo Guo, Yaobo Liang, Dongyan Zhao, Nan Duan

Abstract

The growing dependence on Large Language Models (LLMs) for completing user instructions necessitates a comprehensive understanding of their robustness in real-world situations.To address this need, we introduce the Power-Point Task Completion-Robustness (PPTC-R) benchmark, designed to evaluate LLMs' robustness to user task instructions and different software versions (PowerPoint versions).Specifically, we create adversarial user instructions by manipulating instructions at the sentence, semantic, and language levels.To assess robustness across software versions, we vary the number of available APIs to simulate both the latest and earlier version environments.We benchmark 3 closed-source and 4 open-source LLMs against these robustness settings to evaluate how variations affect their API calls for task completion.Our findings reveal that GPT-4 demonstrates the highest performance and strong robustness, especially in version updates and multilingual settings.However, all LLMs show a significant decline in robustness when faced with multiple simultaneous challenges (e.g., multi-turn interactions), resulting in notable performance drops.We further analyze the robustness behavior and error patterns of LLMs in our benchmark, providing valuable insights for researchers to understand LLM robustness in task completion and to develop more resilient LLMs and agents.The code and data are available at https: //github.com/ZekaiGalaxy/PPTCR. * We put the new APIs in the supplementary

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AgentBench: Evaluating LLMs as Agents cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
  • AutoPresent: Designing Structured Visuals from Scratch cites WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. Large Language Model-driven Meta-structure Discovery in Heterogeneous Information NetworkLin Chen, Fengli Xu, Nian Li, Zhenyu Han et al.KDD 2024 · 12 citations
  2. AutoPresent: Designing Structured Visuals from ScratchJiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou, Yi-Hao Peng et al.CVPR 2025

Builds on10

  1. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming ZhangNeurIPS 2023 · 2,317 citations
  2. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language AgentsShunyu Yao, Howard Chen, John Yang, Karthik NarasimhanNeurIPS 2022 · 1,477 citations
  3. WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsCan Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng et al.ICLR 2024 · 1,206 citations
  4. AgentBench: Evaluating LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al.ICLR 2024 · 748 citations
  5. Adversarial NLI: A New Benchmark for Natural Language UnderstandingYixin Nie, Adina Williams, Emily Dinan, Mohit Bansal et al.ACL 2020 · 602 citations

Related papers

  1. PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task CompletionYiduo Guo, Zekai Zhang, Yaobo Liang, Dongyan Zhao et al.ACL Findings 2024 · 6 citations
  2. AppBench: Planning of Multiple APIs from Various APPs for Complex User InstructionHongru Wang, Rui Wang, Boyang Xue, Heming Xia et al.EMNLP 2024 · 2 citations
  3. ResiliBench: Evaluating Agentic Workflow Adaptation in Stochastic EnvironmentsRuicheng Ao, Zeping Min, Tingyu Zhu, Wotao Yin et al.ICLR 2026
  4. When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task DescriptionsMaya Larbi, Amal Akli, Mike Papadakis, Rihab Bouyousfi et al.ICSE 2026
  5. ShortcutsBench: A Large-Scale Real-world Benchmark for API-based AgentsHaiyang Shen, Yue Li, Desong Meng, Dongqi Cai et al.ICLR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines