Lune

ACL Findings2026

Large Language Models for IT Automation Tasks: Are We There Yet?

Md. Mahadi Hassan, John Salvador, Akond Ashfaque Ur Rahman, Santu Karmaker

Abstract

LLMs show promise in code generation, yet their effectiveness for IT automation tasks, particularly for tools like Ansible, remains understudied. Existing benchmarks rely primarily on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools. We present ExITBench (Executionbased IT Automation Benchmark), a benchmark of 126 diverse tasks (e.g., configuring servers and managing files) in which each task captures state reconciliation-a core property of IT automation tools. ExITBench evaluates LLMs' ability to generate functional Ansible automation scripts via dynamic execution in controlled environments. We evaluate 14 open-source and 3 proprietary LLMs and find that GPT-4.1-Mini achieves the best pass@10 rate of 23.9%, while Claude-3.5-Sonnet achieves the best pass@1 performance. To explain the low performance, we analyze 1,517 execution failures across the evaluated LLMs and identify two prevalent semantic error categories: failures in state-reconciliation reasoning (42.117% combined from variable (12.287%), host (10.363%), path (10.511%), and template (8.956%) issues) and deficiencies in module-specific execution knowledge (26.203% combined from attribute & parameter (17.617%) and module (8.586%) errors). Our findings reveal key limitations in LLMs' ability to address state reconciliation and apply specialized module knowledge, indicating that reliable IT automation with LLM-based agents need major advances in state reasoning and domain-specific execution.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models cites DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
  • CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models cites WizardCoder: Empowering Code Large Language Models with Evol-Instruct
  • SWE-bench: Can Language Models Resolve Real-world Github Issues? cites CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models
  • WizardCoder: Empowering Code Large Language Models with Evol-Instruct cites DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Unfulfilled Promises: LLM-Based Detection of OS Compatibility Issues in Infrastructure as CodeGeorgios-Petros Drosos, Georgios Alexopoulos, Thodoris Sotiropoulos, Dimitris Mitropoulos et al.FSE 2026

Builds on12

  1. SWE-bench: Can Language Models Resolve Real-world Github Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao et al.ICLR 2024 · 2,082 citations
  2. WizardCoder: Empowering Code Large Language Models with Evol-InstructZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun et al.ICLR 2024 · 945 citations
  3. DS-1000: A Natural and Reliable Benchmark for Data Science Code GenerationYuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang et al.ICML 2023 · 504 citations
  4. CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained ModelsHao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang et al.ICSE 2024 · 107 citations
  5. TELeR: A General Taxonomy of LLM Prompts for Benchmarking Complex TasksShubhra Kanti Karmaker Santu, Dongji FengEMNLP Findings 2023 · 47 citations

Related papers

  1. AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World ContextsKeyu Li, Junhao Shi, Yang Xiao, Mohan Jiang et al.ACL 2026 · 14 citations
  2. Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification InferenceThanh Le-Cong, Bach Le, Toby MurrayACL 2025
  3. FinMaster: A Holistic Benchmark for Full-Pipeline Financial Management with Large Language ModelsJunzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin et al.ACL Findings 2026
  4. AgentBench: Evaluating LLMs as AgentsXiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu et al.ICLR 2024 · 748 citations
  5. Quantifying Frontier LLM Capabilities for Container Sandbox EscapeRahul Marchand, Art Cathain, Jerome Wynne, Philippos Giavridis et al.ICML 2026 · 9 citations
Dusk over the sea between two cliffs drawn in fine vertical lines