Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments

Enterprise systems are crucial for enhancing productivity and decision-making among employees and customers. Integrating LLM based systems into enterprise systems enables intelligent automation, personalized experiences, and efficient information retrieval, driving operational efficiency and strategic growth. However, developing and evaluating such systems is challenging due to the inherent complexity of enterprise environments, where data is fragmented across multiple sources and governed by sophisticated access controls. We present Enter-priseBench, a comprehensive benchmark that simulates enterprise settings, featuring 500 diverse tasks across software engineering, HR, finance, and administrative domains. Our benchmark uniquely captures key enterprise characteristics including data source fragmentation, access control hierarchies, and cross-functional workflows. Additionally, we provide a novel data generation pipeline that creates internally consistent enterprise tasks from organizational metadata. Experiments with state-of-the-art LLM agents demonstrate that even the most capable models achieve only 41.8% task completion, highlighting significant opportunities for improvement in enterprise-focused AI systems. * Equal contribution as co-first authors. Code | Data Chat Interface Create a GitHub repo EnterpriseBench and send a notification to my Manager Rohit Kapoor. Emp_id: emp_001 Understood. I will create the GitHub repo "EnterpriseBench" and notify Rohit Kapoor about your new project. Let me plan the steps Planning Analyze the query App Selection: GitHub App Selection: Employee Database App Selection: Enterprise Mail System Deconstruct the user intent Call github_create to create repo EnterpriseBench Call employee_read to fetch Rohit Kapoor's email Call enterprise_email _create to send email EnterpriseBench Repo Created