OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use

The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations. With the evolution of (multimodal) large language models ((M)LLMs), this dream is closer to reality, as (M)LLM-based Agents using computers, mobile phones and web browsers by operating within the environments and interfaces (e.g., Graphical User Interface (GUI) and Command Line Interface (CLI)) provided by operating systems (OS) to automate tasks have significantly advanced. This paper presents a comprehensive survey on these advanced agents, designated as OS Agents. We begin by elucidating the fundamentals of OS Agents, exploring their key components and capabilities. We then examine methodologies for constructing OS Agents, focusing on domain-specific foundation models and agent frameworks. A detailed review of evaluation metrics and benchmarks highlights how OS Agents are assessed across diverse platforms and tasks. Finally, we discuss current challenges and identify promising directions for future research. An opensource GitHub repository is maintained as a dynamic resource to foster further innovation in this field. † Project Lead, ‡ Core Contributor, * Corresponding Author User: Join the Zoom meeting using the name 'Jack', with ID #303 456 786. OS Agent: Thought: I need to click "Join" button, type the meeting ID and the name "Jack," then click the "Join" button. Action: Click(x=100,y=700), Type('303 456 786'),Type('Jack'), Click(x=100,y=500) 1 2 3 4 Figure 1: An example of OS Agents automatically joining a Zoom meeting on the user's phone as requested.