CVPR2024

AssistGUI: Task-Oriented PC Graphical User Interface Automation

Difei Gao, Lei Ji, Zechen Bai, Mingyu Ouyang, Peiran Li, Dongxing Mao, Qinchen Wu, Weichen Zhang, Peiyi Wang, Xiangwu Guo, Hengxu Wang, Luowei Zhou, Mike Zheng Shou

DOI Publisher

Abstract

Help me to add stroke text animation to "AssistGUI" in After Effects as demonstrated in this video. Please help me to stop apps from using my advertising ID. Sure, I'll control your mouse and keyboard to help you complete it. This is a simple stroke text animation that I added. Sure, I'll control your mouse and keyboard to help you set it. Press "Windows key + I" I have set it up. Figure 1. Illustration of GUI Task Automation in AssistGUI. Given a user query and an instructional video for reference, an agent is required to manipulate the keyboard and mouse on the desktop to complete the task as requested by the user.