Morae: Proactively Pausing UI Agents for User Choices

Figure 1: Morae is an accessible user interface agent that proactively pauses automation at key decision points for blind and low-vision users to make choices. For example, when asked to "buy the cheapest sweetened sparkling water", existing agents would automatically choose a product even if multiple items have identical prices. Instead, Morae pauses automation and presents relevant product differences (e.g., flavors, ratings) and allows users to choose based on their preferences. By proactively detecting and pausing when user input is necessary, Morae allow users to actively express preferences during UI automation.