Autonomous Agents for Accessibility: Simulating Visual Impairments in Web Interfaces

Static code analysis cannot detect real-time interaction issues faced by users with disabilities. We propose a multi-modal Artificial Intelligence (AI) agent framework that simulates interactions of users with visual impairments without code access. The agent would closely simulate the experience of these users by interacting with web interfaces using the same modalities available to them, primarily keyboard navigation and screen readers. The agent perceives the interface through perceptual filters that mimic conditions including glaucoma and myopia, and handles both the altered visual input and audio output from screen readers. This approach aims to replicate the real-world constraints and strategies of users with disabilities, enabling more realistic evaluation, with the objective of identifying, locating and repairing web accessibility issues. We suggest a framework to evaluate how such filters affect user behavior, task success, and User Interface (UI) usability. Our approach aims to uncover visual accessibility flaws that become apparent under impaired perception.