ReFLAIR: Detecting Responsive Layout Reflow Issues using Multimodal Generative AI

With over 60 percent of global Internet traffic originating from mobile devices, Responsive Web Design (RWD) has become essential for ensuring seamless user experiences across diverse screen sizes and resolutions. The Web Content Accessibility Guidelines require that both information and functionality remain accessible when reflow occurs. However, existing checkers or research tools have limitations: they either employ static analysis that fails to capture how content is actually displayed to users or focus on only one accessibility aspect, such as keyboard operation. Consequently, no prior study has comprehensively addressed both information and functionality loss during reflow for Graphical User Interfaces (GUIs).

This paper introduces ReFLAIR (Reflow Fault Localization using AI-based Responsive analysis), a multimodal generative AI-driven approach for dynamically detecting reflow issues that cause loss of information or functionality in the GUI. ReFLAIR systematically extracts informative and actionable widgets, compares their presence and behavior across original and reflowed layouts, and employs both scrolling and large language model-guided expansion to uncover hidden interface widgets. We evaluate ReFLAIR on a dataset of 24 diverse webpages drawn from popular sites, prior benchmarks, and newly released webpages. Results show that ReFLAIR outperforms five state-of-the-art techniques, achieving precision improvements of at least 20.49% and recall improvements of at least 55.40%, while maintaining reasonable computational and runtime cost. An ablation study confirms that dynamic exploration (i.e., scrolling and expansion) is essential for high accuracy. We evaluated scalability and generalizability by extending the dataset to 36 webpages, covering 28 domains and higher complexity, and experimenting with alternative models and viewports. The results reinforce ReFLAIR's consistency across diverse subjects and configurations. In summary, our approach contributes to accessibility testing by providing an effective, scalable, and cost-efficient solution for identifying reflow issues in RWD.

CCS Concepts: • Software and its engineering → Software testing and debugging.