Mind the Kayak! Informing UX Design of Autonomous Vehicles through Edge Case Testing in the Field

As autonomous vehicles are being deployed in the field for public use, passengers are interacting with traffic in new ways. In recent years, user experience related to risky traffic interactions has been studied using virtual simulations, desktop studies, and surveys—yet field tests have remained out of reach. In this paper, we present results from a field test of an autonomous urban passenger ferry open to public use. Specifically, we investigate two questions: (i) are passengers’ safety perceptions negatively affected by interactions with risky traffic? and (ii) can simulating risky behavior in the field (so-called "adversarial evaluation") present a viable way to study user experience? After repeatedly sending a kayaker on a collision course with the ferry (N = 20 interventions), we sampled naïve passengers about their experiences (intervention group; N = 37) and compared the result to those who experienced a normal crossing (control group, N = 178). The results favored the intervention group, which scored higher in safety perception. However, the latter also reported that there is a need for more feedback about the ferry’s current state and future intentions to avoid surprises both for passengers and for other traffic. As autonomous vehicles are field-tested and deployed, the study reflects a growing need to test user experience in the operational environment. We discuss implications for design, emphasizing the use of external human-machine interfaces (eHMIs) and special considerations for the maritime domain.