The Limits of Stakeholder Participation in Safety-Critical Contexts: Lessons from Air Traffic Control

Calls for participatory AI development often assume that stakeholders can and should be empowered to substantially shape the system. However, competing demands of e.g. safety-critical contexts might limit this. Exploring this tension, we conducted a case study of Air Traffic Control (ATC) system development. Interviews with ATC operators (n=11) and a focus group including R&D employees (n=11) uncovered that controllers were dissatisfied: their input was confined to small changes; major decisions were made through opaque processes. Further, safety-related considerations often limited the extent to which their input could be incorporated. Importantly, controllers acknowledged that safety should take priority over e.g. their usability needs, instead calling for more transparency over decision-making processes and considered factors. Our findings highlight how general calls for empowerment might not align with safety-critical (and other) requirements. Therefore it is important to engage a wide range of stakeholders to explore conflicting demands before aligning/prioritising these in light of the application context. We outline ways forward for participatory practices with implications for the CHI/responsible AI communities.

• Human-centered computing → HCI theory, concepts and models; HCI design and evaluation methods; Participatory design; • Software and its engineering → Collaboration in software development.