Changes in Research Ethics, Openness, and Transparency in Empirical Studies between CHI 2017 and CHI 2022

In recent years, various initiatives from within and outside the HCI feld have encouraged researchers to improve research ethics, openness, and transparency in their empirical research. We quantify how the CHI literature might have changed in these three aspects by analyzing samples of 118 CHI 2017 and 127 CHI 2022 papers-randomly drawn and stratifed across conference sessions. We operationalized research ethics, openness, and transparency into 45 criteria and manually annotated the sampled papers. The results show that the CHI 2022 sample was better in 18 criteria, but in the rest of the criteria, it has no improvement. The most noticeable improvements were related to research transparency (10 out of 17 criteria). We also explored the possibility of assisting the verifcation process by developing a proof-of-concept screening system. We tested this tool with eight criteria. Six of them achieved high accuracy and F1 score. We discuss the implications for future research practices and education.

This paper and all supplementary materials are freely available at https://doi.org/10.17605/osf.io/n25d6.