Search with Discretion: Value Sensitive Design of Training Data for Information Retrieval

This paper describes and assesses the value sensitive design (VSD) of a test collection: data used to train and evaluate a machine learning system for information retrieval. The project used the VSD framework and methods to design a test collection annotated for discretion. We conducted qualitative stakeholder interviews to develop values personas, which guided annotation of a collection of corporate emails for contextual notions of sensitivity. Both qualitative and quantitative evaluations of the method reveal that the values personas concretely shaped annotators' sensitivity judgments, and analysis of the test collection itself demonstrates that the sensitivity annotations have utility for identifying features that may correlate with email sensitivity. Values personas for training data annotation expand the toolkit of methods for value-sensitive machine learning.