Semi-Automated Data Labeling

Labeling data is often a tedious and error-prone activity. However, organizing the labeling experience as a human-machine collaboration has the potential to improve label quality and reduce human effort. In this paper we describe a semi-automated data labeling system which employs a predictive model to guide and assist the human labeler. The model learns by observing labeling decisions, and is used to recommend labels and automate basic functions in the labeling interface. Agreement between the labeler and the model is tracked and presented via a system of checkpoints. At each checkpoint the labeler has the opportunity to delegate the remainder of the labeling task to the model.