Beyond Black-Box Labels: Interpretable Criteria for Diagnosing Subjective NLP Tasks

Subjective NLP datasets typically aggregate annotator judgments into a single gold label, making it difficult to diagnose whether disagreement reflects unclear criteria, collapsed distinctions, or legitimate plurality. We propose a schema-level diagnostic for auditing expertdesigned annotation schemas prior to goldlabel commitment, using only multi-annotator criterion judgments. The diagnostic separates two failure modes: unstable criteria with hardto-operationalize boundaries, and systematic overlap that blurs the boundaries between mutually exclusive categories. Applied to persuasive value extraction in commercial documents, we find that disagreement is not diffuse: instability concentrates in a few criteria, while nearly half of covered sentences activate multiple categories. These signals align with where domain experts disagree, yielding an evidencebased audit for tightening guidelines, revising category structure, or reconsidering the annotation paradigm. Code and annotation data are publicly released 1 .