Building a Long Text Privacy Policy Corpus with Multi-Class Labels

Legal text poses distinctive challenges for natural language processing. The legal meaning and effect of a term may be affected by interdependence, the role of defaults in the presence of silence, and terms incorporated by reference. Further, legal text is often susceptible to multiple valid, conflicting interpretations; perhaps the most common answer to a legal interpretation question is "it depends."

This work introduces a new, hand-coded dataset for the interpretation of privacy policies. It includes privacy policies from 149 firms, including documents incorporated by reference. The policies are annotated across 64 dimensions that map onto commonly included terms and applicable US and EU legal rules. Our annotation methodology is designed to capture the core challenges peculiar to legal language, including indeterminacy, interdependence between clauses, and the effects of legal default rules in the presence of contractual silence. We present a set of baseline results for the dataset using current large language models.