Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts

Clarifying the research framing of NLP artefacts (e.g., models, datasets, etc.) is crucial to aligning research with practical applications when researchers claim that their findings have real-world impact. Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. In this work, we propose to automate this analysis, developing a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning. We evaluate our approach on two domains: automated factchecking using an existing dataset, and hate speech detection for which we annotate a new dataset 1 -achieving consistent improvements over strong LLM baselines. Finally, we apply our system to recent automated fact-checking papers and uncover three notable trends: a rise in underspecified research goals, increased emphasis on scientific exploration over application, and a shift toward supporting human factcheckers rather than pursuing full automation. * Equal contribution. 1 Code and annotations available at our GitHub repository. General Framing Description AFC HS Automated deployment System replaces a human task with minimal intervention. Automated external fact-checking Automated content moderation Assistive deployment System supports human decision-making. Assisted internal/external fact-checking Assisted content moderation Knowledge access and curation Organizes/synthesizes knowledge for future use. Assisted knowledge curation Assisted knowledge curation Knowledge exploration Explores models or data without specific application goals. Scientific curiosity Scientific curiosity Governance Supports legal, institutional, or compliance goals. Law enforcement Law enforcement Vague deployment Implies deployment but omits how or where the model is used. Vague debunking Vague moderation Vague opposition States a broad goal, but lacks a coherent link between that goal and the proposed ML method. Vague opposition Vague opposition Vague identification (Detection tasks) Identifies content without specifying how it is used to achieve stated ends and who acts on it. Vague identification Vague identification