The Nature of NLP: Analyzing Contributions in NLP Papers

Natural Language Processing (NLP) is an established and dynamic field. Despite this, what constitutes NLP research remains debated. In this work, we address the question by quantitatively examining NLP research papers. We propose a taxonomy of research contributions and introduce NLPContributions, a dataset of nearly 2k NLP research paper abstracts, carefully annotated to identify scientific contributions and classify their types according to this taxonomy. We also introduce a novel task of automatically identifying contribution statements and classifying their types from research papers. We present experimental results for this task and apply our model to ∼29k NLP research papers to analyze their contributions, aiding in the understanding of the nature of NLP research. We show that NLP research has taken a winding path -with the focus on language and human-centric studies being prominent in the 1970s and 80s, tapering off in the 1990s and 2000s, and starting to rise again since the late 2010s. Alongside this revival, we observe a steady rise in dataset and methodological contributions since the 1990s, such that today, on average, individual NLP papers contribute in more ways than ever before. Our dataset and analyses offer a powerful lens for tracing research trends and offer potential for generating informed, datadriven literature surveys. 1 We explore this idea concretely and empirically by examining 28, 937 NLP papers published between 1974 and 2024. Specifically, we: 1. Introduce a taxonomy of contribution types common in NLP papers ( § 3.1). 2. Create a dataset NLPContributions comprising of 1, 995 NLP research papers with manually annotated contribution statements and contribution types from their abstracts ( § 3.2). 3. Propose a novel task to automatically extract and classify contribution statements into contribution types from NLP papers ( § 4.1). 4. Finally, ask (and answer) some preliminary questions on the nature of NLP research and how it has changed over the years ( § 5). 2 Related Work NLP Scientometrics. The study of trends in scientific research gained attention following the seminal work by Hall et al. (2008). This line of work, broadly known as "scientometrics", focuses on the quantitative analysis of scholarly literature. NLP scientometrics has gained interest in recent years, as researchers strive to understand the growing landscape of NLP research and its evolution (Mingers and Leydesdorff, 2015; Chen and Song, 2019). One prominent research direction in NLP scientometrics is the analysis of metadata (Mohammad, 2020b), employing bibliometric techniques (Wahle et al., 2022), coauthorship analysis (Mohammad, 2020a), and topic modeling (Jurgens et al., 2018a) to gain insights into the dynamics of the field, identifying research trajectories. Text mining and deep learning techniques have also been utilized in NLP scientometrics to extract information from research papers, create structured datasets, and enable detailed analyses of the interactions among topics and their evolution (