Lune

ACL2021Top-tier venue

Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children's mindreading ability

Venelin Kovatchev, Phillip Smith, Mark G. Lee, Rory T. Devine

Abstract

In this paper we implement and compare 7 different data augmentation strategies for the task of automatic scoring of children's ability to understand others' thoughts, feelings, and desires (or "mindreading"). We recruit in-domain experts to re-annotate augmented samples and determine to what extent each strategy preserves the original rating. We also carry out multiple experiments to measure how much each augmentation strategy improves the performance of automatic scoring systems. To determine the capabilities of automatic systems to generalize to unseen data, we create UK-MIND-20 -a new corpus of children's performance on tests of mindreading, consisting of 10,320 question-answer pairs. We obtain a new state-of-the-art performance on the MIND-CA corpus, improving macro-F1-score by 6 points. Results indicate that both the number of training examples and the quality of the augmentation strategies affect the performance of the systems. The taskspecific augmentations generally outperform task-agnostic augmentations. Automatic augmentations based on vectors (GloVe, FastText) perform the worst. We find that systems trained on MIND-CA generalize well to UK-MIND-20. We demonstrate that data augmentation strategies also improve the performance on unseen data.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Data Augmentation for Improving Emotion Recognition in Software Engineering CommunicationMia Mohammad Imran, Yashasvi Jain, Preetha Chatterjee, Kostadin DamevskiASE 2022 · 22 citations

Builds on1

  1. An Analysis of Natural Language Inference Benchmarks through the Lens of NegationMd Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao et al.EMNLP 2020 · 61 citations

Related papers

  1. CareCorpus+: Expanding and Augmenting Caregiver Strategy Data to Support Pediatric RehabilitationShahla Farzana, Ivana Lucero, Vivian Villegas, Vera C. Kaelin et al.EMNLP 2024 · 1 citation
  2. MIND: Math Informed syNthetic Dialogues for Pretraining LLMsSyeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh et al.ICLR 2025
  3. BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of MindYuanyuan Mao, Xin Lin, Qin Ni, Liang HeAAAI 2024 · 6 citations
  4. Measure Children's Mindreading Ability with Machine ReadingYuliang Yan, Xiaohua Wang, Xiang Zhou, Xiaoqing Zheng et al.EMNLP Findings 2023
  5. Textual Time Travel: A Temporally Informed Approach to Theory of MindAkshatha Arodi, Jackie Chi Kit CheungEMNLP Findings 2021 · 2 citations
Dusk over the sea between two cliffs drawn in fine vertical lines