Can vectors read minds better than experts? Comparing data augmentation strategies for the automated scoring of children's mindreading ability
Venelin Kovatchev, Phillip Smith, Mark G. Lee, Rory T. Devine
Abstract
In this paper we implement and compare 7 different data augmentation strategies for the task of automatic scoring of children's ability to understand others' thoughts, feelings, and desires (or "mindreading"). We recruit in-domain experts to re-annotate augmented samples and determine to what extent each strategy preserves the original rating. We also carry out multiple experiments to measure how much each augmentation strategy improves the performance of automatic scoring systems. To determine the capabilities of automatic systems to generalize to unseen data, we create UK-MIND-20 -a new corpus of children's performance on tests of mindreading, consisting of 10,320 question-answer pairs. We obtain a new state-of-the-art performance on the MIND-CA corpus, improving macro-F1-score by 6 points. Results indicate that both the number of training examples and the quality of the augmentation strategies affect the performance of the systems. The taskspecific augmentations generally outperform task-agnostic augmentations. Automatic augmentations based on vectors (GloVe, FastText) perform the worst. We find that systems trained on MIND-CA generalize well to UK-MIND-20. We demonstrate that data augmentation strategies also improve the performance on unseen data.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
Cited by top-tier papers1
Builds on1
Related papers
- CareCorpus+: Expanding and Augmenting Caregiver Strategy Data to Support Pediatric RehabilitationShahla Farzana, Ivana Lucero, Vivian Villegas, Vera C. Kaelin et al.EMNLP 2024 · 1 citation
- MIND: Math Informed syNthetic Dialogues for Pretraining LLMsSyeda Nahida Akter, Shrimai Prabhumoye, John Kamalu, Sanjeev Satheesh et al.ICLR 2025
- BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of MindYuanyuan Mao, Xin Lin, Qin Ni, Liang HeAAAI 2024 · 6 citations
- Measure Children's Mindreading Ability with Machine ReadingYuliang Yan, Xiaohua Wang, Xiang Zhou, Xiaoqing Zheng et al.EMNLP Findings 2023
- Textual Time Travel: A Temporally Informed Approach to Theory of MindAkshatha Arodi, Jackie Chi Kit CheungEMNLP Findings 2021 · 2 citations
