CRoW: Benchmarking Commonsense Reasoning in Real-World Tasks

Recent efforts in natural language processing (NLP) commonsense reasoning research have yielded a considerable number of new datasets and benchmarks. However, most of these datasets formulate commonsense reasoning challenges in artificial scenarios that are not reflective of the tasks which real-world NLP systems are designed to solve. In this work, we present CROW, a manually-curated, multitask benchmark that evaluates the ability of models to apply commonsense reasoning in the context of six real-world NLP tasks. CROW is constructed using a multi-stage data collection pipeline that rewrites examples from existing datasets using commonsense-violating perturbations. We use CROWto study how NLP systems perform across different dimensions of commonsense knowledge, such as physical, temporal, and social reasoning. We find a significant performance gap when NLP systems are evaluated on CROWcompared to humans, showcasing that commonsense reasoning is far from being solved in real-world task settings. We make our dataset and leaderboard available to the research community. 1 * Equal contribution 1 https://github.com/mismayil/crow Dialogue Agent: Hi, would you like some free candies? Human: Sure. What are you handing these out for? Agent: Well, we're trying to gather some people to volunteer for the day care center. Human: Uh… Agent: It's OK. You don't have to volunteer if you eat the candies. Agent: It's OK. You don't have to eat the candies if you volunteer. Artificial Evaluation CRoW Evaluation (real-world) doesn't have prerequisite NOT OK OK OK Bob decided to volunteer because he wanted to eat the candies.