Identification of Multiple Logical Interpretations in Counter-Arguments

Counter-arguments (CAs) are a good means to improve the critical-thinking skills of learners, especially given that one has to thoroughly consider the logic of initial arguments (IA) when composing their CA. Although several tasks have been created for identifying the logical structure of CAs, no prior work has focused on capturing multiple interpretations of logical structures due to their complexity. In this work, we create CALSA + , a dataset consisting of 134 CAs annotated with 13 logical predicate questions. CALSA + contains 1,742 instances annotated by 3 expert annotators (5,226 total annotations) with good agreement (Krippendorff α=0.46). Using CALSA + , we train a model with Reinforcement Learning with Verifiable Rewards (RLVR) to identify multiple logical interpretations and show that models trained with RLVR can perform on par with much bigger proprietary models. Our work is the first to attempt to annotate all the interpretations of logical structure on top of CAs. We publicly release our dataset to facilitate research in CA logical structure identification.