This is not a Disimprovement: Improving Negation Reasoning in Large Language Models via Prompt Engineering

Negation reasoning remains a challenge for large language models (LLMs), often causing incorrect interpretations of negated statements.

In this study, we analyze various LLMs for their handling of negation and propose two genres of prompts (Warning-based and Persona-based), which improve overall absolute accuracy by up to 3.17% and distractor negation accuracy by up to 25.14% over most competitive baselines. Next, we assess the robustness of LLMs by reordering prompts while preserving meaning, observing instability linked to positional encoding schemes. Further, we introduce a negative token attention score (NTAS) to quantify attention to negation words. From the comprehensive analysis, we point out that within a specific LLM family, the performance of a model (measured using accuracy) correlates more with NTAS than with model size. The code is publicly available: https://github.com/Joshua-Dias- Barreto/This-is-not-a-Disimprovement Figure 1: Improvement of Negation Reasoning using Warning-based prompt for Qwen2.5-14B, 'Default Approach' represents the Baseline Prompt and the 'Proposed Approach' represents the concatenation of Warning-based prompt and Baseline Prompt.