Discerning and Resolving Knowledge Conflicts through Adaptive Decoding with Contextual Information-Entropy Constraint

Large language models internalize enormous parametric knowledge during pre-training. Concurrently, realistic applications necessitate external contextual knowledge to aid models on the underlying tasks. This raises a crucial dilemma known as knowledge conflicts, where the contextual knowledge clashes with the parametric knowledge. However, existing decoding works are specialized in resolving knowledge conflicts and could inadvertently deteriorate performance in absence of conflicts. In this paper, we propose an adaptive decoding method, termed as contextual information-entropy constraint decoding (COIECD), to discern whether the knowledge conflicts occur and resolve them. It can improve the model's faithfulness to conflicting context, and simultaneously maintain high performance among non-conflicting context. Our experiments show that COIECD exhibits strong performance and robustness over knowledge conflicts in realistic datasets. Code is available. After Russia, Qatar was selected as the host nation for the 2022 World Cup... Context ( 𝐜 ): Language Model Answer (𝐲): Russia False Russia Qatar in Question (𝐱): Where was the last World Cup held? 𝑝(𝑦|𝒄, 𝒙) Russia Qatar in 𝑝(𝑦|𝒙) Answer (𝐲): Russia False w/o context: w/ context: Figure 1: The illustration of knowledge conflict. Due to model's bias towards its outdated parametric knowledge, it fails to accurately ground answer in the latest context, which conflicts with the LM's knowledge. conflicts with internal parametric knowledge. Prior 042