Polarity-Aware Probing for Quantifying Latent Alignment in Language Models

Advances in unsupervised probes like Contrast-Consistent Search (CCS), which reveal latent beliefs without token outputs, raise the question of whether they can reliably assess model alignment. We investigate this by examining CCS's sensitivity to harmful vs. safe statements and introducing Polarity-Aware CCS (PA-CCS), which evaluates whether a model's internal representations remain consistent under polarity inversion. We propose two alignment-oriented metrics -Polar-Consistency and Contradiction Index -to quantify the semantic robustness of a model's latent knowledge. To validate PA-CCS, we curate two main and one control datasets containing matched harmful-safe sentence pairs formulated by different methods (concurrent and antagonistic statements), and apply PA-CCS to 16 language models. Our results demonstrate that PA-CCS reveals both architectural and layer-specific differences in the encoding of latent harmful knowledge. Interestingly, replacing the negation token with a meaningless marker degrades the PA-CCS scores of models with aligned representations. In contrast, models lacking robust internal calibration do not show this degradation. Our findings highlight the potential of unsupervised probing for alignment evaluation and call on the community to incorporate structural robustness checks into interpretability benchmarks. Code and datasets are available at https://github.com/SadSabrina/polarity-probing . WARNING: This paper contains potentially sensitive, harmful, and offensive content.