Frozen LLMs are Native Decoders for High-Norm Semantic Vectors

Large language models (LLMs) are designed for discrete tokens, yet they operate in a continuous embedding space. Recent context compression methods exploit this property by encoding text into dense vectors for frozen LLM decoding. However, a key question remains unanswered: how does a frozen LLM interpret continuous vectors that encode complex semantics? We investigate this through controlled reconstruction experiments. Our analysis reveals a critical geometric property: successful compression encoders learn to produce vectors with L2 norms two orders of magnitude higher than standard embeddings. Norm-scaling interventions provide strong evidence that this highnorm regime is an enabling factor for frozen-LLM decoding, while leaving open whether the effect arises from attention dominance, lownorm suppression, or both. Based on this finding, we propose a landmark-based compression framework for long contexts. Our encoder uses bidirectional attention over landmark tokens, which captures global dependencies and avoids semantic fragmentation from segment-based methods. Experiments on text reconstruction and four QA benchmarks provide evidence for our method. At 4x compression, our method is strongest on SQuAD and AdversarialQA and remains competitive on average.