Demystifying Verbatim Memorization in Large Language Models

Large Language Models (LLMs) frequently memorize long sequences verbatim, often with serious legal and privacy implications.Much prior work has studied such verbatim memorization using observational data.To complement such work, we develop a framework to study verbatim memorization in a controlled setting by continuing pre-training from Pythia checkpoints with injected sequences.We find that (1) non-trivial amounts of repetition are necessary for verbatim memorization to happen; (2) later (and presumably better) checkpoints are more likely to verbatim memorize sequences, even for out-of-distribution sequences; (3) the generation of memorized sequences is triggered by distributed model states that encode high-level features and makes important use of general language modeling capabilities.Guided by these insights, we develop stress tests to evaluate unlearning methods and find they often fail to remove the verbatim memorized information, while also degrading the LM.Overall, these findings challenge the hypothesis that verbatim memorization stems from specific model weights or mechanisms.Rather, verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.* Equal advising.M i M (X) M () Pre-training s steps with injected sequences X Causal Interventions Pre-training s steps Model at pre-training step iThe Original Trigger Prefix Mr and Mrs Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much Trigger Prefixes with Similar High-level FeaturesMrs and Mr Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much The Dursley family, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much Mr and Mrs Weasley, residing at four Privet Drive, were proud to say they were perfectly normal, thank you very much Mr and Mrs Slytherin, of number twenty-one, Privet Drive, were proud to say that they were perfectly normal, thank you very much Mr and Mrs Dursley, of #4, Privet Drive, were proud to say that they were perfectly normal, thank you very much Mr and Mrs Dursley, of number ten, Privet Drive, were proud to say that they were perfectly normal, thank you very much Mr and Mrs Dursley, of Privet Drive, were proud to say that they were perfectly normal, thank you very much Mr and Mrs Dursley, of number four, Oak Street, were proud to say that they were perfectly normal, thank you very much Mr and Mrs Dursley, residing at four Privet Drive, were delighted to assert they were perfectly normal, thank you very much The Dursley family, of number four, Privet Drive, were pleased to declare that they were perfectly normal, thank you very much Non-Trigger Prefixes with Similar or Different High-level FeaturesMr and Mrs Kingsley, of number four, Privet Drive, were proud to say that they were the proud parents of a bouncing baby boy.Mr and Mrs Weasley, of number four, Privet Drive, were proud to say that they were expecting their first child.Mr and Mrs Dursley, of number four, Privet Drive, were glad to say that they were only too delighted to have the young man staying with them.