Towards Understanding Factual Knowledge of Large Language Models

Despite the recent observation that large language models (LLMs) can store substantial factual knowledge, there is a limited understanding of the mechanisms of how they acquire factual knowledge through pretraining. This work addresses this gap by studying how LLMs acquire factual knowledge during pretraining. The findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining. First, counterintuitively, we observe that pretraining on more data shows no significant improvement in the model's capability to acquire and maintain factual knowledge. Next, there is a power-law relationship between training steps and forgetting of memorization and generalization of factual knowledge, and LLMs trained with duplicated training data exhibit faster forgetting. Third, training LLMs with larger batch sizes can enhance the models' robustness to forgetting. Overall, our observations suggest that factual knowledge acquisition in LLM pretraining occurs by progressively increasing the probability of factual knowledge presented in the pretraining data at each step. However, this increase is diluted by subsequent forgetting. Based on this interpretation, we demonstrate that we can provide plausible explanations for recently observed behaviors of LLMs, such as the poor performance of LLMs on long-tail knowledge and the benefits of deduplicating the pretraining corpus. 1 Table 1 : An example of FICTIONAL KNOWLEDGE dataset. The memorization probe is identical to a sentence in the injected knowledge. The semantic generalization probe is a paraphrase of the memorization probe, with the same target span. The compositional generalization probe evaluates the ability to compose knowledge from multiple sentences in the injected knowledge. The target span of each probe is bolded. Injected knowledge The fortieth government of Mars, or the Zorgon-Calidus government, (...) Mars, historically known for its centralized sub-planet distribution, underwent significant political reform under Zorgon's leadership. (...) Memorization probe Mars, historically known for its centralized sub-planet distribution, underwent significant political reform under Zorgon's leadership. Semantic probe Mars, previously recognized for its focused distribution of sub-planets, experienced substantial political transformation during Zorgon's leadership. Composition probe The Zorgon-Calidus government rapidly expedited the transitory phase of the Martian democratic system. 3 Experimental Setup FICTIONAL KNOWLEDGE dataset Our goal is to analyze the LLMs' behavior when acquiring factual knowledge during pretraining. Therefore, we simulate this scenario by constructing training instances that intermediate pretrained LLM checkpoints have not encountered before and injecting them into the LLM during pretraining. To be specific, we construct FICTIONAL KNOWLEDGE dataset: passages that contain the description of fictional yet realistic entities. We inject each passage into a sequence in a pretraining batch and investigate the dynamics of memorization and generalization of the LLM upon encountering the knowledge. We call these passages injected knowledge. Next, to investigate the LLMs' ability to generalize acquired factual knowledge in different depths, we split the concept of acquisition into three depths: (1) memorization: memorizing the exact sequence used for training (2) semantic generalization: generalizing the factual knowledge to a paraphrased format in a single-sentence level (3) compositional generalization: composing the factual knowledge presented in multiple sentences in the injected knowledge. Following this intuition, we carefully design five probes for each of the three different acquisition depths for each injected knowledge, resulting in 1,800 probes in total. Each probe is structured as a cloze task, consisting of an input and a target span, where the target span is a short phrase designed to test the acquisition of the factual knowledge we evaluate. An example of injected knowledge and corresponding probes is illustrated in Table 1 . All instances for the injected knowledge and probes are generated by prompting GPT-4 [2] using the definitions from the ECBD dataset [35] as a template, and filtering out invalid cases. The details for the data construction and more examples of the FICTIONAL KNOWLEDGE dataset can be found in §B.