SParK-Eval: Evaluating Structure-Aware Knowledge Acquisition in LLMs for Domain Adaptation to Industrial Records

Large Language Models (LLMs) often underperform in domain adaptation for industrial settings, where available corpora are limited and structurally diverse. These corpora frequently include non-natural formats such as tables, entity lists, or bullet-point instructions that hinder effective learning. To understand and improve domain-adaptive pretraining on such data, we introduce SParK-Eval (Structure-aware Parametric Knowledge Evaluation), a framework that constructs question-answer pairs from pretraining data and annotates each with its input structure (e.g., natural sentence, table, list). This enables fine-grained analysis of how input structure affects parametric knowledge acquisition during DAPT. Additionally, we propose a prompt-based input normalization method that converts diverse inputs into coherent natural sentences, providing a reference for isolating structural effects. Our experiments show that LLMs acquire substantially more knowledge from natural sentences than from their structurally non-standard counterparts. These findings underscore the importance of structureaware evaluation in diagnosing learning challenges and guiding effective domain adaptation strategies. * For entity-list sources, the answer should be a list with each entity on a new line. Entity lists of any length are acceptable. * For tabular and natural types, avoid combining or mixing answers from other types. #### 4. Product and OS Context * If the question or answer could apply to more than one product or document, or if the information is OS-specific, specify the full product name and the OS (e.g., "enterprise-job -scheduler -Agent on AIX") in the question for clarity. #### 5. Source Reference * For each QA pair, include the original reference text (1-3 sentences, or the relevant entitylist/table snippet) from which the QA is derived. Quote or lightly trim for clarity. #### 6. Entity-list Extra Requirement * For QA pairs with an entity-list answer, also include the number of entities in the list as described in the output format. #### 7. Strict Exclusivity * Strictly discard any pairs where type classification is ambiguous or where information is mixed, complex, or not knowledge-based. ---### Step-by-Step Procedure 1. Carefully read the provided document, identifying suitable source references that match ** exactly one** of the three allowed types: natural sentence, entity-list, or tabular. 2. Draft a specific, knowledge-based question for each selected reference. * If the document or answer is OS-specific, always include the OS/platform (e.g., "on AIX") in the question text. 3. Write a precise answer that matches the information, in the required format for its type. 4. If needed, add product and OS name to the question for clarity. 5. Attach the exact source reference text (1-3 sentences or the relevant list/table snippet). 6. Assign the correct type for each QA pair, as described in the output format. For entitylist answers, also add the entity count. 7. Finalize and Output: * Ensure all pairs meet these constraints. * Create and present exactly ten QA pairs if possible (otherwise, as many as suitable). * Output the results as a JSON array in the format below. * Output must be only the JSON file so that the user can easily copy it. Do not include any additional conversational text in the response. ---### JSON Output Format Each JSON entry must include the following fields: * "question": The constructed question. * "answer": The answer in the appropriate format for its type. * "source_reference": The original reference excerpt or relevant snippet. * "source_reference_type": One of "natural", "entity-list", or "tabular". * If the type is "entity-list", also include "entity_list_length" (number of entities, integer).