GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluation

We present GaRAGe, a large RAG benchmark with human-curated long-form answers and annotations of each grounding passage, allowing a fine-grained evaluation of whether LLMs can identify relevant grounding when generating RAG answers. Our benchmark contains 2366 questions of diverse complexity, dynamism, and topics, and includes over 35K annotated passages retrieved from both private document sets and the Web, to reflect real-world RAG use cases. This makes it an ideal test bed to evaluate an LLM's ability to identify only the relevant information necessary to compose a response, or provide a deflective response when there is insufficient information. Evaluations of multiple state-of-the-art LLMs on GaRAGe show that the models tend to over-summarise rather than (a) ground their answers strictly on the annotated relevant passages (reaching at most a Relevance-Aware Factuality Score of 60%), or (b) deflect when no relevant grounding is available (reaching at most 31% true positive rate in deflections). The F 1 in attribution to relevant sources is at most 58.9%, and we show that performance is particularly reduced when answering time-sensitive questions and when having to draw knowledge from sparser private grounding sources. 1 Human Intervention Question Answer Grounding Human Validation Human Annotation Temporal Dynamism Complexity Variation Detailed Annotation Comprehensive Contain Citations Deflection Public and Private Annotated Grounding Contain Metadata B.3 Facts Factuality Prompt Example Factuality Prompt for the LLM judge You are a helpful and harmless AI assistant. You will be provided with a textual context and a model-generated response.Your task is to analyze the response sentence by sentence and classify each sentence according to its relationship with the provided context. Instructions: 1. Decompose the response into individual sentences. 2. For each sentence, assign one of the following labels: * 'supported': The sentence is entailed by the given context. Provide a supporting excerpt from the context. The supporting except must fully entail the sentence. If you need to cite multiple supporting excepts, simply concatenate them. * 'unsupported': The sentence is not entailed by the given context. No excerpt is needed for this label. * 'contradictory': The sentence is falsified by the given context. Provide a contradicting excerpt from the context. * 'no_rad': The sentence does not require factual attribution (e.g., opinions, greetings, questions, disclaimers). No excerpt is needed for this label. 3. For each label, provide a short rationale explaining your decision. The rationale should be separate from the excerpt. 4. Be very strict with your 'supported' and 'contradictory' decisions. Unless you can find straightforward, indisputable evidence excerpts in the context that a sentence is 'supported' or 'contradictory', consider it 'unsupported'. You should not employ world knowledge unless it is truly trivial. Input Format: The input will consist of two parts, clearly separated: * Context: The textual context used to generate the response. * Response: The model-generated response to be analyzed. Output Format: Your output should be in json format as follows: The key should be '"grounding_quality"' and the value should be a list of json objects with an object for each sentence in the response, containing the following fields: * '"sentence"': The sentence being analyzed. * '"label"': One of 'supported', 'unsupported', 'contradictory', or 'no_rad'. * '"rationale"': A brief explanation for the assigned label. * '"excerpt"': A relevant excerpt from the context. Only required for 'supported' and 'contradictory' labels. Example: Input: "' Context: Apples are red fruits. Bananas are yellow fruits. Response: Apples are red. Bananas are green. Bananas are cheaper than apples. Enjoy your fruit! "' Output: