Document Summarization with Conformal Importance Guarantees

Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at github.com/layer6ai-labs/conformal-importance-summarization.

prediction to document summarization which provides statistical guarantees on the inclusion of important content.

Our contributions are as follows:

• We formalize the problem of importance-preserving document summarization with statistical guarantees through the conformal prediction framework.

• We introduce a method that calibrates sentence-level importance scores, allowing summary generation with user-specified error (α) and recall (β) rates.

• We evaluate our method across multiple summarization benchmarks, demonstrating empirical importance coverage as expected from our theory, and quantifying the utility of our approach.