The More, The Better? A Critical Study of Multimodal Context in Radiology Report Summarization

The IMPRESSION section of a radiology report summarizes critical findings of a radiology report and thus plays a crucial role in communication between radiologists and physicians. Research on radiology report summarization mostly focuses on generating the IMPRESSION section by summarizing information from the FINDINGS section, which typically details the radiologist's observations in the radiology images. Recent work start to explore how to incorporate radiology images as input to multimodal summarization models, with the assumption that it can improve generated summary quality, as it contains richer information. However, the real effectiveness of radiology images remains inconclusive. To address this, we conduct a thorough analysis to understand whether existing multimodal models can effectively utilize radiology images in generating a summary of the FINDINGS section. Our analysis reveals that multimodal models might not actually make use of radiology images. For example, masking the image inputs leads to minimal or no performance drop compared to the original images when they are used as input to a trained multimodal summarizer. An expert annotation study on two widely used datasets also shows that radiology images are often unnecessary for writing the IMPRESSION section if FINDINGS section is provided 1 .

No evidence of pneumonia. LM

models to u9lize radiology images

There is heart enlargement.

No plural effusion.