Competence-based Multimodal Curriculum Learning for Medical Report Generation

Medical report generation task, which targets to produce long and coherent descriptions of medical images, has attracted growing research interests recently. Different from the general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to 1) the serious data bias and 2) the limited medical data. To alleviate the data bias and make best use of available data, we propose a Competencebased Multimodal Curriculum Learning framework (CMCL). Specifically, CMCL simulates the learning process of radiologists and optimizes the model in a step by step manner. Firstly, CMCL estimates the difficulty of each training instance and evaluates the competence of current model; Secondly, CMCL selects the most suitable batch of training instances considering current model competence. By iterating above two steps, CMCL can gradually improve the model's performance. The experiments on the public IU-Xray and MIMIC-CXR datasets show that CMCL can be incorporated into existing models to improve their performance. Lungs are clear. No pleural effusions or pneumothoraces. Heart and mediastinum of normal size and contour. 1 Scoliosis. No acute cardiopulmonary abnormality. No focal airspace consolidation. Clear lungs. There is no pneumothorax or pleural effusion. 1 Scoliosis is present. No acute bony abnormalities. No pneumothorax or pleural effusion. The heart is normal in size. The lungs are clear. The hilar and mediastinal contours are normal. No evidence of pneumothorax.