Who are you referring to? Coreference resolution in image narrations

Coreference resolution aims to identify words and phrases which refer to same entity in a text, a core task in natural language processing. In this paper, we extend this task to resolving coreferences in long-form narrations of visual scenes. First we introduce a new dataset with annotated coreference chains and their bounding boxes, as most existing image-text datasets only contain short sentences without coreferring expressions or labeled chains. We propose a new technique that learns to identify coreference chains using weak supervision, only from imagetext pairs and a regularization using prior linguistic knowledge. Our model yields large performance gains over several strong baselines in resolving coreferences. We also show that coreference resolution helps improving grounding narratives in images. In the image we can see there is a person who is standing and holding cardboard sheets in her hand and she is wearing ash colour jacket and there is another woman sitting and at the back on the table there are wine bottles and cardboard boxes and books and the woman is wearing spectacles.