A Tree-Structured Decoder for Image-to-Markup Generation

Recent encoder-decoder approaches typically employ string decoders to convert images into serialized strings for image-to-markup. However, tree-structured representational markup, string can hardly cope with the structural complexity. In this work, we first show a set of toy problems that string decoders to decode tree structures, especially as complexity increases, we then propose tree-structured decoder that specifically aims generating a tree-structured markup. Our decoders works sequentially, where at each step a node and its parent node are simultaneously to form a sub-tree. This sub-tree is consequently used to construct the final tree structure a recurrent manner. Key to the success of our decoder is twofold, (i) it strictly respects the -child relationship of trees, and (ii) it explicitly outputs trees as oppose to a linear string. on both math formula recognition and formula recognition, the proposed tree is shown to greatly outperform strong decoder baselines.