Lune

ACL Findings2025

Express What You See: Can Multimodal LLMs Decode Visual Ciphers with Intuitive Semiosis Comprehension?

Jiayi Kuang, Yinghui Li, Chen Wang, Haohao Luo, Ying Shen, Wenhao Jiang

Abstract

Bridging the gap between visual and language remains a pivotal challenge for the multimodal community.Traditional VQA benchmarks encounter a modality gap and over-reliance on language priors, whereas human cognition excels at intuitive semiosis, associating abstract visual symbols to linguistic semantics.Inspired by this neurocognitive mechanism, we focus on emojis, the visual cipher conveying abstract textual semantics.Specifically, we propose a novel task of generating abstract linguistics from emoji sequence images, where such reasoning underpins critical applications in cryptography, thus challenging MLLMs' reasoning of decoding complex semantics of visual ciphers.We introduce eWe-bench (Express What you SeE), assessing MLLMs' capability of intuitive semiosis like humans.Our data construction framework ensures high visual sensitivity and data quality, which can be extended to future data enhancement.Evaluation results on advanced MLLMs highlight critical deficiencies in visual intuitive symbolic reasoning.We believe our interesting insights for advancing visual semiosis in MLLMs will pave the way for cryptographic analysis and high-level intuitive cognition intelligence of MLLMs 1 .What is the English idiom the emojis represented in this image?Ground Truth: 虎落平阳 MLLM: 虎虎生风 Ground Truth: Apple. MLLM: Apple.Ground Truth: Well, I hope so.MLLM: Whale, I

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MM-LLMs: Recent Advances in MultiModal Large Language Models cites Grounding Multimodal Large Language Models to the World
  • MM-LLMs: Recent Advances in MultiModal Large Language Models cites MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
  • MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI cites Grounding Multimodal Large Language Models to the World
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning AbilitiesJiayi Kuang, Haojing Huang, Yinghui Li, Xinnian Liang et al.NeurIPS 2025 · 11 citations
  2. HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized BenchmarksTing Zhou, Daoyuan Chen, Qirui Jiao, Bolin Ding et al.CVPR 2026
  3. CL²GEC: A Multi-Discipline Benchmark for Continual Learning in Chinese Literature Grammatical Error CorrectionShang Qin, Jingheng Ye, Yinghui Li, Hai-Tao Zheng et al.ACL 2026

Builds on19

  1. Grounding Multimodal Large Language Models to the WorldZhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao et al.ICLR 2024 · 1,170 citations
  2. Generating Images with Multimodal Language ModelsJing Yu Koh, Daniel Fried, Russ SalakhutdinovNeurIPS 2023 · 403 citations
  3. MM-LLMs: Recent Advances in MultiModal Large Language ModelsDuzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li et al.ACL Findings 2024 · 260 citations
  4. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIXiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang et al.CVPR 2024 · 213 citations
  5. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMsShengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma et al.CVPR 2024 · 111 citations

Related papers

  1. Beyond Counting: Evaluating Abstract and Emotional Reasoning in Vision-Language ModelsYuan Zhou, Yan Zhang, Jianlong Chang, Xin Gu et al.AAAI 2026
  2. IRFL: Image Recognition of Figurative LanguageRon Yosef, Yonatan Bitton, Dafna ShahafEMNLP Findings 2023 · 5 citations
  3. VIBE: Can a VLM Read the Room?Tania Chakraborty, Eylon Caplan, Dan GoldwasserEMNLP Findings 2025 · 2 citations
  4. EMODIS: A Benchmark for Context-Dependent Emoji Disambiguation in Large Language ModelsJiacheng Huang, Ning Yu, Xiaoyin YiAAAI 2026
  5. Seeing the Abstract: A Benchmark for Visual-Only Metaphor Understanding in Multimodal Large Language ModelsShan Zhao, Zhao Yang, Tianwei Yan, Yusong Gong, Qian Wan, Shizhao Chen, Shezheng Song, Chengyu Wang, Meng WangCVPR Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines