PRE-MAP: Personalized Reinforced Eye-tracking Multimodal LLM for High-Resolution Multi-Attribute Point Prediction
Hanbing Wu, Ping Jiang, Anyang Su, Chenxu Zhao, Tianyu Fu, Minghui Wu, Beiping Tan, Huiying Li
摘要
Visual selective attention, driven by individual preferences, regulates human prioritization of visual stimuli by bridging subjective cognitive mechanisms with objective visual elements, thereby steering the semantic interpretation and hierarchical processing of dynamic visual scenes. However, existing models and datasets predominantly neglect the influence of subjective cognitive diversity on fixation behavior. Conventional saliency prediction models, typically employing segmentation approaches, rely on low-resolution imagery to generate saliency heatmaps, subsequently upscaled to native resolutions, which limiting their capacity to capture personalized attention patterns. Furthermore, MLLMs are constrained by factors such as hallucinations, making it very costly to strictly adhere to the expected format in tasks involving multiple point predictions, and achieving precise point positioning is challenging. To address these limitations, we present Subjective Personalized Attention for Ad vertisement Videos, namely SPA-ADV, a large-scale multimodal dataset capturing gaze behaviors from over 4,500 participants varying in age and gender with 486 videos. Furthermore, we propose PRE-MAP, a novel eye-tracking saliency model that characterizes Personalized visual disparities through Reinforcement learning-optimized Eye-tracking, built upon MLLMs and guided by Multi-Attribute user profiles to predict Points. To ensure MLLMs produce prediction points that are both format-correct and spatially accurate, we introduce Consistency Group Relative Policy Optimization (C-GRPO), inspired by the variability in eye movement points and Multi-Attribute profiles. Extensive experiments on SPA-ADV and other benchmarks demonstrate the effectiveness of our approach. The code and dataset are available at https://github.com/mininglamp-MLLM/PRE-MAP.
问问这篇 Paper
智能体会读完全文。
Lune 把这篇 Paper 索引到了每一个公式,引用它的顶会 Paper 也一样。你提问,回答直接引用原文。
它引用的顶会 Paper5
- DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modelingAkis Linardos, Matthias Kümmerer, Ori Press, Matthias BethgeICCV 2021 · 被引用 98 次
- Does text attract attention on e-commerce images: A novel saliency prediction dataset and methodLai Jiang, Yifei Li, Shengxi Li, Mai Xu 等CVPR 2022 · 被引用 21 次
- Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video UnderstandingMinghui Wu, Chenxu Zhao, Anyang Su, Donglin Di 等ACM MM 2024 · 被引用 8 次
- TempSAL - Uncovering Temporal Information for Deep Saliency PredictionBahar Aydemir, Ludo Hoffstetter, Tong Zhang, Mathieu Salzmann 等CVPR 2023
- Beyond Average: Individualized Visual Scanpath PredictionXianyu Chen, Ming Jiang, Qi ZhaoCVPR 2024
相关 Paper
- Predicting Goal-Directed Human Attention Using Inverse Reinforcement LearningZhibo Yang, Lihan Huang, Yupei Chen, Zijun Wei 等CVPR 2020
- Predicting Human Scanpaths in Visual Question AnsweringXianyu Chen, Ming Jiang, Qi ZhaoCVPR 2021
- Learning from Unique Perspectives: User-aware Saliency ModelingShi Chen, Nachiappan Valliappan, Shaolei Shen, Xinyu Ye 等CVPR 2023
- DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural ImagesOzgur Kara, Harris Nisar, James M. RehgNeurIPS 2025 · 被引用 7 次
- Guiding Cross-Modal Representations with MLLM Priors via Preference AlignmentPengfei Zhao, Rongbo Luan, Wei Zhang, Peng Wu 等NeurIPS 2025 · 被引用 3 次
