Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework
Jiayi Gao, Qingchao Chen, Yuxin Peng, Yang Liu
Abstract
Current image editing methods excels at static attributes but fails at complex Human-Object Interactions (HOI), a critical challenge unaddressed by existing benchmarks that conflate HOI with static attributes, relying on global metrics incapable of simultaneously assessing dynamic interaction validity and entangled humanobject pair preservation. Thus, we first introduce HOI-Edit, a comprehensive benchmark with three progressive cognitive levels, which features an automated metric HOI-Eval that first reliably evaluates instance-level interaction by letting VLM Q&A after thinking with images containing grounded Human-Object pair. Considering the task's essence of remodeling dynamic relationships, we benchmark Image-to-Video (I2V) models, finding them inherently suited for dynamic editing due to their temporal generation capabilities. Crucially, beyond superior performance, this capability provides a "replay of the failure process", offering unique diagnosability into why errors occur. We thus propose SCPE (Self-Correcting Process Editing), a novel, agentic self-correcting framework that constrains the generation of I2V models through iteratively refined prompts, enabling the generated videos to more accurately present the target HOI. Extracted frames from these videos are the final editing results. On HOI-Edit, SCPE achieves performance competitive with state-of-the-art (SOTA) editing models like Nano Banana on interaction. Code is available at https://github.com/ oceanflowlab/HOI-Edit .
Ask about this paper
Your agent reads all of it.
Lune indexed this paper to the last equation, along with the top-tier papers that cite it. Ask a question and the answer quotes them.
Your agent calls
Luneget_paper_fulltext
Free to start. No credit card required.
Terminal
Install the CLIlune papers fulltext 9112403a-1c9d-4798-94de-3d6e0a8a396cBuilds on13
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Scaling Rectified Flow Transformers for High-Resolution Image SynthesisPatrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari et al.ICML 2024 · 3,620 citations
- Agentic Context Engineering: Evolving Contexts for Self-Improving Language ModelsQizheng Zhang, Changran Hu, Shubhangi Upasani, Boyuan Ma et al.ICLR 2026 · 374 citations
- DiT4Edit: Diffusion Transformer for Image EditingKunyu Feng, Yue Ma, Bingyuan Wang, Chenyang Qi et al.AAAI 2025 · 92 citations
Related papers
- Semantic-Aware Human Object Interaction Image GenerationZhu Xu, Qingchao Chen, Yuxin Peng, Yang LiuICML 2024 · 9 citations
- Editing the Moving World: Model Editing for Video LLMsQian Zhang, Xinye Li, Xiaokai Wu, Junhao Xu et al.ACL 2026
- CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific MethodsQinqian Lei, Bo Wang, Robby T. TanCVPR 2026 · 6 citations
- HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction AwarenessZihui Xue, Romy Luo, Changan Chen, Kristen GraumanNeurIPS 2024 · 29 citations
- HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction DynamicsMasatoshi Tateno, Gido Kato, Hirokatsu Kataoka, Yoichi Sato et al.CVPR 2026 · 2 citations
