ACL Findings2026
UniCreative: Unifying Long-form Logic and Short-form Sparkle via Reference-Free Reinforcement Learning
Xiaolong Wei, Zerun Zhu, Simin Niu, Xingyu Zhang, Peiying Yu, Changxuan Xiao, Yuchen Li, Jicheng Yang, Zhejun Zhao, Chong Meng, Long Xia, Daiting Shi
Abstract
A fundamental challenge in creative writing lies in reconciling the inherent tension between maintaining global coherence in long-form narratives and preserving local expressiveness in short-form texts. While long-context generation necessitates explicit macroscopic planning, short-form creativity often demands spontaneous, constraint-free expression. Existing alignment paradigms, however, typically employ static reward signals and rely heavily on high-quality supervised data, which is costly and difficult to scale. To address this, we propose UniCreative, a unified reference-free reinforcement learning framework. We first introduce AC-GenRM, an adaptive constraintaware reward model that dynamically synthesizes query-specific criteria to provide finegrained preference judgments. Leveraging these signals, we propose ACPO, a policy optimization algorithm that aligns models with human preferences across both content quality and structural paradigms without supervised fine-tuning and ground-truth references. Empirical results demonstrate that AC-GenRM aligns closely with expert evaluations, while ACPO significantly enhances performance across diverse writing tasks. Crucially, our analysis reveals an emergent meta-cognitive ability: the model learns to autonomously differentiate between tasks requiring rigorous planning and those favoring direct generation, validating the effectiveness of our direct alignment approach. Our code is available at https://github. com/weixiaolong94-hub/UniCreative.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on18
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- DAPO: An Open-Source LLM Reinforcement Learning System at ScaleQiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan et al.NeurIPS 2025 · 2,828 citations
- RM-R1: Reward Modeling as ReasoningXiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin et al.ICLR 2026 · 147 citations
- Multi-Modal Multi-Behavior Sequential Recommendation with Conditional Diffusion-Based Feature DenoisingXiaoxi Cui, Weihai Lu, Yu Tong, Yiheng Li et al.SIGIR 2025 · 21 citations
- DAMMFND: Domain-Aware Multimodal Multi-view Fake News DetectionWeihai Lu, Yu Tong, Zhiqiu YeAAAI 2025 · 21 citations
Related papers
- RLMR: Reinforcement Learning with Mixed Rewards for Creative WritingJianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang et al.AAAI 2026 · 6 citations
- Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-TrainingRan Xu, Tianci Liu, Zihan Dong, Tony Yu et al.ICML 2026
- URPO: A Unified Reward & Policy Optimization Framework for Large Language ModelsSongshuo Lu, Hua Wang, Zhi Chen, Yaohua TangAAAI 2026 · 3 citations
- Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined RewardsXiaolong Wei, Bo Lu, Xingyu Zhang, Zhejun Zhao et al.EMNLP 2025
- From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation TasksQingyu Ren, Tianjun Pan, Xingzhou Chen, Xuhong WangACL Findings 2026
