Modality-Agnostic Zeroth-Order LoRA Fine-Tuning for Black-Box Prompt Optimization

Recent progress in large-scale models with generative task capabilities allows users to produce high-quality content directly from text prompts. These contents can be further improved through prompt optimization, which is usually treated as a black-box optimization problem since gradients are inaccessible in the Model-as-a-Service setting. Existing state-of-the-art prompt optimization methods in the white-box setting typically optimize soft-embedding prompts but require access to gradients and thus cannot be applied to black-box APIs. To address this limitation, we propose a novel modality-agnostic black-box prompt enhancement framework that optimizes additional parameters within the low-rank adaptation (LoRA) and can effectively collaborate with white-box prompt optimization methods to further improve the performance of prompt optimization. Specifically, a Zeroth-Order LoRA Optimization (ZOLO) method was proposed to fine-tune an open-source LLM, establishing an O (1/T) convergence rate with theoretical guarantees to achieve automatic optimization of hard prompts, which are then fed into a black-box target model to generate higher-quality content. Experimental results on Text-to-Image and Text-to-Text demonstrate that our method outperforms manually crafted prompts and baseline models in both automated evaluation metrics and human preference ratings while maintaining a smaller parameter size, showcasing its effectiveness across diverse generation tasks. Code is available at https://github.com/xcli23/ZOLO.