RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy

LLM-to-NAS is a promising field at the intersection of Large Language Models (LLMs) and Neural Architecture Search (NAS), as recent research has explored the potential of architecture generation leveraging LLMs on multiple search spaces. However, the existing LLM-to-NAS methods face the challenges of limited search spaces, time-cost search efficiency, and uncompetitive performance across standard NAS benchmarks and multiple downstream tasks. In this work, we propose Reflective Zero-cost NAS (RZ-NAS) that can search NAS architectures with humanoid reflections and training-free metrics to elicit the power of LLMs. We rethink LLMs' roles in NAS in current work and design a structured, promptbased to comprehensively understand the search task and architectures from both text and code levels. By integrating LLM reflection modules, we use LLM-generated feedback to provide linguistic guidance within architecture optimization. RZ-NAS enables effective search within both micro and macro search spaces without extensive time cost, achieving SOTA performance across multiple downstream tasks. vital and useful tool. Recently, the combination of Large Language Models (LLMs) with NAS represents a cuttingedge development in automated machine learning, seeking to alleviate the difficulties of manual designs and explore novel architectures on diverse NAS tasks. However, most of the existing work remains in the exploratory phase, where LLMs generate neural architectures directly through textual prompts (Zhao et al., 2023; Yu et al., 2023; Wei et al., 2023) . This approach suffers from two key drawbacks: (1) Reproducibility: Stochastic LLM responses hinder consistent results. ( 2 ) Interpretability: Text-based prompts lack clarity on the design rationale, making optimization and trust difficult. Moreover, LLM-to-NAS methods that focus on generating architecture code (Lehman et al., 2024) can only support tiny search spaces and simplified networks, thus exhibiting poorer performance compared to established NAS algorithms on standard NAS benchmarks (Chen et al., 2023) . Furthermore, current LLM-to-NAS algorithms rely on iterative or evolutionary methods, facing high computational costs, since each architecture requires full training for evaluation. To address this, we aim to design a novel LLMto-NAS algorithm that (1) enhances LLMs' understanding of NAS architectures from both text-level and code-level, (2) addresses the time-cost issue in existing LLM-to-NAS methods, and (3) achieves better performance and scalability for broader search spaces and standard benchmarks.