Lune

ACL Findings2024

LLM Performance Predictors are good initializers for Architecture Search

Ganesh Jawahar, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Dujian Ding

Abstract

In this work, we utilize Large Language Models (LLMs) for a novel use case: constructing Performance Predictors (PP) that estimate the performance of specific deep neural network architectures on downstream tasks. We create PP prompts for LLMs, comprising (i) role descriptions, (ii) instructions for the LLM, (iii) hyperparameter definitions, and (iv) demonstrations presenting sample architectures with efficiency metrics and 'training from scratch' performance. In machine translation (MT) tasks, GPT-4 with our PP prompts (LLM-PP) achieves a SoTA mean absolute error and a slight degradation in rank correlation coefficient compared to baseline predictors. Additionally, we demonstrate that predictions from LLM-PP can be distilled to a compact regression model (LLM-Distill-PP), which surprisingly retains much of the performance of LLM-PP. This presents a cost-effective alternative for resource-intensive performance estimation. Specifically, for Neural Architecture Search (NAS), we introduce a Hybrid-Search algorithm (HS-NAS) employing LLM-Distill-PP for the initial search stages and reverting to the baseline predictor later. HS-NAS performs similarly to SoTA NAS, reducing search hours by approximately 50%, and in some cases, improving latency, GFLOPs, and model size. The code can be found at: https://github.com/UBC-NLP/llmas.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

12 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge cites SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
  • SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models cites GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers12

  1. CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the EdgeChunlin Tian, Xinpeng Qin, Kahou Tam, Li Li et al.USENIX ATC 2025 · 41 citations
  2. MLZero: A Multi-Agent System for End-to-end Machine Learning AutomationHaoyang Fang, Boran Han, Nick Erickson, Xiyuan Zhang et al.NeurIPS 2025 · 29 citations
  3. Design Principle Transfer in Neural Architecture Search via Large Language ModelsXun Zhou, Xingyu Wu, Liang Feng, Zhichao Lu et al.AAAI 2025 · 24 citations
  4. LLM-Pilot: Characterize and Optimize Performance of your LLM Inference ServicesMalgorzata Lazuka, Andreea Anghel, Thomas P. ParnellSC 2024 · 17 citations
  5. Blackbox Model Provenance via Palimpsestic Membership InferenceRohith Kuditipudi, Jing Huang, Sally Zhu, Diyi Yang et al.NeurIPS 2025 · 12 citations

Builds on10

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu et al.ICML 2023 · 1,493 citations
  3. GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
  4. HAT: Hardware-Aware Transformers for Efficient Natural Language ProcessingHanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai et al.ACL 2020 · 215 citations
  5. NASViT: Neural Architecture Search for Efficient Vision Transformers with Gradient Conflict aware Supernet TrainingChengyue Gong, Dilin Wang, Meng Li, Xinlei Chen et al.ICLR 2022 · 114 citations

Related papers

  1. RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost StrategyZipeng Ji, Guanghui Zhu, Chunfeng Yuan, Yihua HuangICML 2025
  2. RankNAS: Efficient Neural Architecture Search by Pairwise RankingChi Hu, Chenglong Wang, Xiangnan Ma, Xia Meng et al.EMNLP 2021 · 10 citations
  3. Revolutionizing Training-Free NAS: Towards Efficient Automatic Proxy Discovery via Large Language ModelsHaidong Kang, Lihong Lin, Hanling WangNeurIPS 2025 · 3 citations
  4. Generic Neural Architecture Search via RegressionYuhong Li, Cong Hao, Pan Li, Jinjun Xiong et al.NeurIPS 2021 · 40 citations
  5. LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical EncodingYuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zheng et al.EMNLP 2025
Dusk over the sea between two cliffs drawn in fine vertical lines