EMNLP Findings2023
Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models
Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, Heng Ji
Abstract
The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns. High-performing open-source alternatives are crucial for information-sensitive and high-volume applications but often lag behind in performance. To address this gap, we propose (1) A untargeted variant of iterative selfcritique and self-refinement devoid of external influence. (2) A novel ranking metric -Performance, Refinement, and Inference Cost Score (PeRFICS) -to find the optimal model for a given task considering refined performance and cost. Our experiments show that SoTA open source models of varying sizes from 7B -65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks on the Vicuna benchmark. Vicuna-13B takes it a step further and outperforms ChatGPT postrefinement. This work has profound implications for resource-constrained and informationsensitive environments seeking to leverage LLMs without incurring prohibitive costs, compromising on performance and privacy. The domain-agnostic self-refinement process coupled with our novel ranking metric facilitates informed decision-making in model selection, thereby reducing costs and democratizing access to high-performing language models, as evidenced by case studies.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- DiscoveryBench: Towards Data-Driven Discovery with Large Language Models cites Reflexion: language agents with verbal reinforcement learning
- QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
- QLoRA: Efficient Finetuning of Quantized LLMs cites Measuring Massive Multitask Language Understanding
- QLoRA: Efficient Finetuning of Quantized LLMs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
- Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models cites Measuring Massive Multitask Language Understanding
- Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Cited by top-tier papers3
- LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi FengNeurIPS 2024 · 865 citations
- DiscoveryBench: Towards Data-Driven Discovery with Large Language ModelsBodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra et al.ICLR 2025
- Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language ModelsKyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee et al.ACL 2025
Builds on11
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
- QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
- Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations
Related papers
- The False Promise of Imitating Proprietary Language ModelsArnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng et al.ICLR 2024 · 30 citations
- InstructZero: Efficient Instruction Optimization for Black-Box Large Language ModelsLichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang et al.ICML 2024 · 64 citations
- Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
- Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain SetupsRazvan-Alexandru Smadu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop et al.EMNLP 2024 · 4 citations
- Benchmarking Large Language Models on Controllable Generation under Diversified InstructionsYihan Chen, Benfeng Xu, Quan Wang, Yi Liu et al.AAAI 2024 · 42 citations
