Lune

EMNLP Findings2023

Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models

Sumuk Shashidhar, Abhinav Chinta, Vaibhav Sahai, Zhenhailong Wang, Heng Ji

Abstract

The dominance of proprietary LLMs has led to restricted access and raised information privacy concerns. High-performing open-source alternatives are crucial for information-sensitive and high-volume applications but often lag behind in performance. To address this gap, we propose (1) A untargeted variant of iterative selfcritique and self-refinement devoid of external influence. (2) A novel ranking metric -Performance, Refinement, and Inference Cost Score (PeRFICS) -to find the optimal model for a given task considering refined performance and cost. Our experiments show that SoTA open source models of varying sizes from 7B -65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks on the Vicuna benchmark. Vicuna-13B takes it a step further and outperforms ChatGPT postrefinement. This work has profound implications for resource-constrained and informationsensitive environments seeking to leverage LLMs without incurring prohibitive costs, compromising on performance and privacy. The domain-agnostic self-refinement process coupled with our novel ranking metric facilitates informed decision-making in model selection, thereby reducing costs and democratizing access to high-performing language models, as evidenced by case studies.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DiscoveryBench: Towards Data-Driven Discovery with Large Language Models cites Reflexion: language agents with verbal reinforcement learning
  • QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • QLoRA: Efficient Finetuning of Quantized LLMs cites Measuring Massive Multitask Language Understanding
  • QLoRA: Efficient Finetuning of Quantized LLMs cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models cites LoRA: Low-Rank Adaptation of Large Language Models
  • Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language Models cites Measuring Massive Multitask Language Understanding
  • Reflexion: language agents with verbal reinforcement learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. LLM Evaluators Recognize and Favor Their Own GenerationsArjun Panickssery, Samuel R. Bowman, Shi FengNeurIPS 2024 · 865 citations
  2. DiscoveryBench: Towards Data-Driven Discovery with Large Language ModelsBodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra et al.ICLR 2025
  3. Plug-in and Fine-tuning: Bridging the Gap between Small Language Models and Large Language ModelsKyeonghyun Kim, Jinhee Jang, Juhwan Choi, Yoonji Lee et al.ACL 2025

Builds on11

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  5. Reflexion: language agents with verbal reinforcement learningNoah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan et al.NeurIPS 2023 · 5,828 citations

Related papers

  1. The False Promise of Imitating Proprietary Language ModelsArnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng et al.ICLR 2024 · 30 citations
  2. InstructZero: Efficient Instruction Optimization for Black-Box Large Language ModelsLichang Chen, Jiuhai Chen, Tom Goldstein, Heng Huang et al.ICML 2024 · 64 citations
  3. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations
  4. Investigating Large Language Models for Complex Word Identification in Multilingual and Multidomain SetupsRazvan-Alexandru Smadu, David-Gabriel Ion, Dumitru-Clementin Cercel, Florin Pop et al.EMNLP 2024 · 4 citations
  5. Benchmarking Large Language Models on Controllable Generation under Diversified InstructionsYihan Chen, Benfeng Xu, Quan Wang, Yi Liu et al.AAAI 2024 · 42 citations
Dusk over the sea between two cliffs drawn in fine vertical lines