Lune

ACL Findings2024

GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network

Shuzhou Yuan, Ercong Nie, Michael Färber, Helmut Schmid, Hinrich Schütze

Abstract

Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used. However, fine-tuning still remains crucial to further enhance their adaptability. Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality. We address this issue by introducing a prompt-based parameter-efficient finetuning (PEFT) approach. GNNAVI leverages insights into ICL's information flow dynamics, which indicates that label words act in prompts as anchors for information propagation. GN-NAVI employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN. Our experiments on text classification tasks with GPT-2 and Llama2 show GNNAVI surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. We compare GNNAVI with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency. Our analysis reveals that GNNAVI enhances information flow and ensures a clear aggregation process. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • Fine-Tuning Language Models with Just Forward Passes cites LoRA: Low-Rank Adaptation of Large Language Models
  • Fine-Tuning Language Models with Just Forward Passes cites Language Models are Few-Shot Learners
  • An Explanation of In-context Learning as Implicit Bayesian Inference cites Language Models are Few-Shot Learners
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Graph-Guided Textual Explanation Generation FrameworkShuzhou Yuan, Jingyi Sun, Ran Zhang, Michael Färber et al.EMNLP 2025 · 1 citation

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. An Explanation of In-context Learning as Implicit Bayesian InferenceSang Michael Xie, Aditi Raghunathan, Percy Liang, Tengyu MaICLR 2022 · 1,030 citations
  4. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe et al.EMNLP 2022 · 634 citations
  5. Fine-Tuning Language Models with Just Forward PassesSadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian et al.NeurIPS 2023 · 495 citations

Related papers

  1. All You Need is One: Capsule Prompt Tuning with a Single VectorYiyang Liu, James Liang, Heng Fan, Wenhao Yang et al.NeurIPS 2025 · 16 citations
  2. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
  3. A Study of Parameter Efficient Fine-tuning by Learning to Efficiently Fine-TuneTaha Ceritli, Savas Özkan, Jeongwon Min, Eunchung Noh et al.EMNLP Findings 2024 · 3 citations
  4. From Bottom to Top: Extending the Potential of Parameter Efficient Fine-TuningJihao Gu, Zelin Wang, Yibo Zhang, Ziji Zhang et al.EMNLP 2024 · 3 citations
  5. Prompt Tuning Strikes Back: Customizing Foundation Models with Low-Rank Prompt AdaptationAbhinav Jain, Swarat Chaudhuri, Thomas W. Reps, Christopher M. JermaineNeurIPS 2024 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines