Lune

EMNLP Findings2024

Characterizing Text Datasets with Psycholinguistic Features

Marcio Monteiro, Charu Karakkaparambil James, Marius Kloft, Sophie Fellenz

Abstract

Fine-tuning pretrained language models on task-specific data is a common practice in Natural Language Processing (NLP) applications. However, the number of pretrained models available to choose from can be very large, and it remains unclear how to select the optimal model without spending considerable amounts of computational resources, especially for the text domain. To address this problem, we introduce PsyMatrix, a novel framework designed to efficiently characterize text datasets. Psy-Matrix evaluates multiple dimensions of text and discourse, producing interpretable, lowdimensional embeddings. Our framework has been tested using a meta-dataset repository that includes the performance of 24 pretrained large language models fine-tuned across 146 classification datasets. Using the proposed embeddings, we successfully developed a metalearning system capable of recommending the most effective pretrained models (optimal and near-optimal) for fine-tuning on new datasets.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How cites Scalable Diverse Model Selection for Accessible Transfer Learning
  • Guided Recommendation for Model Fine-Tuning cites Scalable Diverse Model Selection for Accessible Transfer Learning
  • Landmark-Guided Policy Optimization for Multi-Objective Language Model Selection cites Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Landmark-Guided Policy Optimization for Multi-Objective Language Model SelectionMarcio Monteiro, Weichen Li, Puyu Wang, Marius Kloft et al.ICML 2026

Builds on4

  1. Scalable Diverse Model Selection for Accessible Transfer LearningDaniel Bolya, Rohit Mittapalli, Judy HoffmanNeurIPS 2021 · 61 citations
  2. Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and HowSebastian Pineda-Arango, Fabio Ferreira, Arlind Kadra, Frank Hutter et al.ICLR 2024 · 27 citations
  3. BERT Goes Off-Topic: Investigating the Domain Transfer Challenge using Genre ClassificationDmitri Roussinov, Serge SharoffEMNLP Findings 2023 · 2 citations
  4. Guided Recommendation for Model Fine-TuningHao Li, Charless C. Fowlkes, Hao Yang, Onkar Dabeer et al.CVPR 2023

Related papers

  1. Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language ModelsXinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang et al.ACL 2025 · 15 citations
  2. Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding DimensionsJinsung Yoon, Rajarishi Sinha, Sercan Ömer Arik, Tomas PfisterEMNLP 2024 · 1 citation
  3. Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
  4. Exploring and Predicting Transferability across NLP TasksTu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni et al.EMNLP 2020 · 104 citations
  5. Meta-Task Prompting Elicits Embeddings from Large Language ModelsYibin Lei, Di Wu, Tianyi Zhou, Tao Shen et al.ACL 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines