Lune

CVPR2024Top-tier venue

ParameterNet: Parameters are All You Need for Large-Scale Visual Pretraining of Mobile Networks

Kai Han, Yunhe Wang, Jianyuan Guo, Enhua Wu

Abstract

The large-scale visual pretraining has significantly improve the performance of large vision models. However, we observe the low FLOPs pitfall that the existing low-FLOPs models cannot benefit from large-scale pretraining. In this paper, we introduce a novel design principle, termed Pa-rameterNet, aimed at augmenting the number of parameters in large-scale visual pretraining models while minimizing the increase in FLOPs. We leverage dynamic convolutions to incorporate additional parameters into the networks with only a marginal rise in FLOPs. The Param-eterNet approach allows low-FLOPs networks to take advantage of large-scale visual pretraining. Furthermore, we extend the ParameterNet concept to the language domain to enhance inference results while preserving inference speed. Experiments on the large-scale ImageNet-22K have shown the superiority of our ParameterNet scheme. For example, ParameterNet-600M can achieve higher accuracy than the widely-used Swin Transformer (81.6% vs. 80.9%) and has much lower FLOPs (0.6G vs. 4.5G). The code will be released at https://parameternet.github.io/ .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites A Simple Framework for Contrastive Learning of Visual Representations
  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale cites Language Models are Few-Shot Learners
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Swin Transformer: Hierarchical Vision Transformer using Shifted Windows cites Training data-efficient image transformers & distillation through attention
  • Training data-efficient image transformers & distillation through attention cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. CL-MFAP: A Contrastive Learning-Based Multimodal Foundation Model for Molecular Property Prediction and Antibiotic ScreeningGen Zhou, Sugitha Janarthanan, Yutong Lu, Pingzhao HuICLR 2025

Builds on33

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsZe Liu, Yutong Lin, Yue Cao, Han Hu et al.ICCV 2021 · 31,683 citations
  3. A Simple Framework for Contrastive Learning of Visual RepresentationsTing Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey E. HintonICML 2020 · 24,064 citations
  4. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  5. Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations

Related papers

  1. Parameter-efficient is not Sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense PredictionsDongshuo Yin, Xueting Han, Bin Li, Hao Feng et al.ACM MM 2024 · 18 citations
  2. Run, Don't Walk: Chasing Higher FLOPS for Faster Neural NetworksJierun Chen, Shiu-Hong Kao, Hao He, Weipeng Zhuo et al.CVPR 2023
  3. Parallel Scaling Law for Language ModelsMouxiang Chen, Binyuan Hui, Zeyu Cui, Jiaxi Yang et al.NeurIPS 2025 · 33 citations
  4. Evolving Subnetwork Training for Large Language ModelsHanqi Li, Lu Chen, Da Ma, Zijian Wu et al.ICML 2024 · 2 citations
  5. VeCAF: Vision-language Collaborative Active Finetuning with Training Objective AwarenessRongyu Zhang, Zefan Cai, Huanrui Yang, Zidong Liu et al.ACM MM 2024 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines