Visually Consistent Hierarchical Image Classification
Seulki Park, Youren Zhang, Stella X. Yu, Sara Beery, Jonathan Huang
Abstract
Hierarchical classification predicts labels across multiple levels of a taxonomy, e.g., from coarse-level Bird to mid-level Hummingbird to fine-level Green hermit, allowing flexible recognition under varying visual conditions. It is commonly framed as multiple single-level tasks, but each level may rely on different visual cues: Distinguishing Bird from Plant relies on global features like feathers or leaves, while separating Anna's hummingbird from Green hermit requires local details such as head coloration. Prior methods improve accuracy using external semantic supervision, but such statistical learning criteria fail to ensure consistent visual grounding at test time, resulting in incorrect hierarchical classification. We propose, for the first time, to enforce internal visual consistency by aligning fineto-coarse predictions through intra-image segmentation. Our method outperforms zero-shot CLIP and state-of-the-art baselines on hierarchical classification benchmarks, achieving both higher accuracy and more consistent predictions. It also improves internal image segmentation without requiring pixel-level annotations.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- Free-Grained Hierarchical Visual Recognition cites The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition
- Free-Grained Hierarchical Visual Recognition cites Learning Transferable Visual Models From Natural Language Supervision
- RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs cites The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition
- RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs cites Learning Transferable Visual Models From Natural Language Supervision
- Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models cites The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition
- Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers6
- The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual RecognitionYuwen Tan, Yuan Qing, Boqing GongCVPR 2026 · 6 citations
- Free-Grained Hierarchical Visual RecognitionSeulki Park, Zilin Wang, Stella X. YuCVPR 2026 · 3 citations
- Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal ModelsHulingxiao He, Zhi Tan, Yuxin PengCVPR 2026 · 3 citations
- How does Chain of Thought decompose complex tasks?Amrut Nadgir, Vijay Balasubramanian, Pratik ChaudhariICML 2026 · 1 citation
- RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMsLogan Lawrence, Oindrila Saha, Rangel Daroya, Mustafa Chasmai et al.CVPR 2026
Builds on25
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Training data-efficient image transformers & distillation through attentionHugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa et al.ICML 2021 · 8,974 citations
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable FeaturesSangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh et al.ICCV 2019 · 5,843 citations
- RandAugment: Practical Automated Data Augmentation with a Reduced Search SpaceEkin Dogus Cubuk, Barret Zoph, Jonathon Shlens, Quoc LeNeurIPS 2020 · 4,453 citations
- BREEDS: Benchmarks for Subpopulation ShiftShibani Santurkar, Dimitris Tsipras, Aleksander MadryICLR 2021 · 193 citations
Related papers
- ProTeCt: Prompt Tuning for Taxonomic Open Set ClassificationTz-Ying Wu, Chih-Hui Ho, Nuno VasconcelosCVPR 2024 · 3 citations
- Unsupervised Hierarchical Semantic Segmentation with Multiview Cosegmentation and Clustering TransformersTsung-Wei Ke, Jyh-Jing Hwang, Yunhui Guo, Xudong Wang et al.CVPR 2022 · 34 citations
- Consistency-aware Feature Learning for Hierarchical Fine-grained Visual ClassificationRui Wang, Cong Zou, Weizhong Zhang, Zixuan Zhu et al.ACM MM 2023 · 6 citations
- Language-Driven Multi-Label Zero-Shot Learning with Semantic GranularityShouwen Wang, Qian Wan, Junbin Gao, Zhigang ZengICCV 2025 · 2 citations
- Zero-shot Referring Image Segmentation with Global-Local Context FeaturesSeonghoon Yu, Paul Hongsuck Seo, Jeany SonCVPR 2023
