Visually Consistent Hierarchical Image Classification

Hierarchical classification predicts labels across multiple levels of a taxonomy, e.g., from coarse-level Bird to mid-level Hummingbird to fine-level Green hermit, allowing flexible recognition under varying visual conditions. It is commonly framed as multiple single-level tasks, but each level may rely on different visual cues: Distinguishing Bird from Plant relies on global features like feathers or leaves, while separating Anna's hummingbird from Green hermit requires local details such as head coloration. Prior methods improve accuracy using external semantic supervision, but such statistical learning criteria fail to ensure consistent visual grounding at test time, resulting in incorrect hierarchical classification. We propose, for the first time, to enforce internal visual consistency by aligning fineto-coarse predictions through intra-image segmentation. Our method outperforms zero-shot CLIP and state-of-the-art baselines on hierarchical classification benchmarks, achieving both higher accuracy and more consistent predictions. It also improves internal image segmentation without requiring pixel-level annotations.