Proxy Learning of Visual Concepts of Fine Art Paintings from Styles through Language Models
Diana Kim, Ahmed Elgammal, Marian Mazzone
Abstract
We present a machine learning system that can quantify fine art paintings with a set of visual elements and principles of art. The formal analysis is fundamental for understanding art, but developing such a system is challenging. Paintings have high visual complexities, but it is also difficult to collect enough training data with direct labels. To resolve these practical limitations, we introduce a novel mechanism, called proxy learning, which learns visual concepts in paintings through their general relation to styles. This framework does not require any visual annotation, but only uses style labels and a general relationship between visual concepts and style. In this paper, we propose a novel proxy model and reformulate four pre-existing methods in the context of proxy learning. Through quantitative and qualitative comparison, we evaluate these methods and compare their effectiveness in quantifying the artistic visual concepts, where the general relationship is estimated by language models; GloVe or BERT. The language modeling is a practical and scalable solution requiring no labeling, but it is inevitably imperfect. We demonstrate how the new proxy model is robust to the imperfection, while the other methods are sensitively affected by it.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Related papers
- Language Quantized AutoEncoders: Towards Unsupervised Text-Image AlignmentHao Liu, Wilson Yan, Pieter AbbeelNeurIPS 2023 · 39 citations
- Learning Visual Proxy for Compositional Zero-Shot LearningShiyu Zhang, Cheng Yan, Yang Liu, Chenchen Jing et al.ICCV 2025 · 1 citation
- Knowledge is Power: Hierarchical-Knowledge Embedded Meta-Learning for Visual Reasoning in Artistic DomainsWenbo Zheng, Lan Yan, Chao Gou, Fei-Yue WangKDD 2021 · 15 citations
- ArtEmis: Affective Language for Visual ArtPanos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny et al.CVPR 2021
- A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated TasksHaorui Yu, Ramon Ruiz-Dolz, Qiufeng YiEMNLP Findings 2025
