EMNLP Findings2022
SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation
Zekun Li, Jina Kim, Yao-Yi Chiang, Muhao Chen
Abstract
Named geographic entities (geo-entities for short) are the building blocks of many geographic datasets. Characterizing geo-entities is integral to various application domains, such as geo-intelligence and map comprehension, while a key challenge is to capture the spatialvarying context of an entity. We hypothesize that we shall know the characteristics of a geoentity by its surrounding entities, similar to knowing word meanings by their linguistic context. Accordingly, we propose a novel spatial language model, SPABERT ( ), which provides a general-purpose geo-entity representation based on neighboring entities in geospatial data. SPABERT extends BERT to capture linearized spatial context, while incorporating a spatial coordinate embedding mechanism to preserve spatial relations of entities in the 2-dimensional space. SPABERT is pretrained with masked language modeling and masked entity prediction tasks to learn spatial dependencies. We apply SPABERT to two downstream tasks: geo-entity typing and geoentity linking. Compared with the existing language models that do not use spatial context, SPABERT shows significant performance improvement on both tasks. We also analyze the entity representation from SPABERT in various settings and the effect of spatial coordinate embedding.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
11 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding cites VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding cites SimCSE: Simple Contrastive Learning of Sentence Embeddings
- A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery cites GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
- CityGPT: Empowering Urban Spatial Cognition of Large Language Models cites GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
- MoRA: Mobility as the Backbone for Geospatial Representation Learning at Scale cites ReFound: Crafting a Foundation Model for Urban Region Understanding upon Language and Visual Foundations
- MoRA: Mobility as the Backbone for Geospatial Representation Learning at Scale cites GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding
- MoRA: Mobility as the Backbone for Geospatial Representation Learning at Scale cites CityGPT: Empowering Urban Spatial Cognition of Large Language Models
Cited by top-tier papers11
- A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific DiscoveryYu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang et al.EMNLP 2024 · 28 citations
- GeoLM: Empowering Language Models for Geospatially Grounded Language UnderstandingZekun Li, Wenxuan Zhou, Yao-Yi Chiang, Muhao ChenEMNLP 2023 · 25 citations
- ReFound: Crafting a Foundation Model for Urban Region Understanding upon Language and Visual FoundationsCongxi Xiao, Jingbo Zhou, Yixiong Xiao, Jizhou Huang et al.KDD 2024 · 18 citations
- CityGPT: Empowering Urban Spatial Cognition of Large Language ModelsJie Feng, Tianhui Liu, Yuwei Du, Siqi Guo et al.KDD 2025 · 9 citations
- MoRA: Mobility as the Backbone for Geospatial Representation Learning at ScaleYa Wen, Jixuan Cai, Qiyao Ma, Linyan Li et al.ICLR 2026 · 5 citations
Builds on18
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- ViLT: Vision-and-Language Transformer Without Convolution or Region SupervisionWonjae Kim, Bokyung Son, Ildoo KimICML 2021 · 2,258 citations
- VL-BERT: Pre-training of Generic Visual-Linguistic RepresentationsWeijie Su, Xizhou Zhu, Yue Cao, Bin Li et al.ICLR 2020 · 1,825 citations
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and ComprehensionMike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad et al.ACL 2020 · 1,224 citations
Related papers
- Unaligned Message-Passing and Contextualized-Pretraining for Robust Geo-Entity ResolutionYuwen Ji, Wenbo Xie, Jiaqi Zhang, Chao Wang et al.AAAI 2025
- Geospatial Entity ResolutionPasquale Balsebre, Dezhong Yao, Gao Cong, Zhen HaiWWW 2022 · 21 citations
- Geo-BERT Pre-training Model for Query Rewriting in POI SearchXiao Liu, Juan Hu, Qi Shen, Huan ChenEMNLP Findings 2021 · 16 citations
- LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentionIkuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda et al.EMNLP 2020 · 562 citations
- Semantics-Aware BERT for Language UnderstandingZhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li et al.AAAI 2020 · 396 citations
