Remote Sensing Retrieval-Augmented Generation: Bridging remote sensing imagery and comprehensive knowledge with a multimodal dataset and retrieval-augmented generation model

Recent progress in vision-language models (VLMs) has demonstrated impressive capabilities across a variety of tasks in the natural image domain. Motivated by these advancements, the remote sensing community has begun to adopt VLMs for remote sensing vision-language tasks, including image captioning, scene understanding, and visual question answering (VQA). However, existing remote sensing VLMs typically rely on closed-set scene understanding and focus on generic scene descriptions, yet lack the ability to incorporate external knowledge. This limitation hinders their capacity for semantic reasoning over complex or context-dependent queries that involve domain-specific or world knowledge. To address these challenges, we first introduced a multimodal remote sensing world knowledge (RSWK) dataset, which comprises high-resolution satellite imagery and detailed textual descriptions for 14,820 well-known landmarks across 16 categories from 184 countries, integrating both remote sensing domain knowledge and broader world knowledge. Building upon this dataset, we proposed a novel remote sensing retrieval-augmented generation (RS-RAG) framework, which consists of two key components. The multimodal knowledge vector database (MKVD) construction module encodes remote sensing imagery and associated textual knowledge into a unified vector space. The knowledge retrieval and response generation module retrieves and reranks relevant knowledge based on image and/or text queries and incorporates the retrieved content into a knowledge-augmented prompt to guide the VLM in producing contextually grounded responses. We validate the effectiveness of RS-RAG on our designed benchmark covering three vision-language tasks: image captioning, image classification, and VQA, where RS-RAG significantly outperformed state-of-the-art baselines. By bridging remote sensing imagery and comprehensive knowledge, RS-RAG empowers remote sensing VLMs with enhanced contextual reasoning, enabling them to generate more accurate, informative, and semantically grounded outputs across a wide range of tasks.