Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
Malte Ostendorff, Nils Rethmeier, Isabelle Augenstein, Béla Gipp, Georg Rehm
German Research Centre for Artificial Intelligence University of Göttingen University of Copenhagen
阅读操作
确认中在文库中上传 PDF 后可生成中文音频讲解。
摘要与影响
Learning scientific document representations can be substantially improved through contrastive learning objectives, where the challenge lies in creating positive and negative training samples that encode the desired similarity semantics.Prior work relies on discrete citation relations to generate contrast samples.However, discrete citations enforce a hard cutoff to similarity.This is counter-intuitive to similarity-based learning and ignores that scientific papers can be very similar despite lacking a direct citation -a core problem of finding related research.Instead, we use controlled nearest neighbor sampling over citation graph embeddings for contrastive learning.This control allows us to learn continuous similarity, to sample hard-to-learn negatives and positives, and also to avoid collisions between negative and positive samples by controlling the sampling margin between them.The resulting method SciNCL outperforms the state-of-theart on the SciDocs benchmark.Furthermore, we demonstrate that it can train (or tune) language models sample-efficiently and that it can be combined with recent training-efficient methods.Perhaps surprisingly, even training a general-domain language model this way outperforms baselines pretrained in-domain.
逐年被引趋势
关键指标
同类平均 = 1
同领域 · 同年份 · 同类型
Google Scholar 与 OpenAlex 的被引统计范围不同,数值存在差异属正常。
AI 辅助阅读
依据:摘要
可就本文提问;依据不足时会说明。
学术脉络
学科主题
计算机 / AITopic Modeling
Advanced Graph Neural Networks · Recommender Systems and Techniques