In defense of dual-encoders for neural ranking

Transformer-based models have proven successful in information retrieval problems, which seek to identify relevant documents for a given query. There are two broad flavours of such models: cross-attention (CA) models, which learn a joint query-document embedding, and dual-encoder (DE) models, which learn separate embeddings for the query and the document. Empirically, CA models are often more accurate, which has motivated several works that seek to bridge this performance gap. However, a fundamental question remains less explored: does this gap reflect a limitation in DE models' capacity, or training procedure? In this paper, we study this question, with three contributions. First, we establish theoretically that with a sufficiently large encoder size, DE models can capture a broad class of scores without cross-attention. Second, we show that on real-world problems, the gap between CA and DE models may be due to the latter overfitting to the training set. To mitigate this, we propose a distillation strategy that focuses on preserving the ordering amongst documents, and confirm its efficacy on neural re-ranking benchmarks.