Detecting Interference using Dyadic Data in Online Controlled Experiments

Interference in online controlled experiments poses significant challenges, as treatment effects may propagate through user interactions. This phenomenon systematically biases conventional treatment effect estimators and substantially complicates the interpretation of experimental results. Detecting the presence of interference is crucial for ensuring the reliability of conclusions, ultimately enhancing decision-making processes. We formalize a framework for detecting the existence of interference with dyadic data, which is ubiquitous in social network platforms. Dyadic data, typically generated from user interactions, have not been fully utilized in online controlled experiments. We first introduce the dyadic-robust inference techniques, then illustrate how the dyadic outcomes link to the user-level metrics and how interference manifests. We then propose a comprehensive approach to detect interference effects based on dyadic-robust t-statistics, and we also provide a subsampling approach to reduce the computational complexity of the proposed methods. We also provide several approaches to leverage pre-experiment data, such as selecting the optimal treatment probability, designing stratified randomization in the pre-experiment stage, or reducing the variance with both user-level and dyad-level control variates at the post-experiment stage. Our approach is computationally efficient and easily integrated with existing pipelines. We provide theoretical guarantees for our inference procedure across various asymptotic regimes. Finally, we conduct several empirical evaluations under various dyadic data-generating processes to assess the performance and robustness of the proposed procedures. Finally, we introduce an interference detection pipeline and demonstrate it with an experiment from Weixin.