Optimal testing using combined test statistics across independent studies
Lasse Vuursteen, Botond Szabó, Aad van der Vaart, Harry van Zanten
Abstract
Combining test statistics from independent trials or experiments is a popular method of meta-analysis. However, there is very limited theoretical understanding of the power of the combined test, especially in high-dimensional models considering composite hypotheses tests. We derive a mathematical framework to study standard meta-analysis testing approaches in the context of the many normal means model, which serves as the platform to investigate more complex models. We introduce a natural and mild restriction on the meta-level combination functions of the local trials. This allows us to mathematically quantify the cost of compressing trials into real-valued test statistics and combining these. We then derive minimax lower and matching upper bounds for the separation rates of standard combination methods for e.g. p-values and e-values, quantifying the loss relative to using the full, pooled data. We observe an elbow effect, revealing that in certain cases combining the locally optimal tests in each trial results in a sub-optimal meta-analysis method and develop approaches to achieve the global optima. We also explore the possible gains of allowing limited coordination between the trial designs. Our results connect meta-analysis with bandwidth constraint distributed inference and build on recent information theoretic developments in the latter field.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Related papers
- Unified Lower Bounds for Interactive High-dimensional Estimation under Information ConstraintsJayadev Acharya, Clément L. Canonne, Ziteng Sun, Himanshu TyagiNeurIPS 2023 · 35 citations
- Simultaneous Confidence Bounds for Aggregated Effects via Exact Subset OptimizationWeihang Xu, Huajie Qian, Wotao Yin, Xinshang WangICML 2026
- Beyond the Best: Distribution Functional Estimation in Infinite-Armed BanditsYifei Wang, Tavor Z. Baharav, Yanjun Han, Jiantao Jiao et al.NeurIPS 2022 · 2 citations
- Theoretical bounds on estimation error for meta-learningJames Lucas, Mengye Ren, Irene Raissa Kameni, Toniann Pitassi et al.ICLR 2021 · 12 citations
- Efficient Ensemble Conditional Independence Test Framework for Causal DiscoveryZhengkang Guan, Kun KuangICLR 2026 · 6 citations
