Online Multiple Hypothesis Testing

The vast majority of implementations of online method for FDR control are either part of an experimental setup, that does not straight-forwardly generalize towards applications outside this setup, or are geared towards tests for which all test results are already available (i.e. they do not have an actual online API).

For that reason, this repository implements a wide range of methods for FDR/FWER control for actual online multiple hypothesis testing with an intuitive test_one() method:

• Alpha Spending (Bonferroni, ...)
• Online Fallback
• [Generalized] Alpha Investing (Foster/Stine, ...)
• LOND (Original, Modified, Dependent)
• LORD (LORD3, LOND++, D-LORD, Dependent, DecayLORD)
• SAFFRON (Standard, DecaySAFFRON)
• ADDIS (Standard, DecayADDIS)
• Batch-BH and Batch-StBH

Instantiate an online testing procedure (e.g. Addis()) and simply test p-values sequentially with .test_one():

from online_fdr.investing.addis.addis import Addis
from online_fdr.utils.generation import DataGenerator, StandardGaussianProcess

N = 100
generator = DataGenerator(n=N, contamination=0.1, dgp=StandardGaussianProcess())

addis = Addis(alpha=0.05, wealth=0.025, lambda_=0.25, tau=0.5)  # procedure

for i in range(0, N):
    p_value, label = generator.sample_one()
    result = addis.test_one(p_value)  # sequential testing

This work is inspired by the R package 'onlineFDR'. This package, and most of its methods, are largely validated by the implementations of said package. Key differentiator is the design choice in regard to method calls for sequential testing, as this implementation allows for truly temporal applications ('onlineFDR' requires a [static] data.frame for testing).

Getting started

The code does not require external dependencies. It's recommended to use with Python 3.12, although previous versions (at least until 3.9) should generally be compatible.