  We propose a method for multiple hypothesis testing with familywise error
rate (FWER) control, called the i-FWER test. Most testing methods are
predefined algorithms that do not allow modifications after observing the data.
However, in practice, analysts tend to choose a promising algorithm after
observing the data; unfortunately, this violates the validity of the
conclusion. The i-FWER test allows much flexibility: a human (or a computer
program acting on the human's behalf) may adaptively guide the algorithm in a
data-dependent manner. We prove that our test controls FWER if the analysts
adhere to a particular protocol of "masking" and "unmasking". We demonstrate
via numerical experiments the power of our test under structured non-nulls, and
then explore new forms of masking.
