  In the coming years, advanced gravitational wave detectors will observe
signals from a large number of compact binary coalescences. The majority of
these signals will be relatively weak, making the precision measurement of
subtle effects, such as deviations from general relativity, challenging in the
individual events. However, many weak observations can be combined into precise
inferences, if information from the individual signals is combined in an
appropriate way. In this study we revisit common methods for combining multiple
gravitational wave observations to test general relativity, namely (i)
multiplying the individual likelihoods of beyond-general-relativity parameters
and (ii) multiplying the Bayes Factor in favor of general relativity from each
event. We discuss both methods and show that they make stringent assumptions
about the modified theory of gravity they test. In particular, the former
assumes that all events share the same beyond-general-relativity parameter,
while the latter assumes that the theory of gravity has a new unrelated
parameter for each detection. We show that each method can fail to detect
deviations from general relativity when the modified theory being tested
violates these assumptions. We argue that these two methods are the extreme
limits of a more generic framework of hierarchical inference on hyperparameters
that characterize the underlying distribution of single-event parameters. We
illustrate our conclusions first using a simple model of Gaussian likelihoods,
and also by applying parameter estimation techniques to a simulated dataset of
gravitational waveforms in a model where the graviton is massive. We argue that
combining information from multiple sources requires explicit assumptions that
make the results inherently model-dependent.
