  Gravitational waves in general relativity contain two polarization degrees of
freedom, commonly labeled plus and cross. Besides those two tensor modes,
generic theories of gravity predict up to four additional polarization modes:
two scalar and two vector. Detection of nontensorial modes in gravitational
wave data would constitute a clean signature of physics beyond general
relativity. Previous measurements have pointed to the unambiguous presence of
tensor modes in gravitational waves, but the presence of additional generic
nontensorial modes has not been directly tested. We propose a model-independent
analysis capable of detecting and characterizing mixed tensor and nontensor
components in transient gravitational wave signals, including those from
compact binary coalescences. This infrastructure can constrain the presence of
scalar or vector polarization modes on top of the tensor modes predicted by
general relativity. Our analysis is morphology-independent (as it does not rely
on a waveform templates), phase-coherent, and agnostic about the source sky
location. We apply our analysis to data from GW190521 and simulated data and
demonstrate that it is capable of placing upper limits on the strength of
nontensorial modes when none are present, or characterizing their morphology in
the case of a positive detection. Tests of the polarization content of a
transient gravitational wave signal hinge on an extended detector network,
wherein each detector observes a different linear combination of polarization
modes. We therefore anticipate that our analysis will yield precise
polarization constraints in the coming years, as the current ground-based
detectors LIGO Hanford, LIGO Livingston, and Virgo are joined by KAGRA and LIGO
India.
