  In education, health, and human services, an intervention program is usually
implemented by many local organizations. Determining which organizations are
more effective is essential for theoretically characterizing effective
practices and for intervening to enhance the capacity of ineffective
organizations. In multisite randomized trials, site-specific intention-to-treat
(ITT) effects are likely invalid indicators for organizational effectiveness
and may lead to inequitable decisions. This is because sites differ in their
local ecological conditions including client composition, alternative programs,
and community context. Applying the potential outcomes framework, this study
proposes a mathematical definition for the relative effectiveness of an
organization. The estimand contrasts the performance of a focal organization
with those that share the features of its local ecological conditions. The
identification relies on relatively weak assumptions by leveraging observed
control group outcomes that capture the confounding impacts of alternative
programs and community context. We propose a two-step mixed-effects modeling
(2SME) procedure. Simulations demonstrate significant improvements when
compared with site-specific ITT analyses or analyses that only adjust for
between-site differences in the observed baseline participant composition. We
illustrate its use through an evaluation of the relative effectiveness of
individual Job Corps centers by reanalyzing data from the National Job Corps
Study, a multisite randomized trial that included 100 Job Corps centers
nationwide serving disadvantaged youths. The new strategy promises to alleviate
consequential misclassifications of some of the most effective Job Corps
centers as least effective and vice versa.
