  The Turing test for comparing computer performance to that of humans is well
known, but, surprisingly, there is no widely used test for comparing how much
better human-computer systems perform relative to humans alone, computers
alone, or other baselines. Here, we show how to perform such a test using the
ratio of means as a measure of effect size. Then we demonstrate the use of this
test in three ways. First, in an analysis of 79 recently published experimental
results, we find that, surprisingly, over half of the studies find a decrease
in performance, the mean and median ratios of performance improvement are both
approximately 1 (corresponding to no improvement at all), and the maximum ratio
is 1.36 (a 36% improvement). Second, we experimentally investigate whether a
higher performance improvement ratio is obtained when 100 human programmers
generate software using GPT-3, a massive, state-of-the-art AI system. In this
case, we find a speed improvement ratio of 1.27 (a 27% improvement). Finally,
we find that 50 human non-programmers using GPT-3 can perform the task about as
well as--and less expensively than--the human programmers. In this case,
neither the non-programmers nor the computer would have been able to perform
the task alone, so this is an example of a very strong form of human-computer
synergy.
