  Two studies of a human-AI collaborative design tool were carried out in order
to understand the influence design recommendations have on the design process.
The tool investigated is based on an evolutionary algorithm attempting to
design a virtual car to travel as far as possible in a fixed time. Participants
were able to design their own cars, make recommendations to the algorithm and
view sets of recommendations from the algorithm. The algorithm-recommended sets
were designs which had been previously tested; some sets were simply randomly
picked and other sets were picked using MAP-Elites. In the first study 808
design sessions were recorded as part of a science outreach program, each with
analytical data of how each participant used the tool. To provide context to
this quantitative data, a smaller double-blind lab study was also carried out
with 12 participants. In the lab study the same quantitative data from the
large scale study was collected alongside responses to interview questions.
Although there is some evidence that the MAP-Elites provide higher-quality
individual recommendations, neither study provides convincing evidence that
these recommendations have a more positive influence on the design process than
simply a random selection of designs. In fact, it seems that providing a
combination of MAP-Elites and randomly selected recommendations is beneficial
to the process. Furthermore, simply viewing recommendations from the MAP-Elites
had a positive influence on engagement in the design task and the quality of
the final design produced. Our findings are significant both for researchers
designing new mixed-initiative tools, and those who wish to evaluate existing
tools. Most significantly, we found that metrics researchers currently use to
evaluate the success of human-AI collaborative algorithms do not measure the
full influence these algorithms have on the design process.
