  AI approaches are progressing besting humans at game-related tasks (e.g.
chess). The next stage is expected to be Human-AI collaboration; however, the
research on this subject has been mixed and is in need of additional data
points. We add to this nascent literature by studying Human-AI collaboration on
a common administrative educational task. Education is a special domain in its
relation to AI and has been slow to adopt AI approaches in practice, concerned
with the educational enterprise losing its humanistic touch and because
standard of quality is demanded because of the impact on a person's career and
developmental trajectory. In this study (N = 22), we design an experiment to
explore the effect of Human-AI collaboration on the task of tagging educational
content with skills from the US common core taxonomy. Our results show that the
experiment group (with AI recommendations) saved around 50% time (p < 0.01) in
the execution of their tagging task but at the sacrifice of 7.7% recall (p =
0.267) and 35% accuracy (p= 0.1170) compared with the non-AI involved control
group, placing the AI+human group in between the AI alone (lowest performance)
and the human alone (highest performance). We further analyze log data from
this AI collaboration experiment to explore under what circumstances humans
still exercised their discernment when receiving recommendations. Finally, we
outline how this study can assist in implementing AI tools, like ChatGPT, in
education.
