  This paper explores using large isomorphic problem banks to overcome many
challenges of traditional exams in large STEM classes, especially the threat of
content sharing websites and generative AI to the security of exam items. We
first introduce an efficient procedure for creating large numbers of isomorphic
physics problems, assisted by the large language model GPT-3 and several other
open-source tools. We then propose that if exam items are randomly drawn from
large enough problem banks, then giving students open access to problem banks
prior to the exam will not dramatically impact students' performance on the
exam or lead to wide-spread rote-memorization of solutions. We tested this
hypothesis on two mid-term physics exams, comparing students' performance on
problems drawn from open isomorphic problem banks to similar transfer problems
that were not accessible to students prior to the exam. We found that on both
exams, both open bank and transfer problems had the highest difficulty. The
differences in percent correct were between 5% to 10%, which is comparable to
the differences between different isomorphic versions of the same problem type.
Item response theory analysis found that both types of problem have high
discrimination (>1.5) with no significant differences. Student performance on
open-bank and transfer problems are highly correlated with each other, and the
correlations are stronger than average correlations between problems on the
exam. Exploratory factor analysis also found that open-bank and transfer
problems load on the same factor, and even formed their own factor on the
second exam. Those observations all suggest that giving students open access to
large isomorphic problem banks only had a small impact on students' performance
on the exam but could have significant potential in reforming traditional
classroom exams.
