  Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable
information about model performance and behavior. Traditionally, IRT models are
learned using human response pattern (RP) data, presenting a significant
bottleneck for large data sets like those required for training deep neural
networks (DNNs). In this work we propose learning IRT models using RPs
generated from artificial crowds of DNN models. We demonstrate the
effectiveness of learning IRT models using DNN-generated data through
quantitative and qualitative analyses for two NLP tasks. Parameters learned
from human and machine RPs for natural language inference and sentiment
analysis exhibit medium to large positive correlations. We demonstrate a
use-case for latent difficulty item parameters, namely training set filtering,
and show that using difficulty to sample training data outperforms baseline
methods. Finally, we highlight cases where human expectation about item
difficulty does not match difficulty as estimated from the machine RPs.
