  This study aims at improving the performance of scoring student responses in
science education automatically. BERT-based language models have shown
significant superiority over traditional NLP models in various language-related
tasks. However, science writing of students, including argumentation and
explanation, is domain-specific. In addition, the language used by students is
different from the language in journals and Wikipedia, which are training
sources of BERT and its existing variants. All these suggest that a
domain-specific model pre-trained using science education data may improve
model performance. However, the ideal type of data to contextualize pre-trained
language model and improve the performance in automatically scoring student
written responses remains unclear. Therefore, we employ different data in this
study to contextualize both BERT and SciBERT models and compare their
performance on automatic scoring of assessment tasks for scientific
argumentation. We use three datasets to pre-train the model: 1) journal
articles in science education, 2) a large dataset of students' written
responses (sample size over 50,000), and 3) a small dataset of students'
written responses of scientific argumentation tasks. Our experimental results
show that in-domain training corpora constructed from science questions and
responses improve language model performance on a wide variety of downstream
tasks. Our study confirms the effectiveness of continual pre-training on
domain-specific data in the education domain and demonstrates a generalizable
strategy for automating science education tasks with high accuracy. We plan to
release our data and SciEdBERT models for public use and community engagement.
