  Physics education researchers (PER) often analyze student data with
single-level regression models (e.g., linear and logistic regression). However,
education datasets can have hierarchical structures, such as students nested
within courses, that single-level models fail to account for. The improper use
of single-level models to analyze hierarchical datasets can lead to biased
findings. Hierarchical models (a.k.a., multi-level models) account for this
hierarchical nested structure in the data. In this publication, we outline the
theoretical differences between how single-level and multi-level models handle
hierarchical datasets. We then present analysis of a dataset from 112
introductory physics courses using both multiple linear regression and
hierarchical linear modeling to illustrate the potential impact of using an
inappropriate analytical method on PER findings and implications. Research can
leverage multi-institutional datasets to improve the field's understanding of
how to support student success in physics. There is no post hoc fix, however,
if researchers use inappropriate single-level models to analyze multi-level
datasets. To continue developing reliable and generalizable knowledge, PER
should use hierarchical models when analyzing hierarchical datasets. The
supplemental materials include a sample dataset, R code to model the building
and analysis presented in the paper, and an HTML output from the R code.
