  While other fields such as statistics and education have examined various
issues with quantitative work, few studies in physics education research (PER)
have done so. We conducted a two-phase study to identify and to understand the
extent of these issues in quantitative PER . During Phase 1, we conducted a
focus group of three experts in this area, followed by six interviews.
Subsequent interviews refined our plan. Both the focus group and interviews
revealed issues regarding the lack of details in sample descriptions, lack of
institutional/course contextual information, lack of reporting on limitation,
and overgeneralization or overstatement of conclusions. During Phase 2, we
examined 72 manuscripts that used four conceptual or attitudinal assessments
(Force Concept Inventory, Conceptual Survey of Electricity and Magnetism, Brief
Electricity and Magnetism Assessment, and Colorado Learning Attitudes about
Science Survey). Manuscripts were coded on whether they featured various sample
descriptions, institutional/course context information, limitations, and
whether they overgeneralized conclusions. We also analyzed the data to see if
reporting has changed from the earlier periods to more recent times. We found
that not much has changed regarding sample descriptions and
institutional/course context information, but reporting and overgeneralizing
conclusions has improved over time. We offer some questions for researchers,
reviewers, and readers in PER to consider when conducting or using quantitative
work.
