Participations in this summer’s conferences has really got me thinking about a many thing about physics education research. Here I’m going to begin writing about one of those things. To start this conversation, I want to talk about a poster from the 2013 Physics Education Research Conference.
Steve Kanim from New Mexico State University
Steve analyzed Physics Education Research publications from the American Journal of Physics and Physical Review–Special Topic in PER. The analysis was limited to publications that included actual student data (i.e., no discussions, opinions, sharing of best practices, dissemination only papers, etc). Steve finds that 75% of the students we study are enrolled in calc-based physics. This is disproportionate to distribution of classes we teach–only 33% of the students we teach take calc-based physics. The population of students least studied are those in two-year colleges, which comprise 25% of the students we teach (less than one percent of the students we study). Students in algebra-based courses are also under-represented in our research.
Steve is careful not to overly criticize our community’s beginnings. Our field has grown, in part, due to the fact that our research has focused on how even our “best” students struggle to develop functional understandings of basic physics concepts. Rather than blaming our past, Steve’s analysis points to a gap we need to address now.
Steve also looked at this data by disaggregating studies based on the SAT MATH distribution. From this perspective, it still appears that we are studying students on the high end. For me (Steve did not say this), this is especially critical due to the correlations that exist between achievement tests like the SAT and poverty and correlations between poverty and race. It could easily be said that we have been focusing more of our efforts and resources on the privileged. Steve also mentioned some research, which I can’t remember right now, that has found that a SAT Math score of 600 is a threshold for achievement in upper-level physics.
Oh, Force Concept Inventory
Steve’s poster also referenced some research about the FCI, which has also got me thinking again about the Force Concept Inventory (FCI), and how the FCI relates to our field’s focus on the upper end. If you don’t know, the FCI is the most widely used assessment / evaluation instrument in physics education. When using the FCI, normalized gains are the most widely used method to report student learning outcomes.
The idea behind normalized gain is to “take into consideration” students pre-test scores. Normalized gains can be interpreted as the “fraction of gain that could have occurred.” For example: a student who starts with a score of 40% and ends with 70%, gains +30% out of possible +60% gain, thereby having a normalized gain of 50%.
Despite measuring scores this way it appears that normalized gain (can be) strongly correlated with pre-test score. (Coletta and Phillips, 2005).
Underlying this correlation is additional findings that normalized gains on the FCI are strongly correlated with student scores on the Lawson Test of Scientific Reasoning Test, and they also correlated with students’ SAT scores (Coletta and Phillips, 2007).
A potentially huge problem we have as a community is that we report normalized FCI gains with out disaggregating these scores along such measures. I’d argue that this tendency is potentially dangerous, because it can lead us to make claims and offer implications for instruction that are distorted. For examples of how failures to disaggregate student achievement with measures of poverty lead to trouble, see Michael Marder’s prezi on Education and Poverty.
What can we do?
#1 We need Steve to publish his analysis of the mismatch between who we teach and who we study. This will enable those seeking funding to study under-represented populations to point to Steve’s research on the immense need for such research. It will also enable us to press funding institutions to create more parity in funding priorities. I emailed Steve this morning to offer encouragement and any help in making sure this happens.
#2 We need to begin as a community to publicize our own FCI normalized gains along with accompanying data that aides with meaningful disaggregation. This is true not only for publications about research. It should also include standards of reporting to funding agencies, and even standards of reporting on blogs. For example, right now, my own institutions reports normalized FCI gains from our algebra-based physics course to PhysTEC, and PhysTEC shares back data from all PhysTEC supported sites scores without disaggregation. I’ll start this process here: Our normalized gains at MTSU for algebra-based physics hover just below 0.3, and our SAT MATH scores are 460-570 range, with SAT Reading being 460-510 range. Note that this falls nicely in line with the graph above. Along this issue, we should really support the PER user’s guide. Although not on the site yet, they are working hard to create an Assessment Database and Analyzer tool that will make it easier for everyone to upload, use and interpret data in meaningful ways.
#3 Physics Education Researchers as individuals need to go out of their to engage with more research concerning students who aren’t just down the hall. The disproportionate focus on calc-based physics and severe shortage on two-year colleges is not malicious–it comes from convenience and a desire to improve our own local educational settings. Research-intensive universities are more likely to have students at the higher end of preparation and opportunity, and are also likely to have professors who have time and resources to do research. Instructors at two-year colleges have the opposite situation–no time, resources, or support to conduct research, and more likely to have students with less preparation and opportunity. I emailed three community college physics instructors this morning to begin that conversation.
Eric Brewe: “We should think about the use of normalized gain. It over values gains made at high end schools.”
Gasstation without pumps: “One question remains—why are students taking algebra-based physics? … Is the FCI the appropriate measure?”