By Nathan Martin
When it comes to the grocery run, I’ve never been the most sophisticated shopper.
Overwhelmed by the aisles, I will forget essential ingredients, gravitate to the familiar (hello frozen pizza!) and somehow end up with a cart full of kale. I’m not uninformed. I just don’t make the best decisions.
For schools and teachers, choosing which education product to use can feel a bit like a trip to a grocery store–only one with incoherently organized aisles and sporadically restocked shelves, filled with products lacking any kind of a common labelling system. Evaluating the ingredients (or quality) of a product can be impossible. And then there’s the bureaucratic layers (unique to every school or district) which can complicate even the best plans.
So when it comes time to make a decision, schools and teachers can struggle. They take the task seriously and work with what information they have, but as Digital Promise found in Improving EdTech Purchasing (2014)
“With a growing number of products and limited trusted information about them, many districts rely on informal sources instead of data and evidence to make decisions”
Since districts make decisions in this manner, “companies perceive little incentive to produce rigorous evidence.”
This self-perpetuating cycle fails teachers, students or companies trying to make better products. It makes sense that SRI research found the most popular edtech products were familiar ones which fit within the existing system, not the most impactful.
Making better decisions in edtech is complex. The excellent Smart Series Guide to EdTech Procurement (released by Getting Smart and my former colleagues at the Foundation for Excellence in Education) provides an excellent navigational tool. One part of that decision-making process comes with better evidence.
Knowing What Works in EdTech
Education needs a way to know and track “what works.” It also needs ways for people to make better decisions. Living in the U.K., I’ve seen my decision-making improve through tools like BBC recipes to know “what works,” and the easy signals of traffic light packaging to make that final decision. Interestingly, the importance of labels in nutritious decision-making led the U.S. to announce a revamping its food labels for the first time in over 20 years.
While groups like EdSurge, Digital Promise and Graphite are taking important steps to help bring a bit of coherence and research to the edtech grocery store, companies should play a greater role in helping to untangle this Gordian knot.
So what can (and should) companies do to help break this cycle?
Education (in the U.S. at least) will never have a body to compile research, track education products and issue official guidance on what works–barring a massive shift in the U.S. political landscape. Power and decision-making is explicitly reserved to the states. Federal research funding remains minimal. As Kane writes, “There is no ‘FDA’ for education, and there never will be.”
There may never be an FDA for education, but education companies could do more to help schools and teachers understand the research and “nutritional value” of educational products.
Here’s the modest first step:
Like with the traffic-light labeling system or badging, education companies should adopt a standard, easily-recognizable way of identifying what evidence or research backs their product.
Labels help identify ingredients. Badging and micro-credentialing capture the accumulated skills of an individual. Both could be used to help tackle this question of evidence.
Hold the skepticism for a few paragraphs.
Currently, some education companies collect evidence on products. The strength of that evidence varies, ranging from a simple satisfaction survey, case study, or to a Randomized Control Trial (RCT). The evidence normally shows up in claims made about products (i.e., schools in X made Y gains through use of Z!).
For a school or teacher, making a judgement about what this evidence means (and what it supports) can be challenging. There are no standard labels, as terms and the definitions of those terms (such as efficacy, effectiveness, case study) may change from company to company. Schools and teachers know that RCTs are the gold standard for “what works,” but assessing other types can be less clear. And since RCT’s are expensive, time-consuming to conduct and (as AEI pointed out recently) are not without flaws, the answer is not just to conduct more RCT’s.
“It’s not that RCTs are unhelpful; they can be very helpful. It’s just that stopping with a randomized-control trial represents, at its best, an incomplete research process… the federal government should help by supporting research that progresses past initial RCTs and promotes alternative methods for unearthing what drives student outcomes in different circumstances.”
All evidence should be used in making education purchasing decisions, but all evidence and research are not equal and shouldn’t be given equal weight. How evidence is collected and reported determines how much to trust the larger claims of the impact of the product. So, while the experience of teachers and students (whether through surveys or reports) are important, knowing whether those products will have a repeatable and replicable impact relies on more rigorously collected and presented evidence. We, as education companies, should help make that process clearer and more transparent.
EdTech Label Ingredients
Any new set of labels, packaging or badging–whether driven by a set of leading companies or a non-profit — should be:
1. Clearly Defined and Easily Understandable: Defining types of evidence is challenging, but vital (as we’ve found out at Pearson). By agreeing on general definitions of types of evidence, the definitions could link to easily-recognizable badges or labels (imagine what a Correlation Study badge might look like). Like Creative Commons, which provides clear visuals defining copyright and repurpose, these badges could then be used by participating companies to designate what evidence had been collected on their product.
The greater amount of evidence supporting the product, the greater number of badges. By focusing on presenting information in a standard and understandable way, it would make decisions easier for educators– much like the My School Info Challenge tried to do with school performance data.
2. Focused on Creating Value for Teachers and Students: Education is an industry that should increasingly judge its success based on whether it creates real and demonstrable value for students and teachers. It animates the company I work at; as our CEO said in December, “the profit we make is the by-product of making a useful and meaningful addition to society… our ultimate mission is to make people’s lives better. If we fail at that, we fail as a business.”
Sustainable growth will only come by growing impact on the learners served, creating shared value, not just growing the numbers of the learners reached. Greater impact can come with smarter decision-making. There is consensus among edtech companies that purchasing decisions should be smarter–focused on answering the local needs of teachers and students. A coalition of education companies could be formed, dedicated to promoting actionable and understandable research, with the goal of better serving teachers and students. Better purchasing decisions are only as valuable as it leads to better outcomes. That focus is critical.
3. Verifiable and Linked to Evaluation Tools: Like with Mozilla’s badges, these badges, icons or labels could link to the actual research or evidence supporting the badge. It would be an imperfect tracking (there wouldn’t be a What Works Clearinghouse evaluating the actual submitted evidence) but it would allow a standard and verifiable way of displaying evidence. That work should be combined with tools to help schools and districts evaluate the evidence, like Digital Promise’s Evaluating the Studies of Ed-Tech Products. That clear distinction would encourage the market to move closer towards evidence-based decision-making–like we’ve seen with buying habits towards locally-sourced food.
What is Next?
Even in the best circumstances, this proposed first step wouldn’t fix the evidence component of ed-tech purchasing decisions. A self-regulated industry movement would face its external challenges and natural skepticism. Arguments over definitions of different types of studies and evidence. Bad actors pushing poor quality evidence.
But those challenges are not new to the market. It’s part of the reason why schools and teachers struggle to evaluate claims and products. Solving the problem of knowing what works (and creating quality products) is only one part of the evidence equation–there has to be a demand for products that can show they make a difference to learners. It will take a huge effort. Research needs to be done differently, catalogues built, new tools will need to be developed. Companies need to change how they market and label their products.
Smarter procurement and purchasing (a better education “grocery store”) is complex. Better evidence is one part. Labels or badging might not fix it, but it could help move towards a future process that is more coherent, understandable and centered around evidence.
For more, see:
- Shift To Digital: EdTech ‘Look Fors’
- Why Less Could be More in EdTech
- Working Your Way Out of an EdTech Mess
Stay in-the-know with all things EdTech and innovations in learning by signing up to receive the weekly Smart Update. This post includes mentions of a Getting Smart partner. For a full list of partners, affiliate organizations and all other disclosures please see our Partner page.