Most social programs unfortunately are found not to produce the hoped-for effects when rigorously evaluated. When evaluated in scientifically rigorous studies, programs in K-12 education, employment and training, crime prevention, and other areas too often are found to produce weak or no effects. Although programs that produce sizable effects on important life outcomes exist, they tend to be the exception—a pattern that occurs not just in social policy, but in other fields where rigorous evaluations are conducted such as medicine and business.1
Meanwhile, the United States has failed to make significant progress in key areas such as:
- Poverty: The U.S. poverty rate now stands at 12.7 percent and has shown little overall change (whether by official or alternative National Academy measures) since the late 1970s.2
- K-12 education: Reading and math achievement of 17-year-olds—the end product of our K-12 education system—is virtually unchanged over the past 40 years, according to official measures,3 despite a 90 percent increase in public spending per student (adjusted for inflation).4
- Well-being of low- to moderate-income Americans: The average yearly income of the bottom 40 percent of U.S. households, now at $23,700, has changed little since the 1970s.5
To Help Address This Problem:
This site seeks to identify those social programs shown in rigorous studies to produce sizable, sustained benefits to participants and/or society, so that they can be deployed to help solve social problems. The specific purpose is to enable policy officials and other readers to distinguish credible findings of program effectiveness from the many others that claim to be. Although many types of research have value in the evidence-building process, this site focuses on the results of well-conducted randomized controlled trials (RCTs), which are widely regarded as the strongest method of evaluating program effectiveness.6 The site also focuses on whether such studies show sizable, sustained effects on outcomes of clear policy importance (e.g., high school graduation, workforce earnings, teen pregnancies) and not just intermediate outcomes (e.g., children’s ability to recognize letters or numbers, positive parenting practices, take-up of services) that may or may not lead to important outcomes. Our focus on important, sustained effects distinguishes this site from many other clearinghouses of evidence-based programs.
Our Team and Process:
This site is administered by the Arnold Ventures’ Evidence-Based Policy team. Arnold Ventures’ core objective is to improve lives by investing in evidence-based solutions that maximize opportunity and minimize injustice. Arnold Ventures’ Evidence-Based Policy team is comprised of the former leadership of the Coalition for Evidence-Based Policy, a nonprofit, nonpartisan organization that, from 2001 to 2015, played a key role in the launch of the evidence-based policy movement.
We have identified the programs on this site through systematic monitoring of all rigorous evaluations published or posted online across all areas of social policy. The goal is to identify programs that, based on our careful review of the studies in consultation with outside experts, meet the criteria for “Top Tier,” “Near Top Tier,” or “Suggestive Tier.” In summarizing the evidence for these programs, we aim for balance and objectivity, and include discussion of any study limitations that may weaken the reliability of the findings. As a philanthropic organization, we receive no financial benefit from any program, enabling us to serve as an impartial reviewer. In the few instances where the organization helped fund a study that we are reporting on, we disclose our funding in the evidence summary.
1 Medicine: John P. A. Ioannidis, “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research,” Journal of the American Medical Association, vol. 294, no. 2, July 13, 2005, pp. 218-228.
Mohammad I. Zia, Lillian L. Siu, Greg R. Pond, and Eric X. Chen, “Comparison of Outcomes of Phase II Studies and Subsequent Randomized Control Studies Using Identical Chemotherapeutic Regimens,” Journal of Clinical Oncology, vol. 23, no. 28, October 1, 2005, pp. 6982-6991.
John K. Chan et. al., “Analysis of Phase II Studies on Targeted Agents and Subsequent Phase III Trials: What Are the Predictors for Success,” Journal of Clinical Oncology, vol. 26, no. 9, March 20, 2008.
Michael L. Maitland, Christine Hudoba, Kelly L. Snider, and Mark J. Ratain, “Analysis of the Yield of Phase II Combination Therapy Trials in Medical Oncology,” Clinical Cancer Research, vol. 16, no. 21, November 2010, pp. 5296-5302.
Jens Minnerup, Heike Wersching, Matthias Schilling, and Wolf Rüdiger Schäbitz, “Analysis of early phase and subsequent phase III stroke studies of neuroprotectants: outcomes and predictors for success,” Experimental & Translational Stroke Medicine, vol. 6, no. 2, 2014.
Social Spending: Coalition for Evidence-Based Policy, Randomized Controlled Trials Commissioned by the Institute of Education Sciences Since 2002: How Many Found Positive Versus Weak or No Effects, July 2013, linked here.
Laura and John Arnold Foundation, Low-cost RCTs Are a Powerful New Tool for Building Scientific Evidence about “What Works” to Address Major Social Problems, December 2015, linked here. Isabel Sawhill and Jon Baron, “We Need A New Start for Head Start,” Education Week, March 3, 2010, linked here.
Business: Jim Manzi, Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society, Perseus Books Group, New York, 2012, pp. 128 and 142. Jim Manzi, Science, Knowledge, and Freedom, presentation at Harvard University’s Program on Constitutional Government, December 2012, linked here.
2 Jessica L. Semega, Kayla R. Fontenot, and Melissa A. Kollar, U.S. Census Bureau, Current Population Reports, P60-259, Income and Poverty in the United States: 2016, U.S. Government Printing Office, Washington, DC, 2017. U.S. Census Bureau, Official and National Academy of Sciences-Based Poverty Rates: 1999-2016, 2017.
Kathleen Short, U.S. Census Bureau, HHES Division, Estimating Resources for Poverty Measurement, 1993 – 2003, 2005. Panel on Poverty and Family Assistance, National Academy of Sciences, Measuring Poverty: A New Approach, 1995, pp. 31-36.
Christopher Wimer, Liana Fox, Irv Garfinkel, Neeraj Kaushal, and Jane Waldfogel, Trends in Poverty with an Anchored Supplemental Poverty Measure, December 2013.
3 The Nation’s Report Card: Trends in Academic Progress 2012, NCES 2013-456, National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education, 2013.
4 Cornman, S.Q., Zhou, L., Howell, M.R., and Young, J. Revenues and Expenditures for Public Elementary and Secondary Education: School Year 2014-15 (Fiscal Year 2015): First Look (NCES 2018-301). U.S. Department of Education. Washington, DC: National Center for Education Statistics, 2017.
Richard H. Barr, Revenues and Expenditures for Public Elementary and Secondary Education, 1973-74 (NCES-76-140). U.S. Department of Health, Education & Welfare, National Institute of Education. Washington, DC: National Center for Education Statistics, 1976.
5 U.S. Census Bureau, Current Population Reports, 2017, op. cit., no. 2. This refers to inflation-adjusted income. It includes income from the economy (such as earnings) but not government transfers (such as Food Stamps). However, the evidence suggests that the overall story of income stagnation for the bottom 40 percent of households changes little even when one adjusts income for government transfers and other items that affect household living standards. Specifically, the Census Bureau’s alternative, National Academy of Sciences-based poverty measures make adjustments for government transfers, as well as factors such as state and local taxes, work expenses such as child care, out-of-pocket medical expenses, and geographic differences in housing costs. These adjustments change the poverty rate in any given year, as well as the composition of those in poverty, but do not change the overall trend in the poverty rate over time – i.e., little overall progress since the late 1970s. (The relevant citations are in endnote 2.) Although the National Academy-based poverty measures only apply to a subset of the bottom 40 percent of U.S. households, their corroboration of no meaningful improvement for that key subset suggest that similar findings would be obtained for the larger group.
6 Institute of Education Sciences and National Science Foundation, Common Guidelines for Education Research and Development, August 2013, linked here.
National Research Council and Institute of Medicine, Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities, Mary Ellen O’Connell, Thomas Boat, and Kenneth E. Warner, Editors (Washington DC: National Academies Press, 2009), recommendation 12-4, p. 371, linked here.
CBO’s Use of Evidence in Analysis of Budget and Economic Policies, Jeffrey R. Kling, Associate Director for Economic Analysis, November 3, 2011, page 31, linked here. U.S. Preventive Services Task Force, “Current Methods of the U.S. Preventive Services Task Force: A Review of the Process,” American Journal of Preventive Medicine, vol. 20, no. 3 (supplement), April 2001, pp. 21-35.
The Food and Drug Administration’s standard for assessing the effectiveness of pharmaceutical drugs and medical devices, at 21 C.F.R. §314.126, linked here. Every Student Succeeds Act, Section 8002 definition of “evidence-based,” Public Law 114-95, December 10, 2015.