Measuring the Impact of Emerging Technologies in Education: A Pragmatic Approach
The evaluation of emerging technologies is important for their impacts to be effectively integrated into learning and teaching settings to bring the best benefit to learners and teachers. Educators, learners, parents, and policymakers alike, therefore, need reliable methodologies for evaluating the effectiveness of such emerging technologies. However, the impact evaluations of technology in education are challenging. This challenge is more significant for emerging technologies, as change is the essence of emerging educational technologies. Therefore, the value of traditional impact evaluations in education requires being reconsidered within this context. Here, we present a pragmatic approach to measuring the impact of emerging technologies in education which focuses on the suitability of the proposed evaluation methods and the types of evidence rather than on the hierarchy of these methods and evidence types. The approach has two main steps. First one is the creation of a clear theory of change to identify outcome measure(s) and assumptions that are behind the expected impact of the emerging technology intervention. Second is the identification of the type of evidence and methods to generate it that are the most appropriate for the current innovation stage of the emerging technology.
KeywordsEvidence-informed practice Emerging technologies Impact evaluation Pragmatism Evidence
- Beraza, I., Pina, A., & Demo, B. (2010, November). Soft & hard ideas to improve interaction with robots for kids & teachers. In Workshop proceedings of conference on simulation, modelling, and programming for autonomous robots, (pp. 549–557). Darmstadt (Germany).Google Scholar
- Condie, R., & Munro, B. (2007). The impact of ICT in schools: Landscape review. UCL Institute of Education. London: United KingdomGoogle Scholar
- Cox, M. J. (2005). Educational conflict: The problems in institutionalizing new technologies in education. In G. Kouzelis, M. Pournari, M. Stoeppler, & V. Tselfes (Eds.), Knowledge in the new technology (pp. 139–165). Frankfurt: Peter Lang.Google Scholar
- Cox, M. J. (2008). Researching IT in education. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 965–981). New York: Springer.Google Scholar
- Cox, M. J., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2003). ICT and pedagogy: A review of the research literature (ICT in schools research and evaluation series, Vol. 18) London: UCL Institute of Education.Google Scholar
- Cox, M., Webb, M., Abbott, C., Blakeley, B., Beauchamp, T., & Rhodes, V. (2004). An investigation of the research evidence relating to ICT pedagogy. A report to DfES. Retrieved from http://dera.ioe.ac.uk/1601/1/becta_2003_attainmentpedagogy_queensprinter.pdf
- Cress, U., Moskaliuk, J., & Jeong, H. (Eds.). (2016). Mass collaboration and education (Vol. 16). Cham: Springer.Google Scholar
- Cukurova, M., Luckin, R., & Baines, E. (2017). The significance of context for the emergence and implementation of research evidence: The case of collaborative problem-solving. Oxford Review of Education, 1–16. https://www.tandfonline.com/doi/abs/10.1080/03054985.2017.1389713.
- De Bruin, C. L. (2015). Conceptualizing effectiveness in disability research. International Journal of Research & Method in Education. https://doi.org/10.1080/1743727X.2015.1033391.
- Department for International Development. (2017). Public Bulleting Board 7749: Education technology to improve learning for all. Retrieved from https://supplierportal.dfid.gov.uk/selfservice/pages/public/viewPublicNotice.cmd?bm90aWNlSWQ9Njc3MjA%3D.
- Dunleavy, M., & Dede, C. (2014). Augmented reality teaching and learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education (pp. 735–745). New York: Springer.Google Scholar
- Fullan, M. (2001). Leading in a culture of change. San Francisco: Jossey-Bass.Google Scholar
- Gulson, K. N., & Symes, C. (2007). Spatial theories of education: Policy and geography matters. New York: Routledge.Google Scholar
- Hattie, J. (2008). Visible learning: A synthesis of over 800 meta analyses related to achievement. New York: Routledge.Google Scholar
- Hoeken, H. (2001). Anecdotal, statistical, and causal evidence: Their perceived and actual persuasiveness. Argumentation, 15(4), 425–437.Google Scholar
- Katterfeldt, E., Cukurova, M., Spikol, D., & Cuartielles, D. (2018). Physical computing with plug-and-play toolkits: Key recommendations for collaborative learning implementations. International Journal of Child-Computer Interaction, 1–23. https://www.sciencedirect.com/science/article/pii/S2212868917300351.
- Kelly, A. E., Baek, J. Y., Lesh, R. A., & Bannan-Ritland, B. (2008). Enabling innovations in education and systematizing their impact. In A. E. Kelly, R. A. Lesh, & J. Y. Baek (Eds.), Handbook of design research methods in education: Innovations in science, technology, engineering, and mathematics learning and teaching (pp. 3–18). Routledge, Abington, Oxon.Google Scholar
- Nardi, B. A. (1996). Studying context: A comparison of activity theory, situated action models, and distributed cognition. In Context and consciousness: Activity theory and human-computer interaction (pp. 69–102). London: MIT Press.Google Scholar
- NESTA. (2016). Using research evidence for success: A practical guide. Retrieved on from https://www.nesta.org.uk/sites/default/files/using_research_evidence_for_success_-_a_practice_guide.pdf.
- O’Leary, Z. (2004). The essential guide to doing research. London: Sage.Google Scholar
- Petty, G. (2009). Evidence-based teaching: A practical approach. Cheltenham: Nelson Thornes.Google Scholar
- Pilkington, R. (2008). Measuring the impact of IT on students’ learning. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education. Berlin/Heidelberg/New York: Springer.Google Scholar
- Reeves, T. C. (2008). Evaluation of the design and development of IT tools in education. In J. Voogt & G. Knezek (Eds.), International handbook of information technology in primary and secondary education. Berlin/Heidelberg/New York: Springer.Google Scholar
- Stern, E. (2015). Impact evaluation: A guide for commissioners and manager. London: Big Lottery Fund and the Department for International Development.Google Scholar
- Tondeur, J., van Braak, J., Guoyuan, S., Voogt, J., Fisser, P., & Ottenbreitt-Leftwich, A. S. (2012). Preparing student teachers to integrate ICT in classroom practice: A synthesis of qualitative evidence. Computers & Education, 59(1), 134–144. https://doi.org/10.1016/j.compedu.2011.10.009.CrossRefGoogle Scholar
- Underwood, J. (2017). Exploring AI language assistants with primary EFL students. In CALL in a climate of change: Adapting to turbulent global conditions–EUROCALL 2017 (p. 317). Southampton: University of Southampton.Google Scholar