Today’s education decision-makers face many challenges when it comes to procurement, not the least of which is determining a product’s effectiveness — that is to say, the likelihood that it will positively impact learning for their students. On the other side of the procurement equation, vendors face challenges around both determining the efficacy of their products and engaging in candid conversations with potential clients.

Interactions between purchasers and vendors face perennial challenges: Leaders may be wary of promises made by vendors, and vendors may feel like they are limited in their opportunities to engage with clients who actually need what they are offering. These issues are compounded for both sides by the rapid pace of technological change, which makes it difficult to quickly and reliably understand how effectively edtech products and services function in various real world contexts. No wonder procurement is such a puzzle.

The new Guidelines for Conducting and Reporting EdTech Impact Research in U.S. K-12 Schools, recently published by the Educational Technology Industry Network (ETIN) and Empirical Education, help bridge this gap. The guide is designed to support vendors as they take smart, reasonable steps to research the effectiveness of their product or service. The guide also serves as an informative lens for decision-makers.

Evidence of student learning is key

Proceedings from a recent collaboration among the Jefferson Accelerator, Digital Promise, and the University of Virginia’s Curry School of Education found that when it comes to purchasing decisions, word-of-mouth rules the day. But in a world awash in ed tech products, education purchasers should prioritize evidence of a product’s impact on student learning as part of procurement decisions.

Not unlike the U.S. Department of Education’s Rapid Cycle Evaluation coach, the ETIN guide walks the reader through considerations around research design, data collection and methodology, and reporting of results for edtech products. The guide’s primary goal is helping edtech providers conduct and support valid research that can then inform the procurement decisions of school and district decision-makers.

The guide:

• Describes advantages of modeling and clarifying the logic behind how and why products would work in various contexts, details that can inform discussion on issues such as infrastructure, tech support, staff professional development, and curriculum and instruction.

• Explores considerations around the importance of sampling, including the use of comparison groups and estimating adequate sample sizes and effect sizes.

• Addresses the basics of human subjects protections and considerations (often managed by institutional review boards), including issues like voluntary participation.

• Recognizes key characteristics of the current digital environment, such as the impact of federal legislation (e.g. the Every Student Succeeds Act) on standards of evidence and the proliferation of cloud services (and related risks around data privacy).

• Describes the difference between efficacy studies (which describe how a product works under ideal conditions) and effectiveness studies (testing a product under varied and more standard conditions), as well as the importance of assessing implementation fidelity in understanding impact.

• Advocates for reporting results in ways that make the research process transparent and the results understandable to varied audiences, and helps decision-makers understand if and how reported results might apply to their local context.

Guide helps education leaders discuss product efficacy

Decision-makers may have good reason to be skeptical of effectiveness claims made by product developers, particularly if those results are not backed by best practices in researching efficacy and effectiveness. Similarly, vendors may be frustrated by a scarcity of opportunities to adequately understand the unique context of potential clients in order to unpack and explore how a product may support (or not, in some cases) the work of teaching and learning in different schools and districts. The ETIN publication is a great step forward in advancing this conversation, offering vendors some clear guidelines around standards of evidence, as well as helping educators and leaders build their own capacity to engage in informed discussion around rapid-cycle evaluation and product efficacy.

This conversation may be situated within a broader one about research-to-practice, including developing informed and efficient procurement procedures and guidelines on product development. In 2015, The U.S. Department of Education’s Office of Educational Technology (directed at that time by Richard Culatta, now ISTE CEO) published the Ed Tech Developer’s Guide, which provides grounded advice to product developers on key issues, such as understanding users and their individual contexts (i.e. the K-12 school, district, and state department); using data to inform design, networking and funding development of ideas; and attending to lynchpin issues such as interoperability, trends in blended and personalized learning, and student data privacy.

Connecting knowledge to tools requires time and training

The need to bridge the research-to-practice gap is not a new one, although it it has never been more pressing. People in the field of education today, broadly speaking, have the knowledge and tools to profoundly shift teaching and learning. But connecting that knowledge with the right tools requires vision, training, and time — all of which can be challenging to come by for the average K12 decision-maker.

Scarcity of choice is no longer a problem for K-12 purchasers; rather in today’s market, procurement representatives at schools and districts must sift through a flooded universe of options to determine the right buy for their teachers and students. This presents challenges to developers, but vendors also have a unique opportunity to bring research-based best practices to learning if they are willing and able to use these practices with their own products and translate the results for time-strapped educational leaders and staff.

The practices articulated in the ETIN guidelines as well as other resources can offer developers a competitive edge in the marketplace by helping them make good choices around product design and testing, thus contributing to a mutual win for both developers and educators, and most importantly, the students who both groups support.

(Photo courtesy of Allison Shelley/The Verbatim Agency for American Education: Images of Teachers and Students in Action.)

Brandon Olszewski is director of research at ISTE. His interests include mixed methods, program evaluation, clever solutions to messy problems and learning interesting new things every day. Sarah Stoeckl, Ph.D., is the senior project manager in the ISTE Standards Department at ISTE. Her interests include empowering, developing, and supporting students and educators, exploring micro-credentials and badges, and miscellaneous word-nerdery.

Turn to the ISTE Seal of Alignment to find high-quality products and services aligned to the globally recognized ISTE Standards.