The Efficacy Fetish in EdTech

Was this forwarded to you by a friend? Sign up, and get your own copy of the news that matters sent to your inbox every week. Sign up for the On EdTech newsletter. Interested in additional analysis? Try with our 30-day free trial and Upgrade to the On EdTech+ newsletter.

The focus on efficacy and research to determine which EdTech tools are effective is hard to argue against. After all, who wouldn’t want to know which tools truly enhance student learning and which fall short?

Unfortunately, the reality has been disappointing. Efficacy is often poorly conceptualized, research designs and methods are frequently flawed, and we lack the skills (and apparently the desire) to reliably distinguish between high- and low-quality studies. As a result, efficacy research has too often become an end in itself—serving more as public relations than as a meaningful guide to what works in EdTech and why.

For the sake of EdTech’s future, this situation should change.

How we got here

People have been concerned about the efficacy of EdTech for a long time. But the push really picked up steam in 2013 when Pearson published its Efficacy Framework. Led by John Fallon and Michael Barber, Pearson defined efficacy as making sure a product does what it’s supposed to do. In practice, that meant showing it could help students achieve specific learning outcomes.

A big part of the idea was to do research to gather evidence on efficacy, use that insight to improve the product, and then help institutions and customers use the tools in ways that would get the best results.

After Pearson released its framework, the company followed up with a series of annual efficacy reports and even launched a product designed from the ground up with efficacy in mind. While Pearson still talks about efficacy, it’s taken more of a back seat since John Fallon and Michael Barber retired in 2017 and 2020, respectively.

But the idea—and the fascination with efficacy—had already taken hold. It became a major focus not just for vendors eager to prove their products work, but also for foundations and nonprofits looking to show that certain kinds of interventions could make a difference. Even higher education institutions got in on the act, using efficacy to justify EdTech adoption and show they were improving student outcomes.

These days, you can’t throw a rock in EdTech without hitting a study about efficacy. They’re everywhere—at conferences, on vendor websites, and all over the EdTech trade press. Even when there isn’t a formal study, there’s always that nagging question: Sure, but is it effective?

With all this attention, you'd think we’d have a solid system by now for figuring out what works. You’d expect that, after more than a decade of studies, our methods for measuring efficacy would be sharper, and EdTech overall would be in a better place.

But that’s not what’s happened.

Instead we are as unsure as we were before 2013 of which tools work to improve outcomes. Some vendors are still hawking EdTech that is demonstrably not effective, and institutions are using EdTech in ways that don’t move the needle.

The roots of the problem

The roots of the problem are numerous. They lie in the poor research designs and methods used in efficacy studies, in some fundamental conceptual flaws in how we understand EdTech efficacy, and in the uncritical way we as a community consume EdTech research. The result is a steady stream of poorly conceived and executed studies that fail to show what they claim to prove.

Problems of design and method

Much of the research on EdTech efficacy is poorly designed based on at least three issues.

• Flawed research designs, including an over-reliance on randomized controlled trials (RCTs) without adequate consideration of confounding factors - For example, a widely shared article about an AI tutor at Harvard claimed that students using the AI tutor learned more in less time than those who didn’t. But as I (and others) have pointed out, the study was very short in duration and failed to account for treatment effects—such as learning gains driven by the novelty of a new teaching method rather than the tool itself.

• An overly simplistic understanding of learning - We still don’t fully understand how learning actually happens, and many study designs used to evaluate EdTech tools are too crude. Bror Saxberg of LearningForge (formerly of the Chan Zuckerberg Initiative and Kaplan) explains this well in a recent podcast. He likens much of EdTech research to a medical trial where a chemical is sprayed on both sick and healthy people, and researchers then conclude that there was no significant difference in effect. It’s a vivid analogy, but it illustrates the point: without a deeper understanding of learning science and more nuanced study designs, EdTech efficacy research is likely to fall short.

• Small sample sizes that limit generalizability - Many studies only show improvements within a narrow subset of students. Laurence Holt and Paul Tough have both critiqued this issue in studies involving Khan Academy, among other products.

Problems with how we conceptualize efficacy

In addition to design flaws, two deeper conceptual issues plague how we think about efficacy in EdTech.

First, the dominant view assumes that a tool is either inherently effective or not and that the goal of research is simply to confirm whether intended outcomes were achieved. But this view ignores pedagogy and the vastly different ways tools are used to achieve different results. A tool might work brilliantly in one context and be completely ineffective in another, even with the same instructor. Any efficacy rating is meaningless without considering how and where the tool is being used.

Second, our models of efficacy drastically oversimplify the environments where learning and student support actually happen. They often imply a straightforward cause-and-effect relationship between a tool and an outcome. That’s how we end up with claims that using a specific LMS "improved student learning."

But an LMS isn’t a three-quarter-inch drill bit, and learning isn’t a hole in the wall. You can’t draw a straight line from tool to outcome—especially with something as complex as an LMS, where countless variables come into play. The problem is made worse when these simplistic assumptions are paired with overly complex statistical methods, giving the illusion of rigor without real insight.

Problems of research and data literacy in EdTech

The flaws in the research itself are serious, but perhaps even more troubling is the lack of research and data literacy among EdTech users and readers. Many simply aren’t equipped to critically evaluate studies.

Worse still, the combination of low critical awareness and a strong desire for "proof" means that weak studies often spread quickly and unchallenged. No one stops to question the assumptions, design, or methodology. The Harvard AI tutor study continues to be widely cited as evidence of AI’s impact, despite its flaws. Articles about the uneven impact of online learning are still being shared, even though they rely on questionable methods. For years, the Purdue Signals project was held up as a model of success, long after serious doubts were raised about its data and evaluation.

Not only does bad research spread, but because most people don’t read or assess it critically, the mere existence of a study becomes a kind of signal or badge—an easy shorthand for a product’s supposed effectiveness.

We see this play out in the growing number of repositories and lists of EdTech tools that have been deemed efficacious. A recent article warning about the dangers of AI with an unsupported recommendation.

There have been multiple calls and efforts to set up these sorts of registries and for governments to be involved.

Where we end up

Studies of efficacy often become a meaningless good housekeeping seal of approval, but like a fetish the usage of efficacy studies can be even more insidious.

When produced or used by institutions, these studies often serve as a justification for a purchase or a project rather than encouraging an open exploration of what works. When used by vendors, the focus on efficacy can be an ethical slippery slope. Given the lack of quality and the uncritical reading by many consumers, the efficacy studies can begin to function as a form of de facto false advertising.

In both cases, the efficacy studies ultimately undermine trust and confidence in EdTech when people find they cant get the same results.

Towards a better way

As an EdTech community, we absolutely should be using data to make better decisions. But the current state of efficacy research—and our unhealthy relationship with it—means that is not happening. So what needs to change?

• We need to re-conceptualize what efficacy means in EdTech and move away from the one shot cause and effect and overly simplistic models that currently dominate the discourse.

• We need to get more comfortable with uncertainty and with understanding that one study does not “prove” that a tool is effective. It shows that the tool, used in a certain way, in a certain setting, had some good outcomes.

• We need to become better consumers of research. People like Al Essa already play an important role in this space, and we at On EdTech try to do our part as well. But we need more of this.

• Finally, we need to call out problems in EdTech research more openly and put pressure on the trade press to do the same. Glossing over weak or misleading studies helps no one.

The main On EdTech newsletter is free to share in part or in whole. All we ask is attribution.

Thanks for being a subscriber.