Was this forwarded to you by a friend? Sign up, and get your own copy of the news that matters sent to your inbox every week. Sign up for the On EdTech newsletter. Interested in additional analysis? Upgrade to the On EdTech+ newsletter.

Two weeks ago, Morgan and I wrote “An AI Tutor Study That Barely Tested Tutoring” about the University of Maryland study making the rounds, building on Morgan’s earlier coverage.

In a nutshell, UMD billed its study as a test of AI tutoring, but it largely tested access to an answer-giving chatbot with little integration into teaching. Only about 15% of students offered the tool ever used it, and just two instructors ended the semester with tutoring mode enabled.

Our criticism focused on the research claims. But a recent interview with UMD Provost Jennifer King Rice raises a more fundamental problem with the leadership decisions behind this initiative. What we missed is that Rice was the report’s senior author and the one who set up the initiative.

The Maryland study illustrates a common problem with how higher education evaluates EdTech: institutional implementation decisions are treated as background context when they are central to the results.

In the Maryland Today interview, Provost Rice describes the origin of the study and her role.

But a significant issue is that she is the chief academic officer at UMD which therefore raises the issue of institutional responsibility. The question is whether leadership applied the same scrutiny to its own implementation choices that it applied to the technology.

UMD (through its Department of Information Technology (DIT)) provided an enormous amount of documentation, webinars, and training opportunities on technical implementation. How to enable the tool, how to configure it, how to receive reports, etc.

But the core implementation choice is that UMD provided “no additional guidance for how [instructors] should incorporate the tool.” Among responding treatment instructors, none reported incorporating it into tasks or assignments.

This is EdTech 101. Tools are only valuable in supporting student learning if they are integrated into instructional practices. Rice and other authors know this truism, as the report itself called out [emphasis added].

Secondarily, this statement also points out that educational effects depend on “realized engagement,” yet the report brushed off the 15% adoption rate of students in the treatment group.

Why wasn’t the initiative designed around the conditions the authors themselves identified as necessary?

In the interview, Provost Rice acknowledged that “making AI tools available without providing faculty or students with clear guidance was not effective.”

Our question is why that understanding did not shape the initiative from the beginning. The report’s research summary already called out the need for instructional scaffolding, thus the finding that simply making AI tools available without guidance was ineffective therefore looks less like a surprising research discovery than a foreseeable and even obvious consequence of the initiative's design.

This was not a research discovery; it was a foreseeable flaw in the design of the initiative.

This issue is not unique to the University of Maryland, but this study does provide useful clarity.

For years, governments and education organizations have called on EdTech vendors to back up their promises with credible evidence, as seen in the U.S. Department of Education’s 2024 National Educational Technology Plan, UNESCO’s 2023 Global Education Monitoring Report , and Digital Promise’s Edtech Evidence & Integrity Pledge. The common theme is pushing education leaders to require vendors to explain their products’ research foundations and prioritize tools with independent product certifications.

However, there is too often a problem with how higher education asks the efficacy question. Too often efficacy questions are premised on an assumption that we are evaluating a relatively simple relationship:

Tool → Outcome → Evidence of Efficacy

But EdTech does not produce outcomes on its own. Between acquiring a tool and measuring an outcome lie a whole series of institutional decisions.

A more realistic model looks more like:

Problem definition → Tool selection → Faculty support and curricular redesign → Student awareness and use → Measurement and adaptation → Outcomes → Evidence of efficacy

This is more difficult and messier than simply asking if a product “works.” But it is a more accurate representation of how EdTech actually functions and whether it produces, or fails to produce, results.

And it doesn’t get vendors off the hook for developing products that have the potential for efficacy. Nor does it guarantee that good implementation will produce good outcomes and improve learning. But it does mean that implementation and efficacy cannot be separated. If students barely use a tool, instructors do not integrate it into their teaching, and an institution does little in response to evidence that those things are happening, what exactly has been evaluated?

That is why this study matters beyond Maryland. It is a particularly visible version of a common higher education problem.

If institutions want to take efficacy seriously—and we sincerely hope that they do—their own decisions need to be part of the analysis. Leaders need to define the problem, create the conditions for meaningful use, and adapt to what is happening. Measuring impact and outcomes is a critical part of establishing efficacy. But it cannot substitute for the institutional work required to produce the impacts and the outcomes in the first place.

Leaving faculty and students to figure things out for themselves is an implementation strategy—a spectacularly bad one.

Academic leaders who demand evidence that EdTech works should expect the same scrutiny of their own decisions. Defining an educational purpose, supporting instructors, and responding to weak adoption are part of leading an initiative. Those responsibilities do not disappear because the evaluation uses randomization. In evidence-based practice, implementation and support and tools are interrelated.

The main On EdTech newsletter is free to share in part or in whole. All we ask is attribution.

Thanks for being a subscriber.