Was this forwarded to you by a friend? Sign up, and get your own copy of the news that matters sent to your inbox every week. Sign up for the On EdTech newsletter. Interested in additional analysis? Upgrade to the On EdTech+ newsletter.

How can it seriously be October? And what did I read this week?

Were you to pick up a research report titled “The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial,” you might expect it to tell you about, well, the effects of course-integrated AI tutoring on student performance and engagement. In the case of a recent research report published in the Annenberg Institute’s working-paper series, you would be sorely disappointed.

There are three claims worth unpacking here: effects, AI tutoring, and student performance and engagement.

But first, what did they actually do? The paper explores the use of an AI-powered course-based chatbot, the Virtual Study Assistant (VSA), in a randomized controlled trial among 2,379 undergraduate students and 30 instructors across multiple disciplines at the University of Maryland. Instructors were randomly assigned to offer the tool or serve as controls within groups of identical or similar courses. The researchers also analyzed a smaller “exact match” sample restricted to sections of the same course.

Let’s start with the effects part, as described in the paper’s abstract.

That sounds consequential, but what, exactly, produced those effects? Only about 15% of students offered access ever used the tool—and only 13% in the same-course sample.

Randomization gives the researchers grounds to estimate the effect of offering the intervention. It does not establish the effect of using the tutor. The lower scores are an average difference between groups offered and not offered access. The randomized comparison does not isolate the causal effect of using the tool.