Early Alerts and Unintended Consequences

Was this forwarded to you by a friend? Sign up, and get your own copy of the news that matters sent to your inbox every week. Sign up for the On EdTech newsletter. Interested in additional analysis? Try with our 30-day free trial and Upgrade to the On EdTech+ newsletter.

Despite being around for over a decade, early alert systems have often failed to live up to their promise in most colleges and universities. These data-driven tools monitor key student metrics like attendance, grades, and participation. They automatically flag students exhibiting signs of struggling, usually triggering alert messages from faculty and advisors.

There are numerous vendor-provided products, early alert features included in many learning management systems, and some institutions build their own systems. Despite being widely used, they have consistently been one of the technologies my clients have struggled with the most. For years, I have asked clients if their early alert systems have moved the needle, and the vast majority have reported that they have not or that they did not know. Over the years, I have seen clients cycle through different early alert systems: acquiring one, running into challenges, and then adopting a different tool. I got into the habit of advising clients looking to procure an early alert system that at least half of the implementations I had encountered had failed by one metric or another.

And yet, lately, I have been seeing a lot of positive coverage of early alert systems and their impact on student success. I am not sure whether this is more proof that hope does, in fact, spring eternal in the human breast, or whether it is due to the growing awareness of the importance of student success and accountability, or renewed interest in AI and using that technology to bring new approaches to early alerts. It is probably a combination of all three.

But I want to highlight two recent pieces that sharply contrast the narratives about early alerts. Given the renewed interest in these systems, it is worth revisiting both their potential benefits and the challenges and unintended consequences of many early alert implementations.

Oklahoma, where student success is just an early alert away

A recent Inside Higher Ed article highlighted a journal article evaluating an early alert implementation at Oklahoma State University. The study examined student success in lower-level courses.

The total sample consisted of 6,716 students across 18 different courses, including economics, statistics, and accounting. The study found that early alerts had a positive impact on student success and were associated with improved outcomes, including:

They used sensitivity analyses to confirm that other variables did not confound the results. Although some analyses of the impact of early alerts have been notoriously problematic, these findings are solid.

However, before rushing to purchase a new early alert system or criticizing those managing your current one for not achieving similar results, it is worth examining a new dissertation on the topic from CUNY.

The laws of challenges and unintended consequences

The dissertation explores obstacles to community college completion, with a particular focus on early alert systems. The author finds that the causes of student dropout are multifaceted, but early interventions in the first semester of a student's college career have a positive impact on long-term outcomes. However, the author's findings on early alerts are more negative.

This is not the first time I have seen an exploration of the unintended effects of early alert systems, but it is by far the most detailed exploration of the topic.

The early alert ecosystem

The findings about the unintended consequences of early alerts are concerning because study after study confirms that a major reason early alerts do not live up to their promise is the way they are implemented. In particular, problems stem from the following issues.

• Lack of faculty and advisor buy-in - Too often, not enough effort is made to ensure that advisors and instructors support the early alert initiative and that it fits within their workflows. One reason some schools find success with a homegrown system after abandoning a vendor-provided solution is the IKEA effect—the process of building something can create buy-in and compliance.

• Lack of follow-up - Many institutions implement early alerts but dedicate far less effort to designing what happens after an alert has been raised. At some institutions, there is no follow-up at all; at others, there is some attempt at follow-up, but it is not always effective. Only a few colleges have given extensive thought to how best to counsel students who receive an alert and need to get back on track.

The net result of these two common problems, given the insights from the CUNY dissertation, is that early alerts might actually increase the likelihood of students dropping out.

Early alerts might be a slightly kinder version of the infamous comment by a former university president about the need to push students out early.

Parting thoughts

But there is a larger lesson here about evaluating the efficacy of EdTech tools. We all seek proof that something works and that our investment has a meaningful payoff. However, there is a risk in looking too narrowly or asking overly limited questions, even in a well-structured study.

EdTech is designed to facilitate learning, and learning is a social activity carried out by people in all their complexity. To truly understand impact, we need to find ways to account for that complexity.

With early alerts, this means we cannot focus solely on outcomes like grades or withdrawal rates. We also need to examine how these systems are implemented, how they function in practice, and how students perceive and interact with them.

The main On EdTech newsletter is free to share in part or in whole. All we ask is attribution.

Thanks for being a subscriber.