Friday Follow Up
Testing AI capabilities, what works and what doesn't, to analyze online learner surveys over the years
Phil Hill
December 12, 2025
Was this forwarded to you by a friend? Sign up, and get your own copy of the news that matters sent to your inbox every week. Sign up for the On EdTech newsletter. Interested in additional analysis? Upgrade to the On EdTech+ newsletter.
Yesterday, I shared a post about AAC&U’s survey on employer attitudes of college graduates, ultimately slamming the chosen lede. We should not view that report as a feel-good way to shoot down the declining confidence in higher ed story. Part of how I did that analysis was to look at previous AAC&U surveys to get a longitudinal view.
My intention was to use NotebookLM’s new Slide Deck capability to organize and visualize the trends between the 2018, 2021, and 2025 versions. Ultimately it was a failure, and I could only use two summary visualizations while going back to my typical manual collation and comparison of reports.
I think it might be interesting to share why that Gen AI usage didn’t work and how I got different results from a longitudinal study of the Voice of the Online Learner survey series. This post will share that analysis along with my experience using AI to aid in the process, hopefully to give a useful view of the current state of AI tools.
The AAC&U Analysis Failure
I should start out by stating how remarkable it is to even be able to test custom slide decks with moderately complex analysis. PowerPoint clip art should face a painful ending if there is any justice, with Copilot suffering the same fate.
The problem in a nutshell goes back to the well-known nature of Gen AI being a probabilistic rather than deterministic process. It got a lot right, but when it got data points or missing data wrong, there was no usable revision capabilities - I had to start over and discover the next mistakes. Hallucinations, you know.
