- On EdTech Newsletter
- Posts
- Misplaced Confidence in Data
Misplaced Confidence in Data
No, we don't have excellent data to measure value of student outcomes
Phil Hill
August 28, 2023
Was this forwarded to you by a friend? Sign up, and get your own copy of the news that matters sent to your inbox every week. Sign up for the On EdTech newsletter. Interested in additional analysis? Try with our 30-day free trial and Upgrade to the On EdTech+ newsletter.
In March 2022 I described the new regulatory activism from the US Department of Education (ED), noting the nature of this activism.
* Use regulatory processes not just as safeguards but also to enable predefined end goals, usually around ‘reining in for-profit’ entities;
* Minimize the amount of time allowed for public comment and debate;
* Informally work through multiple agencies, both federal and state, to set new rules; and
* Worry about the unintended consequences later.
I think that this analysis has held up quite well, especially with the third-party servicer (TPS) guidance debacle and emerging issues with Gainful Employment and Financial Transparency. But I would add one point to that list above.
Overlook the limitations of available data when using it in the service of accountability
Recent congressional testimony exemplified this overconfidence in available data and our ability to measure the value of student outcomes. Hint - data with 1/6 coverage of programs is progress but not excellent in nature.
Testimony
Last month the House Subcommittee on Higher Education and Workforce Investment held a hearing on policy considerations to encourage greater value from US higher education academic programs. There was an interesting contrast between Michael Horn’s testimony and Stephanie Cellini’s that gets to a core issue of return on investment (ROI) of academic programs. Personally, I think that the ROI framing has gone too far, over-simplifying what education is all about, but it is an important consideration at the least when dealing with student debt.
It’s worth pointing out that there is general agreement on several key issues:
We need to reduce student debt levels, particularly in terms of too many students not being able to pay back their loans.
Government-backed financial aid should not support or encourage poorly-performing academic programs based on the same issue of student defaults.
We need better information to inform decision-making.
The analyses and recommendations from Horn and Cellini diverge once you go beyond these three points. For this post I want to mostly focus on Cellini’s statement that is illuminating about the underlying assumptions of current Department of Education (ED) and Arnold Ventures funded coalition policy changes that are dramatically impacting higher education this year and beyond.
Why am I bringing Arnold Ventures into the conversation again? Because Stephanie Cellini is a key member of that coalition, with funding for both her work at George Washington University and her role at the Brookings Institution. I am not implying that Cellini made these statements because of her funding, but I am saying that her view encapsulates the coalition perspective.