Research-Based Interventions vs. Product Management

Author

Phil Hill
February 02, 2023

I wrote a series of posts in the fall about the Bill & Melinda Gates Foundation’s work in postsecondary education centered on digital courseware, challenging their claims of following the data. You can read the first post about the messaging machine, the second on the push for CourseGateway, the third laying out our history 2013 – 2018 working with the foundation, and the fourth on a research background in advance of their response to my questions. Last week Alison Pendergast, Senior Program Officer at the foundation, provided a response. [full-page audio link]

I appreciate Alison Pendergast’s response – it is important in that it is the best-written and thorough description of the foundation’s funding of courseware-centered interventions with “the goal of ensuring that many more students can complete their certificates and degrees and that race, ethnicity, and income do not serve as predictors of student success.” The response was useful in calling out the limits on what data is available and describing the foundation’s plans to adjust investment approaches and help collect better data.

However, there was nothing in the extensive answers to my three questions that changes my analysis laid out in the fall. In my opinion, the flaw in the foundation’s efforts is that they mistake product management for a research-based set of interventions.

The ASU Study

While we are focused on supporting innovative, high-quality digital learning solutions like courseware that we believe can make learning more accessible, relevant, engaging, interactive, personalized, and cost-effective for students, we are also intensely focused on supporting faculty implementing evidence-based practices that can be enabled by courseware (such as active and inclusive teaching strategies) to close opportunity gaps.

There was one new study referened that was new to me and that provides a specific example that courseware-based interventions can work. Arizona State University (ASU) took a deliberate systems approach to their intervention, “wherein organizational and instructional systems are related and changes to one element impact other elements or even sub-systems,” and they used a heavily-resourced team design approach that combined active learning pedagogical design alongside / enabled by adaptive courseware usage. The study was not thorough in documenting results, but it did describe some impressive results, with one course redesign improving C or better course outcomes from 72-80% to 91-94% in adaptive+active versions taught by the same instructor.

ASU shows that this approach can and has led to improved student performance, at their university, with their holistic design approach, but we already had hints of this in the SRI reports. Note that in the 2018 report on Gates foundation courseware implementations, the two from ASU already showed positive effects using courseware.

Whither Community Colleges

But the Gates foundation is looking for scale – how to make interventions to a broad set of institutions, but the 2018 SRI report specifically called out where the courseware interventions were not working.

Only 1 of the 12 studies conducted at two-year colleges produced significantly positive impacts, suggesting that more capacity building and research into implementation strategies need to be done in those settings.

Look at the chart above. A lot of those 11 studies did not just fail to show significantly positive impacts, they actually showed negative impacts. The intervention appears to have worsened student outcomes in many of the very schools that quite often serve Black, Latino, Indigenous students, and students from low-income backgrounds.

Just because an intervention can work does not mean that extrapolating to interventions at a broad set of schools will work. The response confirms that the Gates foundation does not have evidence that courseware-based interventions can improve student performance in gateway courses at a broad set of institutions.

Product Management

There is a fundamental difference between finding the most effective (i.e., research-based) interventions to achieve a stated goal and doing the best job you can with a pre-selected intervention. The Gates foundation is practicing the latter, which is product management. The foundation has a belief, they are funding and promoting interventions at a broad set of institutions, and their idea of research appears to be identifying success factors and better implementation methods of the product category of courseware.

Therefore, we have dedicated resources to a comprehensive research plan over the next four years to understand better the faculty and student user experience with the courseware and gain insights into implementation best practices and effectiveness. This research will help us understand which courseware and implementation factors lead to the best outcomes for our focus students.

The effort is to get others to understand what the Gates foundation believes they already know.

We continue to believe that this moment in time presents both a critical and exciting opportunity for higher ed to understand better how high-quality and affordable digital courseware can help faculty implement active, inclusive pedagogies that lead to more equitable education outcomes for students. We’re proud of the work we’re doing alongside partners and grateful for the opportunity to highlight it.

The response shows the foundation as having good intentions and trying to make adjustments, including learning a hard lesson in not ignoring how faculty implement courseware within their courses, and also including work to include selected communities in their design approach. But the foundation is not challenging their own stated beliefs in the face of disappointing results. A research-based interventions approach would be open to try different methods and even enabling technologies to achieve their goals, or would focus on pedagogy first rather than technology implementation. A product management approach tries to add implementation or design services, or to make enhancements to the pre-selected technology, without asking if underlying assumptions should be questioned. The Gates foundation would do well to understand the difference if they desire to have a more positive impact on educational outcomes.