It Cannot Be Automated in Today's Environment

Was this forwarded to you by a friend? Sign up, and get your own copy of the news that matters sent to your inbox every week. Sign up for the On EdTech newsletter. Interested in additional analysis? Try with our 30-day free trial and Upgrade to the On EdTech+ newsletter.

This year, the US Department of Education (ED) and its Arnold Ventures-funded allies have focused a lot of regulatory activity on online education. There were a set of draft rules that went through negotiated rulemaking in January through March, and a couple of those rules were published as proposed new rules last month. One item is a new requirement for all online courses to take daily attendance. Not just based on who logged into the LMS but who had academic engagement through quizzes, submitting assignments, participating in discussion forums, being active in course content, etc.

In my last two posts I have described how this proposed attendance-taking would add a new burden for tens if not hundreds of thousands of faculty members. The core of the issue is not whether ED’s goals make sense in being more consistent and accurate to measure when a student withdraws (they do). It is in the regulatory burden that will be placed on higher education staff and faculty to implement the new rules. On an ongoing basis, ED claims it will take ten minutes per day per institution, on average, for compliance. My estimates (based on talking to institutions doing reviews on their usage) are that it will take dozens of hours per day in aggregate per institution.

How could the estimates be off by two orders of magnitude?

It’s actually quite simple to understand if you look at the core misunderstanding that I believe ED has in its rulemaking. ED assumes that LMSs already have the information needed to take attendance and that schools merely need to create reports to be sent to the registrar or financial aid office. This assumption is wrong - unless an institution uses an LMS and no other EdTech tool (which is extremely rare) or has the clout to force all vendors to share private data (also rare), there is no mechanism today for third-party applications to send daily activity to the LMS in order to create an aggregated attendance report. It cannot be automated as assumed.

The net effect is that the only realistic way for most schools to follow the regulations is for somebody (mostly instructors) to be required to take attendance manually by looking at all tools being used in their course sections. When you add up all of this time for the new burden, that is how you get dozens of hours per day per institution.

Backing Up The Claim of Manual Intervention

Because this topic is so important (we’re nearing the end of the public comment period followed by ED deciding whether or not to implement the new rules), it is worth doing a deeper dive into why attendance-taking cannot be automated for a large majority of institutions in today’s environment.

As with so much of EdTech, it comes down to the LMS, even though the LMS is not the only tool being used. Faculty have an LMS course shell, but they use Zoom or Microsoft Teams for live or archived sessions, they use other tools for discussions, and quizzes, and interactive content exploration, etc. But for attendance-taking to work as written in the proposed regulation, for each student in each course section, the interactions from multiple systems must be aggregated. Suzy didn’t touch the LMS for four days, but she was active in the publisher courseware system reading course content and taking formative assessments, and/or she was in Zoom sessions and engaging in comments or chats, and/or she was active in enhanced discussion tools. To see if Suzy was active on October 17th, someone or some process has to check all tools to see what Suzy has done that day.

It is true, as stated in the proposed regulations, that data exist [p. 60282, emphasis added].

The LMS knows of the activity taken by students in that system, the video conferencing tool knows, the enhanced discussion tool knows, etc. But no one system is a system of record for all tool usage. And there can be dozens of third-party academic tools per institution, and in any one course section there often are several tools in usage.

Why is this?

The primary integration method between EdTech applications is the Learning Tools Interoperability (LTI) standard from 1EdTech. Based on LTI, a student or faculty member can launch an integrated app through the LMS, and if it is a graded item, the app will send back the results when the activity is complete.

For most implementations and for all that was possible for LTI 1.1, there is an app launch from the LMS to the app, and a report at the end of the results, as described in the LTI FAQ.

In between launch and results, the LMS knows nothing, leading to the following challenges regarding the new rules.

• For a multi-day activity in a third-party app, under LTI 1.1 there will be no data passed about daily activity to the LMS for it to aggregate for each student.

• For non-graded activities, there are no final results sent back to the LMS. Nothing.

Under LTI 1.3 (through LTI Advantage’s new Assignment and Grade Services), there is now the ability to add interim states for sending data from the app to the LMS, and this theoretically could include daily activity before completion. But there are two problems still.

• The standard would have to be universally and consistently applied to create daily attendance-taking, and right now this just does not happen. Third-party apps today do not send daily activity updates to the LMS. Period.

• This will still only solve the graded activities. If a course has non-graded academic engagement activities, this won’t work as the specification is based on assessments and grades.

Other Approaches

From a standards perspective, this situation would suggest that 1EdTech’s Caliper standard for federated data sharing would be a more likely way to automate this daily activity data aggregation per student. There is a revised or new specification being considered that could help this use case of capturing activity from multiple apps, but this capability is at the proof-of-concept stage, and there is no specification available yet. And even if there were, it would require universal and consistent implementation across all EdTech vendors, which is very unlikely, even in the future.

What about non-standards-based data aggregation? There are a subset of schools with the resources and size to demand that all approved vendors have to privately send attendance / engagement data to the school’s data warehouse, using either data exports or private APIs. There are also fully-online programs that at least get LMS activity through custom or advanced API data exports. For some schools, the mechanisms are in place as described by ED’s proposed rules [p. 60282, emphasis added]. But without analysis, ED makes an enormous assumption.

This is crazy. I suspect the SNHUs and WGUs and Liberty Universities of the world are already fully performing this task (by fully, I mean getting data from all app providers), as are the larger for-profit systems. But there is no way that “half of institutions offering distance education courses would already be performing this task.” I estimate 5-10% of institutions are, best case.

For the other 90+% of institutions:

• There are no mechanisms available today that could automate the attendance-taking process.

• For even the new LTI 1.3 availability of mid-state partial activity sharing between app and LMS to solve part of the problem, it is just not done today. The app ecosystem does not support this requirement and vendors have not implemented the capabilities.

Wrong Assumptions

This is why ED’s estimation of the burden for complying with the proposed rules is off by two orders of magnitude. ED and allies have not done the research shown above, and the proposed regulation is simply wrong in its assumption that the process can be easily automated for institutions offering at least one online course.

In my last post, I made an offer to better understand the perspective of ED and its allies.

I repeated this offer twice on X and LinkedIn. Since that time, I have had a more-or-less pleasant conversation with Barmak Nassirian (the primary negotiator pushing these rules), but he never addressed the argument itself. But I will address two of his points.

You are asking institutions to do something novel and unheard of. It might not be creating new science, but I have shown above that the mechanisms are not in place and are not available from the vendor community.

You are categorizing all pushback as “regulated entities hate all regulations,” but what if you are wrong and my analysis is right? Then ED’s burden of compliance is off by two orders of magnitude. And ED and allies have not defended their estimates at all.

I appreciate Wesley’s comment (he just moved from ED to New America).

1. These data are not easily found in LMSs unless the LMS is the only EdTech tool used within online courses at an institution.

2. The regulations would require online attendance-taking before knowing which students are withdrawing, so in effect the regs do call for constant reporting on every student in online courses.

3. Yes, there are distance ed programs taking attendance across apps, but as described above, that is only feasible for programs with the resources and size to mandate vendor data sharing and private API access in pre-approved apps. It is simply not feasible today for the majority of institutions.

4. Great idea, and I will submit this analysis as a formal public comment to the regulations.

By the way, my offer still applies for anyone from ED or from its allies.

Update 8/21: Clarified that Caliper specification already exists but that there is a revision being considered that would more directly address this use case. If ED finalizes this rule, Caliper (while not widely adopted today) may be a potential solution space in the future.

The main On EdTech newsletter is free to share in part or in whole. All we ask is attribution.

Thanks for being a subscriber.