Blackboard Has Worst LMS Problem in Years, Taking K-12 District Offline for Days

In a post three weeks ago, I described the massive increase in usage of LMS systems experienced as part of the COVID-based transition to virtual instruction. As of April 1, Blackboard reported LMS usage increases of 400% for the Learn LMS and 3600% for Collaborate Virtual Classroom (synchronous video).

In my post I noted that “given the scale and timeframe of the transition, I think it’s actually quite remarkable how few problems we have seen.” While I still feel that is true – overall the LMS market has responded quite well to the crisis – we now have what I believe is the worst Blackboard Learn LMS performance problems in years. Fairfax County Public Schools (K-12) in Virginia, with its roughly 189,000 students, experienced such significant performance issues with Learn that they had to shut down their Virtual instruction for a period of days, and as of the end of Monday this week, they are moving all virtual work off of the Blackboard LMS.

What Happened

As reported by WTOP News:

The problems started early last week (April 13) with integration problems with the authorization (sign-in) system that were resolved by FCPS staff, but on Wednesday they started experiencing load issues – where the LMS could not handle the number of concurrent users and was unresponsive, or slow enough to be unusable. The problem was identified, but FCPS and Blackboard decided that they needed time to ensure the problems were fixed and tested, due to heavy customizations of the system made over the past two decades. This decision led to the cancellation of classes for Thursday and Friday of last week.

According to Blackboard representatives, they worked through the weekend testing and confirming the fix. Unfortunately, within a few hours of classes beginning this past Monday (April 20), FCPS experienced a new, mostly unrelated problem that also made the system unusable. That issue was mostly resolved by the end of the day, with a planned upgrade to be performed overnight to ensure system stability. But by that point the FCPS board had had enough and changed plans as announced on the FCPS website Monday night.

The board further announced plans for continuing their virtual instruction by alternative methods.

Blackboard is headquartered in Fairfax County, and FCPS has been a client for nearly 20 years. In January, however, FCPS had already made the decision to transition from Blackboard Learn to Schoology as of Summer 2021. It is not clear if or how FCPS will accelerate that schedule.

Background

I interviewed Tim Tomlinson, Blackboard’s Chief Product Officer, on Tuesday. As of that time, Blackboard had not heard definitive plans from FCPS and had learned about the board’s decision through the website and media.

Tomlinson said this situation at FCPS was by far the worst problem that the company has faced over the past two months of COVID transitions, and he said he personally feels terrible, especially with this happening in Blackboard’s own backyard – colleagues and friends have their children in Fairfax County Schools.

FCPS was running a highly-customized version of Blackboard Learn last updated in 2017 in a managed hosting environment within Blackboards data center. According to Tomlinson, the problems were relating to the LMS itself and not to Collaborate.

The problems were such that they had been addressed in subsequent Learn software releases over the past two years, but it was not clear that these issues would affect FCPS. Once the problems occurred, the challenge was applying fixes could be applied in the specific FCPS environment. This was not just a system using older software, it was a complex system according to Tomlinson.

While COVID-driven transitions have increased system usage, the actual load experienced by FCPS was not unexpected. Blackboard has other K-12 clients with higher usage, and recently they had upsized FCPS’s server configuration based on the forecast load increases. In fact, while recent FCPS usage was higher than typical, it was not the highest experienced by the system. The problem came from the two software flaws.

Risk of Eating Your Cake and Having It, Too

Blackboard is in the process of moving their Learn LMS towards Software as a Service (SaaS, aka Cloud) hosting on Amazon Web Services (AWS). As of our last reporting in November, Blackboard had migrated approximately one third of Learn LMS clients to SaaS hosting models.

Thus far Blackboard still supports older self-hosting models (in client data centers) and managed hosting models (in Blackboard data centers). The problems experienced at FCPS do not impact Learn SaaS deployments, as they have different server configurations and run the latest versions of the software.

This multi-year migration stands in contrast with LMS competitor Canvas – which was built in the cloud – and D2L Brightspace. D2L took a different path and has entirely migrated their client base from self-hosting and managed-hosting to pure AWS-based SaaS hosting over the past 3-4 years. D2L lost some clients along the way based on this approach, but they are now a true cloud-based LMS company.

Blackboard has not been willing to make the difficult decision to force clients off of older hosting models, meaning that the company has increased the number of deployment options they have to support. They will not truly benefit from the move to Learn SaaS until they can retire older models and focus on one code base and one deployment option.

In my opinion, this cautious approach by Blackboard has come back to bite them. While Tomlinson made it clear that the fault was with Blackboard and took full responsibility for the situation, FCPS has made it clear just how difficult it is to support highly-customized, two-year-old software versions in custom data centers. And ironically, by moving to Schoology FCPS will be moving to the cloud without any customizations. Schoology like Canvas only runs with a SaaS-hosting model at AWS.

SaaS-hosting with AWS is not a panacea, however. Instructure has run into several cases of performance issues over the past month, although none have lasted longer than a few hours. And Schoology has had significant problems they acknowledged in a recent company blog post.

This entire situation is unfortunate – for the Fairfax County students and teachers, and for Blackboard as a company. We knew that increased system usage would be a challenge for LMS companies, which have overall handled COVID transitions quite well, but that does not help the students who lost days of school due to the outage.

The post Blackboard Has Worst LMS Problem in Years, Taking K-12 District Offline for Days appeared first on Phil Hill & Associates.