May 3, 2017: Blackboard micro outage (all nodes)

Starting at 11:53 AM, Blackboard was unresponsive for approximately 3 minutes. Students, faculty, and staff may have experienced slow page loads, timeouts, or other errors around this time.

Root cause analysis determined that a sudden rush of traffic exhausted Apache's ability to respond to incoming web requests (a "thundering herd" problem), including the automated health checks that are used to mark individual nodes online or offline. As a result, the load balancer marked all traffic-serving nodes offline for several minutes.

This issue was temporary and should not affect Blackboard on an ongoing basis.

Technical staff are reviewing possible strategies to mitigate this sort of issue in the future.