June 2017: Blackboard Outages

Since June 15th, Blackboard has been impacted by multiple slowdowns and outages, primarily during evening hours between 10:00 PM and 1:00 AM.

Support staff are working to fix the underlying issue. In the meantime, users should be aware that continued outages are possible during those hours.

See the table below for a timeline of individual incidents.

Est. Impact Start Date Start Time End Time Duration Notes
Scheduled Thu, Jun 15 04:51 06:38 1h 47m Blackboard offline for scheduled maintenance.
Low Thu, Jun 15 22:33 22:36 3m Single node marked offline (bbprd7).
High Thu, Jun 15 23:04 00:27 1h 23m Multiple outages across all nodes.
Low Fri, Jun 16 23:07 23:10 3m Single node marked offline (bbprd2).
Moderate Sat, Jun 17 18:23 21:56 3h 33m Three nodes marked offline (bbprd1, bbprd2, bbprd3).1
High Sat, Jun 17 22:21 23:02 41m Blackboard offline for emergency maintenance.
Moderate Sat, Jun 17 23:40 23:49 9m Multiple outages across all nodes.
High Sun, Jun 18 23:02 00:17 1h 15m Multiple outages across all nodes.
High Mon. Jun 19 23:04 00:27 1h 23m Multiple outages across all nodes.
Low Tue, Jun 20 15:12 15:56 44m Emergency maintenance with rolling restart.2
Low Tue, Jun 20 20:57 21:01 4m Single node marked offline (bbprd7).
Moderate Tue, Jun 20 22:59 23:03 4m Two nodes marked offline (bbprd2, bbprd4).
Low Wed, Jun 21 15:04 15:53 49m Emergency maintenance with rolling restart.2
Moderate Sat, Jul 1 23:06 23:13 7m Multiple outages across all nodes.
Low Sat, Jul 1 23:24 24:28 4m Single node marked offline (bbprd2).
Low Mon, Jul 3 23:07 23:15 8m Three nodes marked offline (bbprd3, bbprd4, bbprd5).
Low Mon, Jul 3 23:22 23:25 3m Two nodes marked offline (bbprd3, bbprd4).
Low Tue, Jul 4 23:06 23:11 5m Two nodes marked offline  (bbprd3, bbprd6).
None Wed, Jul 5 10:33 10:46 13m Emergency maintenance (no downtime)
Moderate Wed, Jul 5 23:05 23:11 6m Four nodes marked offline (bbprd2, bbprd3, bbprd5, bbprd7).
Moderate Wed, Jul 5 23:12 23:17 5m Four nodes marked offline (bbprd2, bbprd4, bbprd6, bbprd7).
Low Wed, Jul 5 23:18 23:21 3m One node marked offline (bbprd2).
Low Wed, Jul 5 23:24 23:27 3m Two nodes (bbprd5, bbprd6) marked offline.

When individual nodes are offline, user traffic will automatically be routed to other available nodes. Single-node outages normally do not impact the entire system, but users on the node that goes offline may experience slowdowns or minor errors before their traffic is routed elsewhere.

As more nodes go offline, users are more likely to experience slowdowns and errors.

If all nodes are offline, Blackboard will become unusable.


Some outages on 6/17 were caused by an issue on the University's Netapp storage infrastructure, which affected several other services as well.

2 During a rolling restart, individual nodes are taken offline intentionally in order to perform maintenance without taking the entire cluster offline.