June 2017: Blackboard Outages
Since June 15th, Blackboard has been impacted by multiple slowdowns and outages, primarily during evening hours between 10:00 PM and 1:00 AM.
Support staff are working to fix the underlying issue. In the meantime, users should be aware that continued outages are possible during those hours.
See the table below for a timeline of individual incidents.
| Est. Impact | Start Date | Start Time | End Time | Duration | Notes |
| Scheduled | Thu, Jun 15 | 04:51 | 06:38 | 1h 47m | Blackboard offline for scheduled maintenance. |
| Low | Thu, Jun 15 | 22:33 | 22:36 | 3m | Single node marked offline (bbprd7). |
| High | Thu, Jun 15 | 23:04 | 00:27 | 1h 23m | Multiple outages across all nodes. |
| Low | Fri, Jun 16 | 23:07 | 23:10 | 3m | Single node marked offline (bbprd2). |
| Moderate | Sat, Jun 17 | 18:23 | 21:56 | 3h 33m | Three nodes marked offline (bbprd1, bbprd2, bbprd3).1 |
| High | Sat, Jun 17 | 22:21 | 23:02 | 41m | Blackboard offline for emergency maintenance. |
| Moderate | Sat, Jun 17 | 23:40 | 23:49 | 9m | Multiple outages across all nodes. |
| High | Sun, Jun 18 | 23:02 | 00:17 | 1h 15m | Multiple outages across all nodes. |
| High | Mon. Jun 19 | 23:04 | 00:27 | 1h 23m | Multiple outages across all nodes. |
| Low | Tue, Jun 20 | 15:12 | 15:56 | 44m | Emergency maintenance with rolling restart.2 |
| Low | Tue, Jun 20 | 20:57 | 21:01 | 4m | Single node marked offline (bbprd7). |
| Moderate | Tue, Jun 20 | 22:59 | 23:03 | 4m | Two nodes marked offline (bbprd2, bbprd4). |
| Low | Wed, Jun 21 | 15:04 | 15:53 | 49m | Emergency maintenance with rolling restart.2 |
| Moderate | Sat, Jul 1 | 23:06 | 23:13 | 7m | Multiple outages across all nodes. |
| Low | Sat, Jul 1 | 23:24 | 24:28 | 4m | Single node marked offline (bbprd2). |
| Low | Mon, Jul 3 | 23:07 | 23:15 | 8m | Three nodes marked offline (bbprd3, bbprd4, bbprd5). |
| Low | Mon, Jul 3 | 23:22 | 23:25 | 3m | Two nodes marked offline (bbprd3, bbprd4). |
| Low | Tue, Jul 4 | 23:06 | 23:11 | 5m | Two nodes marked offline (bbprd3, bbprd6). |
| None | Wed, Jul 5 | 10:33 | 10:46 | 13m | Emergency maintenance (no downtime) |
| Moderate | Wed, Jul 5 | 23:05 | 23:11 | 6m | Four nodes marked offline (bbprd2, bbprd3, bbprd5, bbprd7). |
| Moderate | Wed, Jul 5 | 23:12 | 23:17 | 5m | Four nodes marked offline (bbprd2, bbprd4, bbprd6, bbprd7). |
| Low | Wed, Jul 5 | 23:18 | 23:21 | 3m | One node marked offline (bbprd2). |
| Low | Wed, Jul 5 | 23:24 | 23:27 | 3m | Two nodes (bbprd5, bbprd6) marked offline. |
When individual nodes are offline, user traffic will automatically be routed to other available nodes. Single-node outages normally do not impact the entire system, but users on the node that goes offline may experience slowdowns or minor errors before their traffic is routed elsewhere.
As more nodes go offline, users are more likely to experience slowdowns and errors.
If all nodes are offline, Blackboard will become unusable.
1 Some outages on 6/17 were caused by an issue on the University's Netapp storage infrastructure, which affected several other services as well.
2 During a rolling restart, individual nodes are taken offline intentionally in order to perform maintenance without taking the entire cluster offline.