September 1, 2017: Blackboard maintenance

During yesterday's maintenance window, nodes bbprd2 and bbprd6 were unable to complete the CU3 install in a timely fashion. It appears that the installer was encountering an issue causing a very long hang. All other nodes were able to complete the installation. Rather than overshooting the maintenance window, technical staff opted to bring all available nodes online, which allowed a return to normal operation for the remainder of the day.

Between 5:00 AM and 7:30 AM this morning, technical staff performed a recovery operation by syncing files from working nodes to the stuck nodes. This allowed all nodes to return to normal operation.