May 4, 2017: Blackboard slowdown

Blackboard experienced a significant slowdown between 3:02 PM and 3:46 PM today, as follows:

Time Notes
3:02 PM One node (bbprd7) begins experiencing high memory usage, manifesting as a ConcurrentModeFailure warning during garbage collection.
3:05 PM CMF warnings on bbprd7 begin triggering status alerts.
3:26 PM High process count on bbprd7 triggers a new status alert. Users may be experiencing significant slowdown around this time.
3:31 PM Bbprd7 process count returns below alert level.
3:46 PM System admin takes bbprd7 out of traffic rotation. Node begins recovering.
4:08 PM System admin puts bbprd7 back into rotation.