November 4, 2016: Blackboard authentication outage

At approximately 6:40 AM this morning, an error caused username changes for approximately 900 faculty accounts. As a result, any of those users who tried to log into Blackboard would instead see an error message indicating that their username does not exist.

DLT staff were able to revert the username changes, and are currently investigating to determine the root cause of this issue.

Update:

DLT staff along with some help from EAPP staff were able to identify the cause of this issue.

On October 5th, DLT added two features to Blackboard LDPA integration: temporary usernames and persons-only snapshot processing. These two features are meant to work togther to fix usernames in case there are some users who are currently not in LDAP. Blackboard-LDAP connection timed out at 6:56 AM on 4th November because an extract took very long to process. This timeout caused all susequent username lookup to fail and roughly 900 usernames were replaced with temporary usernames. As a result, these users who tried to log into Blackboard would instead see an error message indicating that their username does not exist. Between October 5th and November 3rd, no combination of extracts caused a large enough workload to expose an LDAP connection timeout, so this bug was effectively hidden until 4th November. 

DLT installed a major update to PeopleSoft integration scripts on 10th November, 2016 which improved the LDAP usage to avoid timeout error. Refer this for more details on the update.