Distributed Learning Technologies

ERFC - Update PS-BB integration to avoid REST API errors

This will update the ps-bb integration script on bbutilprd to add the following changes:

- Limited alerts to one every `LOCK_ALERTS_COOLDOWN` hours (default 24)
- REST client now logs most recent request/response to aid in troubleshooting
- REST client now catches and logs non-standard `X-Died` header
- REST client now supports `REST_PRECACHE_USERS_PAGE_SIZE` setting (default 50)

It turns out that our HTTP library is not able to handle certain large responses from Blackboard, which manifests with the following error message in the response's "X-Died" header:

read timeout at /usr/local/share/perl5/Net/HTTP/Methods.pm line 271

Our REST client was not previously aware of this header and instead relied exclusively on the HTTP status code, which unfortunately was marked as 200 OK.

The proposed change will make our REST client aware of that error header, and will also avoid the circumstance which triggers that specific error, by changing user precache page size from 100 to 50. That will bring response size well within levels that can be handled by our client, at the cost of increasing our morning runtime by approximately 30 minutes. If desired, we can adjust the setting later on once we have more empirical data... for the time being, I would prefer to be conservative and ensure maximum stability.

It's imperative that we deploy this fix quickly, or we may not be able to run our scheduled student load on 8/17.

The software update should take only a few seconds, but I'm scheduling a longer window to include monitoring on the restarted integration job.