Enclave wrangler: Recovering from timeouts #1019
Description
I've noticed a lot more DB refresh errors lately, 2-3 a day.
In the past year, most of these have been due to it taking longer than our arbitrary threshold of 2 hours for a draft to be finalized. But these kinds of errors were still happening less often than this new swath I noticed this week.
This week, we're getting a lot more of these, including 3 last night (example):
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='unite.nih.gov', port=443): Max retries exceeded with url: /api/v1/ontologies/ri.ontology.main.ontology.00000000-0000-0000-0000-000000000000/objects/OMOPConceptSet?properties.createdAt.gt=2025-06-24T14%3A10%3A14.583679-04%3A00 (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f7b559ddfc0>, 'Connection to unite.nih.gov timed out. (connect timeout=None)'))
Perhaps when this happens, we could have the refresh wait 5-10 minutes before trying again.
The only reason to try and ameliorate this is because the frequency of these errors are a chore to see in our inboxes. And it makes it less likely that we'll open and look at GItHub action logs for instances of "real errors".