[BUG] :distribution:docker:buildDockerImage failures downloading from http://vault.centos.org

[andrross]

Gradle task :distribution:docker:buildDockerImage fails occasionally with a timeout from http://vault.centos.org

| Errors during downloading metadata for repository 'baseos':
|   - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8/BaseOS/x86_64/os/repodata/repomd.xml [Connection timed out after 30001 milliseconds]
|   - Curl error (28): Timeout was reached for http://vault.centos.org/centos/8/BaseOS/x86_64/os/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
|   - Curl error (28): Timeout was reached for https://vault.centos.org/centos/8/BaseOS/x86_64/os/repodata/repomd.xml [Operation timed out after 30000 milliseconds with 0 out of 0 bytes received]
| Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Example: #2147 (comment)

[reta]

@andrross I think the best we can do is switch to another distribution (question - which one? amazonlinux:2? ubuntu? debian?), @dblock do you have thoughts? thanks!

[andrross]

My questions: Is that CentOS endpoint really that flakey? Are we getting throttled or something? Is something else misconfigured on our side?

[reta]

CentOS is dead, for better or worst, taking into account we use vault for ~1 week only and it was down already - it looks flaky at this moment. The problem was affecting everyone (not only us) - all my local builds were failing, same for others.

[reta]

@andrross down again: #2244 (comment)

[CEHENKLE]

Hmmm...if we're using amazonlinux for our distribution, it makes sense to me change it to that. @bbarani any concerns if we make that switch? (also curious if engineering effectiveness would make the switch or the core team, but he who is less important that what)

[peterzhuamazon]

In my experience centos is that flaky on endpoint.
Previously I just manually force it to use official mirror instead of other vault.

We cannot change to another distro at this point due to dependencies from knn.

[owaiskazi19]

Can we use vault.epel.cloud for more reliability here as a mirror? Switching to a new distribution is a better option though.