Refresh the Ip2Geo cache and retry one more time when we run into an issue

[bzhangam]

Description

Refresh the Ip2Geo cache and retry one more time when we run into an issue.

After we update the datasouce in the primary shard if we fail to send the request to the nodes with replica shard we will not update the cache. Even later the replica is recovered from the primary shard the cache update logic will not be triggered. So and this change to refresh the cache and retry when we run into an issue.

Related Issues

N/A

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[heemin32]

I think we should just add retry in these four methods instead of handling each case outside of these methods. Otherwise, there will be case which we are missing to handle and also hard to maintain for future change.

    public boolean isExpired(final String datasourceName) {
        return getMetadata().getOrDefault(datasourceName, DatasourceMetadata.EMPTY_METADATA).getExpirationDate().isBefore(Instant.now());
    }

    public boolean has(final String datasourceName) {
        return getMetadata().containsKey(datasourceName);
    }

    public DatasourceState getState(final String datasourceName) {
        return getMetadata().getOrDefault(datasourceName, DatasourceMetadata.EMPTY_METADATA).getState();
    }

    public Map<String, Object> getGeoData(final String indexName, final String ip) {
        try {
            return geoDataCache.putIfAbsent(indexName, ip, addr -> geoIpDataDao.getGeoIpData(indexName, ip));
        } catch (ExecutionException e) {
            throw new RuntimeException(e);
        }
    }

[heemin32]

LGTM. Thanks!