[BUG] geo2ip update fails

[bruberg]

Describe the bug

geo2ip processed log events are tagged with source.geo.error: ip2geo_data_expired. Checking the state of one of the data sources show that updating has failed for some time:

# GET /_plugins/geospatial/ip2geo/datasource/city-datasource
{
  "datasources": [
    {
      "name": "city-datasource",
      "state": "AVAILABLE",
      "endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
      "update_interval_in_days": 3,
      "next_update_at_in_epoch_millis": 1761551548911,
      "database": {
        "provider": "maxmind",
        "sha256_hash": "7xh3reRaDjT8HyQ/Up5+QWu3eC0NHRKAvttfmNwNeTM=",
        "updated_at_in_epoch_millis": 1758636974000,
        "valid_for_in_days": 30,
        "fields": [
          "country_iso_code",
          "country_name",
          "continent_name",
          "region_iso_code",
          "region_name",
          "city_name",
          "time_zone",
          "location"
        ]
      },
      "update_stats": {
        "last_succeeded_at_in_epoch_millis": 1758700716603,
        "last_processing_time_in_millis": 366704,
        "last_failed_at_in_epoch_millis": 1761292895482,
        "last_skipped_at_in_epoch_millis": 1757404349787
      }
    }
  ]
}

Note: The last_succeeded_at_in_epoch_millis timestamp is 30 days ago:

# LC_ALL=C date --date=@1758700716
Wed Sep 24 09:58:36 CEST 2025

Trying to update the definition as per https://docs.opensearch.org/latest/ingest-pipelines/processors/ip2geo/#updating-an-ip2geo-data-source (quoting the URL) fails with the following output:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "datasource will expire at 2025-10-24T07:58:36.603Z with the update interval"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "datasource will expire at 2025-10-24T07:58:36.603Z with the update interval"
  },
  "status": 400
}

Note: The date 2025-10-24 was three days ago.

Checking the URL manually with curl (curl -o - https://geoip.maps.opensearch.org/v1/geolite2-country/manifest.json) gives a Cloudfront error:

Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.

Note: I've tried from various ISPs and networks and the exact same error occurs.

Related component

Plugins

To Reproduce

curl -o - https://geoip.maps.opensearch.org/v1/geolite2-country/manifest.json

Expected behavior

I expect the ip2geo data sources to work.

Additional Details

Plugins

# /usr/share/opensearch/bin/opensearch-plugin list
opensearch-alerting
opensearch-anomaly-detection
opensearch-asynchronous-search
opensearch-cross-cluster-replication
opensearch-custom-codecs
opensearch-flow-framework
opensearch-geospatial
opensearch-index-management
opensearch-job-scheduler
opensearch-knn
opensearch-ltr
opensearch-ml
opensearch-neural-search
opensearch-notifications
opensearch-notifications-core
opensearch-observability
opensearch-performance-analyzer
opensearch-reports-scheduler
opensearch-search-relevance
opensearch-security
opensearch-security-analytics
opensearch-skills
opensearch-sql
opensearch-system-templates
opensearch-ubi
prometheus-exporter
query-insights

Host/Environment (please complete the following information):

• OS: Ubuntu
• Version 22.04.5 LTS

Additional context

• OpenSearch v3.3.1 from apt repo

[msfroh]

@opensearch-project/admin Can we please move this to https://github.com/opensearch-project/geospatial, since the ip2geo feature is implemented in https://github.com/opensearch-project/geospatial/tree/main/src/main/java/org/opensearch/geospatial/ip2geo?

[heemin32]

@bruberg Could you share the opensearch log around "last_failed_at_in_epoch_millis": 1761292895482, to see what is the root cause of the failure?

The update of datasource is updating its configuration not the actual datasource. As your datasource is already expired, you cannot update its configuration. One option is you can recreate the datasource.

The external repository works fine. You can try the web browser to access the file instead of curl. https://geoip.maps.opensearch.org/v1/geolite2-country/manifest.json

[bruberg]

The first error about expiry is this (log timestamps in CEST):

[2025-10-24T09:58:40,246][WARN ][o.o.g.i.d.Ip2GeoCachedDao] [servername] Datasource city-datasource is expired. Expiration date is 2025-10-24T07:58:36.603Z and now is 2025-10-24T07:58:40.244933846Z.

Then a few minutes later, where the second line almost matches the "last_failed_at_in_epoch_millis": 1761292895482 timestamp:

[2025-10-24T10:01:34,022][WARN ][o.o.g.i.d.Ip2GeoCachedDao] [servername] Datasource city-datasource is expired. Expiration date is 2025-10-24T07:58:36.603Z and now is 2025-10-24T08:01:34.022533622Z.
[2025-10-24T10:01:35,507][INFO ][o.o.g.i.d.Ip2GeoCachedDao] [servername] Updated datasource metadata for datasource city-datasource successfully.

If it can help explaining anything, my cluster was down between October 17th and October 23rd due to opensearch-project/OpenSearch#19671.

Thanks for the information about recreating the datasource, I'll try that next.

[bruberg]

I've removed the three geospatial datasources (city, country, asn) and recreated them, and two of them are recreated without problems. However, the "GeoLite2 City" database fails upon recreation every time. Is there anything I can do to further debug this problem?

I'm creating the datasource as follows:

curl --netrc --insecure -XPUT 'https://localhost:9200/_plugins/geospatial/ip2geo/datasource/city-datasource' -H 'Content-Type: application/json' -d @geoip.city-datasource.json

This is the content of geoip.city-datasource.json:

{
  "endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
  "update_interval_in_days": 3
}

After about 8 minutes, the datasource is listed as failed. This is the log message from when the creation fails:

[2025-10-28T13:17:07,842][ERROR][o.o.g.i.a.PutDatasourceTransportAction] [servername] Failed to create datasource for city-datasource
org.opensearch.OpenSearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.
	at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96) ~[opensearch-3.3.1.jar:3.3.1]
	at org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:79) ~[opensearch-3.3.1.jar:3.3.1]
	at org.opensearch.action.support.AdapterActionFuture.actionGet(AdapterActionFuture.java:73) ~[opensearch-3.3.1.jar:3.3.1]
	at org.opensearch.geospatial.ip2geo.dao.GeoIpDataDao.freezeIndex(GeoIpDataDao.java:125) ~[opensearch-geospatial-3.3.1.0.jar:3.3.1.0]
	at org.opensearch.geospatial.ip2geo.dao.GeoIpDataDao.putGeoIpData(GeoIpDataDao.java:309) ~[opensearch-geospatial-3.3.1.0.jar:3.3.1.0]
	at org.opensearch.geospatial.ip2geo.jobscheduler.DatasourceUpdateService.updateOrCreateGeoIpData(DatasourceUpdateService.java:94) ~[opensearch-geospatial-3.3.1.0.jar:3.3.1.0]
	at org.opensearch.geospatial.ip2geo.action.PutDatasourceTransportAction.createDatasource(PutDatasourceTransportAction.java:164) [opensearch-geospatial-3.3.1.0.jar:3.3.1.0]
	at org.opensearch.geospatial.ip2geo.action.PutDatasourceTransportAction$1.lambda$onResponse$0(PutDatasourceTransportAction.java:135) [opensearch-geospatial-3.3.1.0.jar:3.3.1.0]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:545) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328) [?:?]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:916) [opensearch-3.3.1.jar:3.3.1]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:1447) [?:?]
Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.
	at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:257) ~[opensearch-3.3.1.jar:3.3.1]
	at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:82) ~[opensearch-3.3.1.jar:3.3.1]
	at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:94) ~[opensearch-3.3.1.jar:3.3.1]
	... 13 more
[2025-10-28T13:17:07,846][INFO ][o.o.g.s.PluginClient     ] [servername] Running transport action with subject: plugin:org.opensearch.geospatial.plugin.GeospatialPlugin
[2025-10-28T13:17:07,854][INFO ][o.o.g.i.d.Ip2GeoCachedDao] [servername] Updated datasource metadata for datasource city-datasource successfully.

Since this is the database that failed earlier, could this be caused by some remaining files somewhere? Can I increase debugging for this plugin somehow?

[heemin32]

Seems like the instance you are using is under scaled to handle the process. Try to increase the plugins.geospatial.ip2geo.timeout value in the cluster setting. The default is 30 seconds. Maybe, try 1 minutes or more.

https://docs.opensearch.org/latest/ingest-pipelines/processors/ip2geo/#cluster-settings

[bruberg]

Increasing the timeout helped, thank you!

However, this has worked in this cluster for a long time, at least until I upgraded to OpenSearch 3.x. Furthermore, the Opensearch node on which I've found this issue runs on a node with 24 CPU cores server and 190G RAM, and OpenSearch's Java heap is set to 30G.

For reference, I installed the City datasource in a triple-node OpenSearch cluster (OpenSearch 2.19.3) where each virtual server has 2 CPU cores and 4G RAM, with a Java heap size of as little as 1,6G RAM. Here, the City datasource installed successfully in about one minute.

To me this indicates that something else is wrong with the cluster or node where it failed.

[heemin32]

@bruberg Hard to tell why it failed after 3.x upgrade. At least there was no major changes in geospatial plugin between 2.19 and 3.x