OOD files missing geometry; problems introduced between July 2023 and January 2024

[peterdresslar]

We are running scripts that are running into many cases where prior year data is changed significantly when we rerun our filtering and processing scripts. Sometime between our script runs executed in July 2023 and our runs executed in January 2024, something seems to have caused a fairly large number of the data points to be missing their geometry. So, for instance:

{ "type": "Feature", "properties": { "quadkey": "0223000333012201", "avg_d_kbps": 150342, "avg_u_kbps": 11570, "avg_lat_ms": 17, "tests": 2, "devices": 2 }, "geometry": null },

This datum, when previously accessed, looked like this.

{ "type": "Feature", "properties": { "quadkey": "0223000333012201", "avg_d_kbps": 150342, "avg_u_kbps": 11570, "avg_lat_ms": 17, "tests": 2, "devices": 2 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -154.945678710938012, 19.580493479202499 ], [ -154.940185546875, 19.580493479202499 ], [ -154.940185546875, 19.575317892869499 ], [ -154.945678710938012, 19.575317892869499 ], [ -154.945678710938012, 19.580493479202499 ] ] ] } },

It looks like the issue is in 2020Q4, 2021Q4, and 2022Q1, at least. The issue appears in both fixed and mobile data. The quadkey "0223000333012201" looks perfectly fine in the most recent files.

There are more examples between commits in our analysis repo: https://github.com/peterdresslar/ookla-open-data-notebook.

Just to make sure to be clear, this is an issue we have encountered when retrieving data from https://registry.opendata.aws/speedtest-global-performance/ . It is not directly related to the code in this repo, but would possibly be encountered by users of the repo.

[erbbri]

Hi Peter,
Thanks for reaching out, we are currently looking into this.
I'm trying to replicate your error and wanted to check on a few things:

• Judging by your notebook, it looks like you are accessing the data via URL in shapefile format, correct?
• In the example you provided, is that specific instance found in Q4 2020 for fixed data?

[peterdresslar]

Edit: Thanks @erbbri ...

1. Accessing the shapefiles, yes, i.e.,

def get_tile_url(service_type: str, year: int, q: int) -> str:
    dt = quarter_start(year, q)
    base_url = (
        "https://ookla-open-data.s3-us-west-2.amazonaws.com/shapefiles/performance"
    )
    url = f"{base_url}/type%3D{service_type}/year%3D{dt:%Y}/quarter%3D{q}/{dt:%Y-%m-%d}_performance_{service_type}_tiles.zip"
    return url

2. Yes, 2020Q4 fixed. Same issue in 2021Q4 for that Quadkey. However, 2022Q1 is OKAY for that QK.

Again, these QKs had geometry when we ran the same scripts in July 2023. I see similar missing geometries in the same quarters in some of the mobile data, but we did not run that side in July 2023. There are a few similar issues in 2020Q3, but they appear in far fewer of our data points. I suppose the whole thing could be a weird processing issue on our end, but we are basically filtering the tiles into a file without processing on the tiles themselves.

[erbbri]

Hi Peter,

Sincerest apologies for the delay in replying.

Our team looked into the issue you were seeing, and unfortunately we were not able to replicate it when we pulled down the data in R. The geometry for the shapefiles seems to be intact for this quadkey and quarter:

We also looked for any for null geometries as well for the quarters you mentioned and were unable to find any. I am wondering if there may be an issue with how the shapefile is processing in your code. Unfortunately we are more of an R-based team, so I cannot speak fully as to what is happening there.

We have refreshed the data for those quarters though, so that may potentially solve your issue.

[peterdresslar]

Thanks very much for the reply (and the reminder that it's time for a quarterly update.) I'll rerun everything this weekend and take a look. Not sure AI is quite at the point where it could resolve this issue, but...

…

On Thu, Apr 25, 2024 at 9:34 AM Bridget Erb ***@***.***> wrote: Hi Peter, Sincerest apologies for the delay in replying. Our team looked into the issue you were seeing, and unfortunately we were not able to replicate it when we pulled down the data in R. The geometry for the shapefiles seems to be intact for this quadkey and quarter: Screenshot.2024-01-22.at.4.15.33.AM.png (view on web) <https://github.com/teamookla/ookla-open-data/assets/48925623/b5c4eca7-ce7c-40db-b95f-cec503da8dc2> We also looked for any for null geometries as well for the quarters you mentioned and were unable to find any. I am wondering if there may be an issue with how the shapefile is processing in your code. Unfortunately we are more of an R-based team, so I cannot speak fully as to what is happening there. We have refreshed the data for those quarters though, so that may potentially solve your issue. — Reply to this email directly, view it on GitHub <#28 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAP7W2XMDMWVRP5FDK4PQE3Y7FLELAVCNFSM6AAAAABCEGVNS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZYGAZTOMZTHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>