docs: Add missing descriptions, sources, and licenses

[dsmedia]

Pull Request: Add missing descriptions, sources, and licenses to datapackage files

Objective:

Following up on the metadata infrastructure work in #634, #639, and #646, this PR adds missing description, source, and license metadata to datapackage_additions.toml. The checklist below indicates which metadata entries are missing and still need to be added. Schema entries (describing each dataset column) will be handled in a subsequent pull request.

Dataset descriptions written to avoid formulations such as "This dataset...", consistent with PEP 257 – Docstring Conventions

✅ PR Merged - 🚧 Work Remaining

Descriptions were added for all datasets.
Resources still missing sources:

• flare-dependencies.json, flare.json, movies.json, sp500.csv, stocks.csv, udistrict.json, weekly-weather.json, windvectors.csv

Resources still missing licenses:

• anscombe.json, driving.json, flare-dependencies.json, flare.json, la-riots.csv, movies.json, ohlc.json, platformer-terrain.json, sp500-2000.csv, sp500.csv, stocks.csv, udistrict.json, volcano.json, weekly-weather.json, windvectors.csv

Open Questions: Complete

• Decide how to handle license information that cannot be determined after research, ensuring validation with frictionless standard for licenses.
• [ ] add disclaimer about license information deferring to separate PR
• Regenerated datackapage.json and datapackage.md

Status:

The following checklist indicates the completion status of the description, sources, and licenses metadata for each dataset. A green checkmark (✅) indicates the metadata is present, a red X (❌) indicates the metadata is missing. The leading checkbox is only checked if all three types of metadata are present.

Process:
Changes are validated using scripts/build_datapackage.py, which generates machine-readable metadata describing the contents of /data/.

Legend:

• ✅ - Indicates the metadata is present
• ❌ - Indicates the metadata is missing
• [x] - Indicates all three types of metadata are present
• [ ] - Indicates one or more types of metadata are missing

Checklist:

• 7zip.png: description✅, sources✅, licenses✅
• airports.csv: description✅, sources✅, licenses✅
• annual-precip.json: description✅, sources✅, licenses✅
• anscombe.json: description✅, sources✅, licenses❌
• barley.json: description✅, sources✅, licenses✅
• birdstrikes.csv: description✅, sources✅, licenses✅
• budget.json: description✅, sources✅, licenses✅
• budgets.json: description✅, sources✅, licenses✅
• burtin.json: description✅, sources✅, licenses✅
• cars.json: description✅, sources✅, licenses✅
• co2-concentration.csv: description✅, sources✅, licenses✅
• countries.json: description✅, sources✅, licenses✅
• crimea.json: description✅, sources✅, licenses✅
• disasters.csv: description✅, sources✅, licenses✅
• driving.json: description✅, sources✅, licenses❌
• earthquakes.json: description✅, sources✅, licenses✅
• ffox.png: description✅, sources✅, licenses✅
• flare-dependencies.json: description✅, sources❌, licenses❌
• flare.json: description✅, sources❌, licenses❌
• flights-10k.json: description✅, sources✅, licenses✅
• flights-200k.arrow: description✅, sources✅, licenses✅
• flights-200k.json: description✅, sources✅, licenses✅
• flights-20k.json: description✅, sources✅, licenses✅
• flights-2k.json: description✅, sources✅, licenses✅
• flights-3m.parquet: description✅, sources✅, licenses✅
• flights-5k.json: description✅, sources✅, licenses✅
• flights-airport.csv: description✅, sources✅, licenses✅
• football.json: description✅, sources✅, licenses✅
• gapminder-health-income.csv: description✅, sources✅, licenses✅
• gapminder.json: description✅, sources✅, licenses✅
• gimp.png: description✅, sources✅, licenses✅
• github.csv: description✅, sources✅, licenses✅
• global-temp.csv: description✅, sources✅, licenses✅
• income.json: description✅, sources✅, licenses✅
• iowa-electricity.csv: description✅, sources✅, licenses✅
• jobs.json: description✅, sources✅, licenses✅
• la-riots.csv: description✅, sources✅, licenses❌
• londonBoroughs.json: description✅, sources✅, licenses✅
• londonCentroids.json: description✅, sources✅, licenses✅
• londonTubeLines.json: description✅, sources✅, licenses✅
• lookup_groups.csv: description✅, sources✅, licenses✅
• lookup_people.csv: description✅, sources✅, licenses✅
• miserables.json: description✅, sources✅, licenses✅
• monarchs.json: description✅, sources✅, licenses✅
• movies.json: description✅, sources❌, licenses❌
• normal-2d.json: description✅, sources✅, licenses✅
• obesity.json: description✅, sources✅, licenses✅
• ohlc.json: description✅, sources✅, licenses❌
• penguins.json: description✅, sources✅, licenses✅
• platformer-terrain.json: description✅, sources✅, licenses❌
• political-contributions.json: description✅, sources✅, licenses✅
• population.json: description✅, sources✅, licenses✅
• population_engineers_hurricanes.csv: description✅, sources✅, licenses✅
• seattle-weather-hourly-normals.csv: description✅, sources✅, licenses✅
• seattle-weather.csv: description✅, sources✅, licenses✅
• sp500-2000.csv: description✅, sources✅, licenses❌
• sp500.csv: description✅, sources❌, licenses❌
• stocks.csv: description✅, sources❌, licenses❌
• udistrict.json: description✅, sources❌, licenses❌
• unemployment-across-industries.json: description✅, sources✅, licenses✅
• unemployment.tsv: description✅, sources✅, licenses✅
• uniform-2d.json: description✅, sources✅, licenses✅
• us-10m.json: description✅, sources✅, licenses✅
• us-employment.csv: description✅, sources✅, licenses✅
• us-state-capitals.json: description✅, sources✅, licenses✅
• volcano.json: description✅, sources✅, licenses❌
• weather.csv: description✅, sources✅, licenses✅
• weekly-weather.json: description✅, sources❌, licenses❌
• wheat.json: description✅, sources✅, licenses✅
• windvectors.csv: description✅, sources❌, licenses❌
• world-110m.json: description✅, sources✅, licenses✅
• zipcodes.csv: description✅, sources✅, licenses✅

[dsmedia]

Here is the code to validate the statistical description of normal-2d.json added in a86b9dd.

import pandas as pd
from scipy import stats

# Read data
df = pd.read_json("https://raw.githubusercontent.com/vega/vega-datasets/main/data/normal-2d.json")

# Generate key statistics
stats_output = {
    "Sample size": len(df),
    "Means": df.mean().round(3).to_dict(),
    "Standard deviations": df.std().round(3).to_dict(),
    "Correlation": round(df.corr().iloc[0, 1], 3),
    "Ranges": {col: [round(df[col].min(), 3), round(df[col].max(), 3)] for col in df.columns},
    "Normality p-values": {col: round(stats.normaltest(df[col]).pvalue, 3) for col in df.columns}
}

# Print statistics
print("Dataset Statistics for Description:")
print(f"Sample size: {stats_output['Sample size']} points")
print(f"Centers: {stats_output['Means']}")
print(f"Standard deviations: {stats_output['Standard deviations']}")
print(f"Correlation: {stats_output['Correlation']}")
print(f"Ranges: {stats_output['Ranges']}")
print(f"Normality test p-values: {stats_output['Normality p-values']}")

...which produced:

Dataset Statistics for Description:
Sample size: 500 points
Centers: {'u': 0.005, 'v': -0.011}
Standard deviations: {'u': 0.192, 'v': 0.199}
Correlation: 0.026
Ranges: {'u': [-0.578, 0.533], 'v': [-0.534, 0.606]}
Normality test p-values: {'u': 0.68, 'v': 0.763}

[dsmedia]

Some really great stuff in here @dsmedia!

Let me know if you need any clarifications

Have gone through and addressed all I believe - appreciate the diligent review @dangotbanned !

[dangotbanned]

Some really great stuff in here @dsmedia!
Let me know if you need any clarifications

Have gone through and addressed all I believe - appreciate the diligent review @dangotbanned !

Thanks @dsmedia, hopefully will have a chance to look it over tomorrow

[dangotbanned]

Some really great stuff in here @dsmedia!
Let me know if you need any clarifications

Have gone through and addressed all I believe - appreciate the diligent review @dangotbanned !

Thanks @dsmedia, hopefully will have a chance to look it over tomorrow

@dsmedia all the new changes are great 🎉

Reading through the new (https://github.com/vega/vega-datasets/blob/dbecc75a57d3a5cc0bbeea1861b0254d3be4e15c/datapackage.md) some more things stood out to me.
Hopefully they should be easy to resolve

[dangotbanned]

Thanks @dsmedia, feel free to merge whenever you're ready (and #672)