Standardize dataset metadata in DATASETS.toml - #70
Draft
user27182 wants to merge 97 commits into
Draft
Conversation
Follows the REUSE convention: one plain-text file per licence, named by its SPDX identifier. Standard licence texts come from the SPDX license list. Terms that SPDX does not list are recorded as `LicenseRef-` identifiers: the Stanford 3D Scanning Repository conditions of use, the Utah Teapot's public-domain-by-convention status, SimScale's public project terms, NASA SVS media usage, the Open Data Commons Database Contents License, and a statement of what `LicenseRef-Unknown` means. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A single central table records, for each of the 200 datasets in this repository, what the data is, where it came from, who made it, what licence it carries, what changed between the source and the copy here, and how confident that record is. Every one of the 1258 tracked files under `Data/` is claimed by exactly one `[[dataset]]` block. The licensing keys follow the REUSE specification, so `SPDX-License-Identifier` and `SPDX-FileCopyrightText` carry their standard meaning. Two lookup tables sit alongside the datasets: `[license.*]` gives each licence a title, a URL, a path to its full text and flags for commercial use, attribution and ShareAlike; `[collection.*]` names the upstreams that several datasets share. `provenance` is either "verified", "inferred" or "unknown", so a researched record is distinguishable from a reasoned guess and from a genuine gap. 54 datasets carry `LicenseRef-Unknown`, and each says in `notes` what was established, what was not, and what a downstream user should do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Removes the `LICENSE`, `License`, `LICENSE.txt`, `LICENSE_PARAVIEW` and `*.license` files under `Data/`. Everything they recorded is now a field in `DATASETS.toml`: the source URL in `source_url`, the licence in `SPDX-License-Identifier`, the required credit in `attribution`, the caveats in `notes`, and the full licence text in `LICENSES/`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Removes the `README`, `README.md`, `README.txt`, `README.VTK.txt` and `CITATION.txt` files under `Data/`. Their descriptive content is now the `description`, `source_url` and `notes` fields of the matching `[[dataset]]` block, the processing they documented is `modification`, and the works they asked to be cited are `references`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Checks that DATASETS.toml is well formed and complete. It verifies that every tracked file under `Data/` is claimed by exactly one `[[dataset]]`, that no block carries an unknown or missing key, that names are unique and the blocks are sorted, that every `SPDX-License-Identifier` resolves to a `[license.*]` table with its text on disk, and that the conditionally required fields are present: `attribution` when the licence requires credit, `modification` when `modified` is set, `notes` when the provenance is not verified, `source_url` unless the provenance is unknown. It also rejects `LICENSE`, `README`, `CITATION` and `.license` files under `Data/`. Each failure prints the place, the problem and a concrete remedy, including a ready-to-paste `[[dataset]]` block for an uncovered file. Needs only Python 3.11 or newer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs tools/validate_datasets.py on every pull request and on pushes to master. The checkout is blobless, so the job does not download the data itself. On failure the workflow writes a job summary explaining the rule, the command to reproduce the check locally, the validator's full output and a link to CONTRIBUTING.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CONTRIBUTING.md gives the field reference, two worked examples, guidance on choosing a provenance value, the licence requirements for new data, and how to add a licence that is not yet declared. The README explains what DATASETS.toml is and shows how to read one dataset's record downstream. The pull request template now asks for the fields the validator checks rather than for a prose licence file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The forbidden-file set matched exact, case-sensitive names, so `license.txt`, `Readme.md`, `CITATION.md` and `NOTICE.txt` all passed. Match on the case-folded stem instead, which also covers `LICENCE`, `COPYING` and `data.License`. `fnmatch` does not treat `/` specially, so `dir/*` matched everything nested under `dir/` and was indistinguishable from `dir/**`. A dataset written with `dir/*` therefore claimed files added under it later, and the coverage check never reported them. Compile the pattern to a regex where `*` and `?` stop at a separator and `**` crosses one, which is the meaning CONTRIBUTING.md documents. Both were reachable together: a directory holding a mesh, a `license.txt` and a `Readme.md`, claimed with `path = ["newthing/*"]`, validated clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`main` caught only `TOMLDecodeError`, so a value of the wrong shape reached the checks and crashed. Writing `license = "CC-BY-4.0"` — an easy slip given that the licence tables use quoted keys — raised `AttributeError` from `.items()`, and a `path` written as a string was iterated character by character, reporting one nonsense pattern per character. Check the top-level shape and the array-valued fields first, and stop there when they are wrong, so a contributor gets the same report format as every other failure rather than a traceback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The recorded thing ID was reconstructed from the model name and never confirmed. It is wrong: `thing:1420661` has no Wayback capture at all, while `thing:1541337` is titled "Victorian Goblet face illusion by Alzibiff" and matches the licence label the bundled file carried. The publication date, 5 May 2016, also settles the version the licence label omits: it postdates CC 4.0, so the CC-BY-4.0 reading stands on a date rather than on a guess. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Stanford page carries two separate conditions and only the commercial one was recorded. The other names these two models directly: the dragon as a symbol of Chinese culture and Lucy as a Christian angel, with a request not to animate, morph, apply Boolean operators to, or simulate nasty things happening to them. It binds in practice — `examples/01-filter/clip_closed_surface.py` clips Lucy — and a gallery card built from these entries would have told a reader that commercial use was the only constraint. The bunny entry already recorded the exemption from this restriction, so only the restriction itself was missing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`misc_pyvista_era` declined to confirm whether the top-level `mars.jpg` and `stars.jpg` were duplicates of the copies under `solar_textures/` and `planet3d-matlab/`. They are byte-identical, so both were recorded as having undetermined terms when they are in fact CC-BY-4.0 (Solar System Scope) and MIT (Tamas Kis). Move each into the entry that owns its original, which removes two files from the undetermined bucket. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It is not a time series and not derived from `bunny.ply`. All 64 files hold the same 872-point mesh, which compares equal to `Bunny.vtp`; what varies between them is field data named `Camera:Position`, `Actor:Orientation`, `Property:SpecularPower` and the like, so this is a scene export fixture. `ExportBunny.vtp` is also a `vtkMultiBlockDataSet`, not PolyData. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
UNISIM licenses the database under the ODbL and its contents under the DbCL. Only the DbCL half was recorded, whose table says no attribution and no ShareAlike — so this entry failed CONTRIBUTING.md's own rule that a ShareAlike licence must carry `share_alike = true` and a warning, and a gallery card would have told a reader no credit was needed. The source also makes citation mandatory rather than requested, in any publication or report of a study that uses the benchmark. That is now in `attribution`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The deleted `Data/DICOM_Stack/CITATION.txt` carried a statement the CPTAC program asks publications to include, and it survived nowhere in DATASETS.toml. It is an obligation rather than background, so it belongs in `attribution` alongside the data citation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Each was traceable, and three of the four move out of the undetermined bucket. `biplane` and `openvdb_sphere_points` are VTK's own reader test inputs, named in `IO/IOSS/Testing/CMakeLists.txt` and `IO/OpenVDB/Testing/CMakeLists.txt`, so they sit on the same footing as the rest of the VTK Data collection. `step_model` is `tests/testdata/red_cube_blue_cylinder.step` in the CadQuery repository, which is Apache-2.0 — though its STEP header shows a translation-service export rather than a CadQuery one, so the note says CadQuery may itself be a redistributor. `netcdf` stays undetermined but is no longer anonymous: its own global attributes identify it as IPSL-CM4_v1 output for the IPCC Fourth Assessment SRES A2 experiment, so CMIP3 terms attach to it. Its `source_url` had also moved to the Unidata archive host. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The grey nurse shark was published on 22 August 2013, not in June; the archived Thingiverse page carries the timestamp. The CC BY-SA 3.0 inference still holds, since August still precedes CC 4.0. The date appeared in CONTRIBUTING.md's worked example too. `cow` described a collection README that this branch deletes. `bunny_coarse` said only "decimated and converted". It is also scaled by a factor of two and recentred: 35947 points to 872, with extent ratios of 2.0021, 1.9968 and 2.0014 against `bunny.ply`. That is the part a user comparing the two datasets would trip over. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A `source_url` a reader cannot follow to the stated terms is worse than none,
because the record looks checkable and is not.
- `magnetic_coil_field` — the linked magpylib example page 404s; point at
the project instead, and say plainly that a downstream user is unlikely to
be bound by a solver's licence for output it computed.
- `tango_icons` — `tango.freedesktop.org` returns 400; use the archived page
that still carries the licence statement.
- `stars_cloud_hyg` — the GitHub repository now redirects readers to
Codeberg and carries no licence text at all, so a "verified" entry could
not be reproduced from it. Codeberg's README states the version rule that
makes CC BY-SA 2.5 the right licence for `hygdata_v3.csv`.
- `washington_bust` — pointed at the Smithsonian collection root, which says
nothing about this object; use the object page, as `lincoln_life_mask`
already did.
- `earth_models` — cited ETOPO5 while the files are the ten-minute product,
on a host that now redirects.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
None of these changes a licence except `cad_model_case`, but each replaces a
statement about what was not checked with one about what is now known.
- The Laser Design note said no terms were found anywhere on the site. The
site does carry terms pages and a CyberOptics copyright assertion; they
cover purchase and website use, not reuse of the models, so the conclusion
holds but the reasoning was wrong.
- `great_white_shark` — the candidate Thingiverse upload exists, and its own
uploader disclaims authorship and names a CGTrader page as the origin.
That is a stronger reason to leave the licence undetermined than "not
independently verified" was.
- `cad_model_case` moves to verified: the archived page's structured data
names the CC BY 4.0 licence URL directly, so the version is stated at the
source rather than inferred.
- `dicom_stack` — the collection page lists licences per data type and the
image rows are not uniform, so the note now says which rows CC BY 3.0
comes from. The Zenodo dataset DOI is added to `whole_body_ct`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file said "quoted verbatim from the repository page" over text that had been edited: "However," was dropped from the start of the restriction, "without our permission" was expanded to name the laboratory, and the sentence granting the right to publish images — the one the `attribution` field depends on — was missing entirely. Restore the passage as the page states it, and add the "Inappropriate uses of these models" section, which carries the cultural-significance request and the bunny and armadillo exemption. In a licence text file, verbatim has to be literal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page is generated from `dataset_gallery.rst`, so the published URL uses an underscore. Both files linked the hyphenated form, which 404s — and it is the link a contributor is most likely to follow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`provenance` describes the origin only, and the nine Laser Design scans are `verified` with an undetermined licence — a certain source that publishes no terms. Anything rendering a badge from this file should take "can I use this?" from `SPDX-License-Identifier`, not from `provenance`, and the field reference now says so. The `path` rules were described as "`dir/**` matches everything under `dir/`", which left the meaning of `dir/*` to whatever the validator happened to do. A downstream consumer has to resolve a file to its dataset with these rules, so they are part of the format: `*` and `?` stop at a separator and only `**` crosses one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An unqualified Apache-2.0 `LICENSE` at the root of a repository that now documents a CC BY-NC-SA dataset inside it is the contradiction this branch exists to remove. The README now says what each root file covers: `LICENSE` came with the 2019 VTKExamples import and covers that material and the tooling, `Copyright.txt` is VTK's notice for the inherited collection, and neither licenses `Data/` as a whole. The REUSE wording was narrowly true and read as more than it claimed. This repository borrows REUSE's key names and its `LICENSES/` layout; it is not REUSE-covered, because REUSE wants per-file sidecars and 1258 of those would be unreviewable next to one table. Saying so is better than implying otherwise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`filter: blob:none` still materialises every blob in the working tree, so the job was downloading 1.8 GB of data to read an index. The validator only needs `git ls-files`, which reads the index and is unaffected by a sparse worktree. Excluding `Data/` gives a 2.1 MB checkout, and the validator's output is unchanged: 200 datasets cover 1258 files under Data/, with `Data/` absent from disk. Also drop the checkout credentials, which this job never uses. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All nine scans were recorded as `provenance = "verified"`, but only seven appear on the current sample-files listing. `woman` and `drill` do not, so the only evidence they came from Laser Design is PyVista's own docstring. That is an inference, not a confirmation, and the notes now say so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`license_terms` dropped the `WITH` keyword but kept the exception after it, so `Apache-2.0 WITH LLVM-exception` demanded a `[license.LLVM-exception]` table and a licence text that will never exist. No entry uses an exception today; the first one to would have been rejected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The entry claimed a NonCommercial clause that neither source states. The artists' project page says "The 3d data is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License" and links creativecommons.org/licenses/by-sa/4.0; the archived original `nefertitihack.alloversky.com` page, which hosted the very `.obj` this repository ships, says the same thing and links the same URL. The NC reading came from the sidecar file this branch deletes, not from the source, and it was recorded as `provenance = "verified"` — meaning the page had been read. The notes then built on it: "not compatible with commercial redistribution" and "no permissively licensed Nefertiti scan is currently known to exist" are both wrong. Commercial redistribution is permitted; only the ShareAlike obligation propagates. This was the repository's only non-commercial dataset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two claims on an entry marked `provenance = "verified"` did not survive checking. `Data/DICOM_Stack/metadata.csv`, which ships in this repository, records `Modality = MR`, `Study Description = FOREARMROUTINE`, `Series Description = AX SCOUT`, `Number of Images = 3`. It is a three-slice MR scout of a forearm, not a stack of CT slices. The collection page lists CC BY 4.0 for the DICOM images as of Version 10, updated 2023-05-10. CC BY 3.0 was the licence when the sidecar file this entry replaces was written; the recommended citation has likewise moved to the 2019 Version 10 form. Also adds the Zenodo dataset DOI to `whole_body_ct`, which CC BY attribution for the dataset needs and which the paper DOI does not cover. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Commit a8f3faa described four changes and contained one: the script that wrote them raised partway through and only the great white shark edit was re-applied. This lands the two that were lost and still stand. The DICOM stack pair is in the preceding commit. The Laser Design note said no terms were found elsewhere on the site. The "Terms of Use" link in the sample-file page's own footer goes to CyberOptics' website terms, which grant permission to download one copy "for personal, non-commercial transitory viewing only" and forbid using the materials "for any commercial purpose" or transferring or mirroring them. Whether that boilerplate is meant to reach the sample downloads is arguable, and the conclusion is unchanged — no grant of redistribution rights exists anywhere on the site. But the note asserted a fact about the site that the site contradicts, one click from `source_url`. `cad_model_case` reads CC BY 4.0 from the licence URL in the page's own structured data rather than from the version-less label. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All six said the archive held a "full-resolution mesh". It does not: the zips were already decimated to keep each archive under 50 MB, and the `_decimated` files are a second, much heavier reduction that no field described. Measured point counts are now recorded for both. `black_vase` said "converted without decimation", contradicted by both files. `bird_bath`'s "decimated to roughly 1.8 million points" described the zip, not the file the name refers to. `owl` and `sea_vase` read "with float32 coordinates and converted to float32 coordinates", and `owl.zip` is in fact float64. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
All six faces are byte-identical to VTK's `Testing/Data/skybox/`, where VTK ships a `readme.txt` beside them naming Humus and CC BY 3.0. The entry told downstream users to credit "VTK Examples" under Apache-2.0 instead, which a collection cannot do to a third party's work. `cubemap_park` already carried the same author and licence from a readme inside its own archive. The two `skybox1` entries get a note saying the sibling set turned out this way, since their own Apache-2.0 rests on the same kind of assumption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The hashing section said a match against VTK's content links proves the collection's notice covers the file. It does not, and the worked example was `skybox2-negx.jpg`, which VTK ships with a readme naming Emil Persson and CC BY 3.0. Look for the sibling readme first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`dualSphereNoTime.pvd` carries `<!-- This file is not from paraview. -->` on its second line, and `dualSphereAnimation4NoPart.pvd` came from pull request 24 as a reader edge case for a PyVista test. Neither name exists in ParaView's data repository, so the Sandia and Kitware copyright the `PVD/**` glob gave them was asserted over files PyVista contributors wrote. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Natural Earth II is Tom Patterson's, and his terms page renounces all claim to it; the Gettysburg Address is Lincoln's, from 1863. Both were inside bulk entries that put Kitware's copyright or the VTK Examples Apache-2.0 over them, which no redistributor can do to public-domain work. The new licence identifier asks each entry to say in `notes` why the work is in the public domain, because the reason decides who the status binds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The entry knew USGS models are public domain but hedged because "the specific quadrangle and vintage were not recorded upstream". They are recorded, in the file's own Type A header: Mount St. Helens post-eruption, 6 September 1980, 30 metre grid. Kitware's collection notice was attaching a copyright to a work that has none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`horse.ply` is byte-identical to the horse the Georgia Tech Large Geometric Models Archive serves, which says only that the model "appears here courtesy of Cyberware, Inc." — permission to host, not a licence. It was sitting under a blanket Apache-2.0 that the redistributor could not grant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both sat in the PVGeo bulk entry as undetermined. The pickle records
`created_with: OSMnx 1.1.1` and the pull request that added it publishes the
`graph_from_address('Holzgerlingen DE', ...)` call, so the contents are
OpenStreetMap's under ODbL. The OMF file has the same git blob hash as
`assets/test_file.omf` in pyvista/omfvista, which ships a licence sidecar
naming MIT and the Global Mining Guidelines Group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The field never held a download link — 186 of its 188 values point at a page about the work, and nothing in the table records where a file is served from. PyVista's dataset loader already uses `source_url_blob` and `source_url_raw` for that, so the old name collided with a real meaning on the reading side, and `Example` was already exposing these as `origin_url` and `origin_title`. CONTRIBUTING now says what the field is: the furthest upstream point that could be established, with `redistributed_from` for the route a copy took. 116 entries name a redistributor there because the trail ran out. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Its 5926 by 2963 is the same unusual size as NASA's global colour mosaic, and the two decode to the same picture — 0.03 out of 255 mean difference, 1 at most, which is only re-encoding. Credit goes to NASA, JHUAPL and SwRI. The NASA licence table was specific to the Scientific Visualization Studio; the guidelines it quotes are agency-wide, so it is now `LicenseRef-NASA-Media` and covers mission imagery too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PVGeo's example scripts kept each original `trainingimages.org` URL as a comment beside the pyvista/data URL that replaced it, and all six files are named there. The library grants research use only, so these were not unlicensed but restricted, which is a different thing to tell a user. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`Example.metadata` was renamed away and pyvista/pyvista#9118 is still open, so both the README and the table header promised an API that does not exist yet. The forbidden-file rule also read as absolute: it applies to tracked files, not to a readme inside an archive, and several archives here carry one as the evidence for their entry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`solar_textures` still claimed Pluto after the texture moved to its own entry, and the VTK Examples bulk description did not account for `GitLfsInstalled`, a fifteen-byte probe marker swept in by the 2019 import. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Its projection, tiepoint beside Ruapehu's crater, 4.2333 metre pixel scale — 1:50,000 at 300 dpi, LINZ's published Topo50 raster spec — and the `BJ34_GeoTifv` filename convention all point at the same sheet, and the LINZ Data Service API reports that layer as CC BY 4.0. The elevation model it was paired with stays undetermined, so the two are now separate entries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`prostate.img` is not a raw greyscale slice: behind its `DICM` magic it is a CT pelvis acquired at Memorial Sloan Kettering on 22 September 1998, with the patient fields holding study codes and the institution and exact date intact. A redistributor's Apache-2.0 cannot license a hospital's patient imaging, so its terms go down as undetermined and the entry now carries the caveat the other human-imaging entries have. `vw_knee` names the Visible Human Project in its own title and description while claiming the author could not be established. It is NLM federal work, which needs no licence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`avg152T1_RL_nifti.nii.gz` and `minimal.nii.gz` are byte-identical to the files the NIfTI-1 project publishes, and the first is the MNI152 template its own entry title already named while the notes said the author was unknown. `vehicle_data.csv` and `cars.csv` are one dataset — UCI's Auto MPG, credited to R. Quinlan under CC BY 4.0 — that had landed in two bulk entries under two different collection licences, neither of which could grant it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The notes said the chain "runs back at least one step further than any page that states terms". The CGTrader page it runs back to is archived and states them: Cartoomotion's model under a General License, which forbids passing the content on "in the form it is downloaded from the site". That is what this repository does with it. The geometry corroborates — 3,672 polygons as 2,592 quads and 1,080 triangles is exactly the 6,264 triangles here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The duplicate-blob check compared only the licence, so nine groups of identical files could and did drift apart on `provenance`: the same bytes recorded as `verified` in the entry that reasoned about them and `inferred` in a bulk entry that swept up the other copy. The check now covers both fields, and the nine groups are resolved by moving each loose copy into the entry that holds the reasoning. `skybox_nz` merges into `skybox_lorensen`, which is the same six faces under a different name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `paraview_pvd_variants` note cited a GitLab call with `all=true`, which returns an empty list for every path including the ones that do exist, so it proved nothing; `ref_name=master` discriminates. `omf_wolfpass` pointed at a `blob/main` URL in a repository whose default branch is `master`, and the Training Images Library page now returns 500, so both move to URLs that resolve. `cad_model_case` said its version came from the page's structured data, which no longer exists there — the note now says what the claim really rests on. CONTRIBUTING and README lose three counts that go stale every review round, gain the dotfile carve-out the validator actually applies, and stop promising a PyVista API that has not merged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The table demanded provenance for every data file while the licence texts beside it had none recorded. Each `[license.*]` table now carries `text_source`, and CI requires it. The twelve SPDX-listed texts are the canonical SPDX text unchanged — checked byte for byte against `spdx.org/licenses/<id>.json`, with the command to repeat that check in CONTRIBUTING. The nine `LicenseRef-` files are not licence texts at all: SPDX does not list those terms, so each states them in its own words and names the page it is based on, which is what `text_source` points at for those. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`mr.001` is a 1994 spine MR whose header holds a real patient name, a full date of birth, a patient identifier embedding that date, sex, age, weight, an accession number, the acquisition time to the millisecond, a device serial number and the name of the hospital radiology department. A Siemens private tag repeats the name, the identifier and the date of birth as burned-in display text, so stripping the standard tags would not de-identify it. It has no `DICM` magic and no file-meta group, which is likely why nobody had parsed it. The identifiers are not reproduced in the entry. `ge_signa` is a cervical-spine MR naming its hospital, with patient fields that look like placeholders. Neither can rest on Kitware's collection notice, on the same reasoning already applied to `prostate`. VTK ships the same `mr.001` bytes, so this one is not only ours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The blunt fin, the combustor and the LOx post are NASA Ames PLOT3D cases, identified from their own grid headers: each decodes to the published dimensions and each file size follows exactly from them. They were split across the two bulk entries under Kitware's BSD-3-Clause and Lorensen's Apache-2.0, neither of which can attach a copyright to work by NASA employees. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`StructuredGrid.vts` had its own entry calling it a "small structured grid". It is byte-identical to `multicomb_0.vts`, so it is one of the NASA combustor blocks and takes that entry's terms. The duplicate-blob check caught this the moment the NASA files moved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
My note said neither upstream records the attribution. Both do: the VTK textbook ships inside `vtk/vtk-examples` and says the data "was prepared at Lawrence Berkeley National Laboratories and is included with their permission on the CD-ROM accompanying this book", with 136 slices per tissue — which is `frog.mhd`'s `DimSize = 500 470 136`. That is permission for one book's CD-ROM, not a redistribution licence, and not Lorensen's to widen into Apache-2.0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The book's figure captions name owners that nine review rounds of readme-hunting never reached. The vase is SUNY Stony Brook's, the iron protein is Scripps Clinic's, and the four files named after companies are not logos at all — they are pre-decimalisation stock quotes the book says were taken from "a public site on the World Wide Web that archives stock prices", which it does not name. A courtesy credit in a textbook names an owner; it is not a licence, and none of these is the redistributor's to widen into a collection licence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It is the only thing enforcing what this pull request claims, and every bug found in it so far — case-folded names, `dir/*` crossing a separator, a `?**` bypass, a crash on a malformed table — was found by accident. Truth tables for `matches` and `is_forbidden` cover where all of those lived, a parametrised case asserts every shape error reaches the report instead of raising, and one test validates the TOML examples in CONTRIBUTING.md, which had drifted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`git ls-files` quotes non-ASCII paths, and the filter dropped any line that did not start with `Data/` — so such a file would be invisible to coverage, to the forbidden-name rule and to the duplicate-blob check, while the summary under-reported the count. Reading `-s -z` from the index fixes that and also makes the duplicate-blob check read the same tree as everything else; it read `HEAD`, so staged work validated locally and failed in CI. An empty `SPDX-License-Identifier` named no licence, so every licence and attribution check was skipped silently. A malformed `[collection.*]` or `[license.*]` value still raised where the neighbouring branch reported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adding `text_source` to the required keys made the copy-paste block in CONTRIBUTING.md fail the validator it documents. Also corrects a filename that does not exist in this repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The validator's output was piped to `tee`, so a traceback went to stderr and the failure summary rendered an empty code block — precisely when the contributor most needs the text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Master added a cow generated from a signed-distance field, with a `LICENSE` and a `README.md` beside it — the arrangement this branch replaces. Both are now a `[[dataset]]` block: CC0-1.0, no attribution required, generated by the `generate_cow.py` sitting in the same directory, whose defaults match the 300000 triangles and 200 mm the README claims. The README's note that the hoof outline was compared against a photograph without any photograph data entering the mesh is worth keeping, so it is in `notes` rather than lost with the file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DATASETS.tomlrecords what every file underData/is, where it came from, who made it, what licence it carries and what changed between the source and the copy here. One central table, so a reviewer sees the whole picture in one diff and PyVista can fetch the lot in one request. The licensing keys borrow their names and meaning from the REUSE specification, andLICENSES/holds each licence's full text named the way REUSE names it;[license.*]and[collection.*]are small lookup tables that the dataset blocks reference.All 1261 tracked files under
Data/are claimed by exactly one of the 220[[dataset]]blocks, andtools/validate_datasets.pyenforces that on every pull request along with the rest of the schema.provenancerecords confidence in the origin only — the licence's status is carried bySPDX-License-Identifier, so a dataset can have a certain source and undetermined terms. What we do not know is visible rather than silently absent.notesLicence roll-up — every licence in the repository, and what it costs a downstream user (21 licences, 220 datasets)
Apache-2.0BSD-2-ClauseBSD-3-ClauseCC-BY-3.0CC-BY-4.0CC-BY-SA-2.5CC-BY-SA-3.0CC-BY-SA-4.0CC0-1.0LicenseRef-DbCL-1.0LicenseRef-NASA-MediaLicenseRef-PublicDomainLicenseRef-SimScale-PublicProjectLicenseRef-StanfordScanningRepositoryLicenseRef-TrainingImagesLibraryLicenseRef-UnknownLicenseRef-UtahTeapot-PublicDomainByConventionLicenseRef-Viewpoint-DataLabsMITODbL-1.0OFL-1.1🔴 72 datasets not cleared for commercial use — 10 whose licence forbids it, 62 whose terms could not be established
The licence forbids commercial use (10)
armadilloLicenseRef-StanfordScanningRepositorybunnyLicenseRef-StanfordScanningRepositorybunny_coarseLicenseRef-StanfordScanningRepositorybunny_exportLicenseRef-StanfordScanningRepositorycowLicenseRef-Viewpoint-DataLabscow_headLicenseRef-Viewpoint-DataLabsdragonLicenseRef-StanfordScanningRepositorylucyLicenseRef-StanfordScanningRepositorysgems_geostatisticsLicenseRef-TrainingImagesLibrarythree_ds_flamingoLicenseRef-Viewpoint-DataLabsTerms could not be established (62) — treat as unlicensed until someone can identify the origin
action_figureantarctica_velocitybanes_heartbibliographyblood_vesselscalculix_frdcarburetorcgnscloudscraterdamavand_volcanodark_matter_halodc_inversiondelaunay_exampledicom_directorydicom_kneedolfin_meshdolfinx_checkpointdrillearth_modelsembryoensight_cylinder_crossflowensight_lshapeensight_simple_tetraexodus_meshfea_bracketfontsfrogge_signagprgreat_white_sharkhead_2hertzian_contact_cylinderhorse_scanhydrogen_orbitaliron_proteinkafadar_lidarlethe_particlesletter_alobsterman_facemeshio_xdmfmisc_pyvista_eramodel_with_variancemount_damavandneghipnetcdfnifti_brainnotch_stressopenfoam_ensight_casepepperprostatepuppypvgeo_examplesroom_surface_meshstock_pricestecplot_asciithermal_probesturbine_bladeurnvasewomanchest_ctBSD-3-Clausecubemap_parkCC-BY-3.0cubemap_spaceLicenseRef-NASA-Mediadicom_stackCC-BY-4.0font_ridiculousOFL-1.1grey_nurse_sharkCC-BY-SA-3.0headsqBSD-3-ClausenefertitiCC-BY-SA-4.0openfoam_cavityBSD-3-Clauseosmnx_street_graphODbL-1.0pluto_textureLicenseRef-NASA-MediareservoirODbL-1.0 AND LicenseRef-DbCL-1.0ruapehu_topo50CC-BY-4.0skybox2CC-BY-3.0stars_cloud_hygCC-BY-SA-2.5step_modelApache-2.0tango_iconsCC-BY-SA-2.5whole_body_ctCC-BY-4.0✅ 130 datasets clear for any use — open licence, commercial use permitted
aero_bracketLicenseRef-SimScale-PublicProjectalpha_channelBSD-3-ClauseamrBSD-3-Clauseangular_sectorBSD-3-Clauseauto_mpgCC-BY-4.0avs_ucdBSD-3-Clausebeach_nrrdBSD-3-ClausebiplaneBSD-3-Clausebiquadratic_quadsApache-2.0bird_bathCC-BY-4.0black_vaseCC-BY-4.0blowBSD-3-Clausebolt_nutBSD-3-ClausebrainApache-2.0cad_model_caseCC-BY-4.0caffeineBSD-3-Clausecake_easyApache-2.0can_crushed_vtuBSD-3-ClausecarotidApache-2.0cellsApache-2.0chi_fieldBSD-3-Clausechombo3dBSD-3-ClauseclownBSD-3-Clausecubemap_debugApache-2.0doormanApache-2.0electronics_coolingLicenseRef-SimScale-PublicProjectemg3d_dipoleApache-2.0emojiApache-2.0ensight_nacaBSD-3-Clauseensight_test_casesBSD-3-Clauseex_blow_5BSD-3-ClausefaceBSD-3-ClausefaultsBSD-3-Clausefilled_contoursBSD-3-Clausefluent_cff_roomBSD-3-Clausefoot_bonesApache-2.0foot_volumeBSD-3-Clausefull_headApache-2.0gambit_prismBSD-3-ClausegearsApache-2.0gettysburg_addressLicenseRef-PublicDomaingis_countriesBSD-3-ClausegourdsApache-2.0grasshopperApache-2.0hdr_texturesCC0-1.0head_mr_volumeBSD-3-ClausehonoluluApache-2.0horseApache-2.0humanApache-2.0idgh_bracketBSD-3-ClauseinfovisBSD-3-Clauseinterpolating_on_stlApache-2.0ivan_angelCC-BY-4.0kitchenApache-2.0letter_kApache-2.0libtiff_testBSD-3-Clauselincoln_life_maskCC0-1.0louis_louvreCC-BY-4.0magnetic_coil_fieldBSD-2-ClausemasonryBSD-3-Clauseminc_imageBSD-3-Clausemoonlanding_imageCC-BY-4.0motorBSD-3-ClausemugApache-2.0multiblock_displayBSD-3-Clausenasa_plot3d_cfdLicenseRef-PublicDomainnatural_earth_iiLicenseRef-PublicDomainnek5000_eddyBSD-3-Clausenifti_samplesLicenseRef-PublicDomainoblique_coneApache-2.0officeBSD-3-Clauseomf_wolfpassMITopenfoam_tubesLicenseRef-SimScale-PublicProjectopenvdb_sphere_pointsBSD-3-ClauseowlCC-BY-4.0parallel_exodus_canBSD-3-Clauseparaview_pvdBSD-3-Clauseparaview_pvd_variantsMITparticlesBSD-3-Clauseperlin_noise_shaderApache-2.0pileated_woodpeckerApache-2.0pine_root_volumeApache-2.0pine_rootsApache-2.0planet3d_matlab_backgroundsCC-BY-4.0plastic_vaseCC-BY-4.0polylineApache-2.0procedural_cowCC0-1.0prostarBSD-3-Clauseprotein_2j6gCC0-1.0protein_2lyzCC0-1.0protein_3gqpCC0-1.0pump_bracketLicenseRef-SimScale-PublicProjectquadratic_cylinderBSD-3-Clausequadratic_pyramidApache-2.0rectilinear_gridBSD-3-Clausesample_gifMITsample_struct_gridBSD-3-Clausesea_vaseCC-BY-4.0semi_diskBSD-3-ClausesextantApache-2.0sharkApache-2.0skybox_lorensenApache-2.0solar_texturesCC-BY-4.0sparse_pointsApache-2.0spcth_time_seriesBSD-3-ClausespiderApache-2.0st_helensLicenseRef-PublicDomainteapotLicenseRef-UtahTeapot-PublicDomainByConventionteapot_vrmlLicenseRef-UtahTeapot-PublicDomainByConventiontensorsBSD-3-ClausetetrahedronApache-2.0torsoApache-2.0toy_plane_point_cloudCC-BY-4.0triquadratic_hexahedronApache-2.0trumpetApache-2.0ucd2dBSD-3-Clauseunstructured_grid_exampleBSD-3-ClauseusaBSD-3-Clausevector_animationApache-2.0victorian_goblet_face_illusionCC-BY-3.0vtk_data_filesBSD-3-Clausevtk_examples_filesApache-2.0vtk_logoBSD-3-Clausevtk_logo_meshApache-2.0vtu_seriesMITvw_kneeLicenseRef-PublicDomainwarping_spheresApache-2.0washington_bustCC0-1.0wavy_pvdMITyinyangApache-2.0Where every claim comes from
Each row above carries the whole chain: the origin that published the work, the route this copy actually travelled, and who it is credited to.
origin_urlis the furthest upstream point that could be established, not a download link — 111 entries name a redistributor there because that is genuinely as far back as anyone got, andredistributed_fromrecords the route when it differs. The licence roll-up links both the text this repository ships and the issuing organisation's own page.LICENSEandREADMEsidecars this branch deletes, then put through nine independent verification rounds and three targeted sweeps — of the two bulk entries, of every human-imaging entry, and of the remaining unknowns. Each round found real errors in what the sidecars asserted.cow.objstates its own terms in its header;dualSphereNoTime.pvdsays<!-- This file is not from paraview. -->on its second line.readmebeside the file upstream, check when a file arrived, hash against upstreams, query URLs through an API, read the CC version fromrel="license", and read the whole pull request discussion. Two origins were found only in issue comments.[license.*]table recordstext_source, CI requires it, and all twelve SPDX-listed texts inLICENSES/are the canonical SPDX text unchanged — verifiable with the snippet in CONTRIBUTING. The nineLicenseRef-files are not licence texts; each states terms SPDX does not list and names the page it is based on.earth.ppm, several photographs, the well tracks,labels.raw,embryo.slc,mni-surface-mesh.obj. Nine Laser Design scans have a known source that publishes no terms at all. These are recorded as undetermined rather than guessed.LICENSEis Apache-2.0 from the 2019 VTK Examples import and does not coverData/;README.mdsays what each root file covers.Resolves #68
Changes drafted by Claude Opus 5 but fully understood by me.
🤖 Generated with Claude Code (Claude Opus 5)