From f995cccf58dafcae64fb7c267d725f05c47c6eb9 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 18:00:16 -0700 Subject: [PATCH 1/7] #343 + audit follow-up: fix Guided Tour numbering and stale Explorer prose figures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #343 (Eric Kansa) — the Guided Tour rendered section headings as "1 Stop 1 —", "2 Stop 2 —" and so on: the site sets number-sections globally, and the headings already carried a hand-written "Stop N". Two fixes, both as requested: - number-sections: false for this page only. The step numbers are written by hand because the numbering is part of the tour's voice, so the automatic numbering is what gets removed, not the manual one. - "Stop" -> "Step" in all seven headings, plus the three prose references, so the page doesn't say "steps" in the titles and "stops" in the body. Coherence audit follow-up — the "How This Demo Works" table asserted file sizes that no longer matched anything published. Corrected against the release manifest, with the cluster counts verified independently against the parquet row counts (38,462 / 112,019 / 176,669 — the manifest doesn't carry row counts). Worth recording WHY they were wrong, because the obvious fix was the wrong one: the old figures (580 KB / 1.6 MB / 2.5 MB) are CORRECT for the 202601 snapshot, which SERIALIZATIONS.md, how-to-use.qmd, data.qmd and tutorials/why_h3.qmd still legitimately document. This page describes the live app, which serves 202608, so it had silently drifted one generation behind. Those four other files were checked against the live 202601 objects (594,541 / 1,627,502 / 2,543,412 B) and are accurate — they are deliberately NOT touched. A "wrong number" in this repo may be a right number for a different release. Also dropped the "4 parquet files" claim, which was false — the published release is 21 objects (18 parquet + 3 JSON search-index files) and the boot manifest check enumerates them. Replaced with the real figure plus a pointer to isamples_202608_release_manifest.json and CANONICAL.md as the authoritative inventory, and a collapsed callout giving the curl/jq and SQL one-liners to re-derive every number on the page without trusting the page. Units are now consistently decimal MB. The old row silently mixed conventions (580 KB was KiB while 1.6/2.5 MB were decimal), which is part of why it read as internally consistent while being a generation stale. Verified: quarto render of both pages clean; tests/test_smoke.py passes; tests/test_frontend_derived.py 40 passed; the #341/#342 count behavior is unaffected (search -> 60/60 dashes, 0 stuck; facet-only -> real counts); no pageerrors. Rendered headings confirmed as "Step 1 ... Step 7" with no duplicate numbering. Refs #343 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- explorer.qmd | 58 ++++++++++++++++++++++++++---- tutorials/explorer_guided_tour.qmd | 27 ++++++++------ 2 files changed, 68 insertions(+), 17 deletions(-) diff --git a/explorer.qmd b/explorer.qmd index e121185..189f263 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -649,7 +649,7 @@ format: ::: {.callout-note collapse="true"} ## How It Works -1. **Instant** (<1s): Pre-aggregated H3 res4 summary (580 KB) → 38K colored circles +1. **Instant** (<1s): Pre-aggregated H3 res4 summary (494 KB) → 38K colored circles 2. **Zoom in**: Automatically switches to res6 (112K) then res8 (176K) clusters 3. **Zoom deeper** (<120 km): Individual sample points from 60 MB lite parquet 4. **Click**: Cluster info or individual sample card with full metadata @@ -7633,13 +7633,57 @@ Pre-aggregated H3 hexagonal indices achieve near-instant globe rendering, with s | Phase | Data | Size | Points | |-------|------|------|--------| -| **Instant** | H3 res4 | 580 KB | 38K clusters (continental) | -| **Zoom in** | H3 res6 | 1.6 MB | 112K clusters (city) | -| **Zoom more** | H3 res8 | 2.5 MB | 176K clusters (neighborhood) | -| **Zoom deep** | Map lite | 60 MB (range req.) | Up to 5K individual samples | -| **Click sample** | Full dataset | ~280 MB (range req.) | Full metadata for 1 sample | +| **Instant** | H3 res4 | 0.5 MB | 38K clusters (continental) | +| **Zoom in** | H3 res6 | 1.3 MB | 112K clusters (city) | +| **Zoom more** | H3 res8 | 2.0 MB | 176K clusters (neighborhood) | +| **Zoom deep** | Map lite | 63 MB (range req.) | Up to 5K individual samples | +| **Click sample** | Full dataset | 300 MB (range req.) | Full metadata for 1 sample | + +*Sizes are decimal MB (10⁶ bytes), matching the manifest's byte counts.* + +**Static files, zero backend.** All queries run in your browser via DuckDB-WASM +with HTTP range requests — only the bytes you need are transferred. The published +release is 21 objects (18 parquet + 3 small JSON search-index files); any one view +touches only a few of them. The authoritative inventory is +[`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) +(human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. + + + +::: {.callout-tip collapse="true"} +## Check these numbers against the source + +Every figure above comes from the published release manifest, which the Explorer +also cross-checks at boot. To re-derive them without taking this page's word for it: + +```bash +curl -s https://data.isamples.org/isamples_202608_release_manifest.json \ + | jq '.files | to_entries[] | select(.key|test("h3_summary|_wide\\.|map_lite")) + | {file: .key, bytes: .value.size_bytes}' +``` + +Cluster counts come from the parquet row counts, not the manifest: -**4 parquet files, zero backend.** All queries run in your browser via DuckDB-WASM with HTTP range requests — only the bytes you need are transferred. +```sql +SELECT count(*) FROM read_parquet( + 'https://data.isamples.org/isamples_202608_h3_summary_res4.parquet'); +``` +::: ## See Also diff --git a/tutorials/explorer_guided_tour.qmd b/tutorials/explorer_guided_tour.qmd index c12482d..f9928da 100644 --- a/tutorials/explorer_guided_tour.qmd +++ b/tutorials/explorer_guided_tour.qmd @@ -12,6 +12,13 @@ format: toc: true toc-depth: 2 theme: cosmo + # #343: the site sets number-sections: true globally, which prefixed these + # headings with an auto-number on top of the "Step N" already in the title — + # so they rendered as "2 Step 2 — ...". The steps are numbered by hand + # because the numbering is part of the tour's voice ("Step 4" is referred to + # as such), so switch OFF the automatic numbering for this page rather than + # stripping the manual numbers. + number-sections: false --- The [Interactive Explorer](https://isamples.org/explorer.html) covers ~6.7 @@ -19,11 +26,11 @@ million physical samples — rock cores, potsherds, tissue vouchers, museum specimens — about 6 million of them on one globe, running entirely in your browser with no server behind it. This tour walks through seven views of it, each answering a question about the data. -Most stops are **live links** — click one and the Explorer opens at the position, -search, and zoom being described (the URL carries that state; a couple of stops +Most steps are **live links** — click one and the Explorer opens at the position, +search, and zoom being described (the URL carries that state; a couple of steps ask you to click one control yourself, where the URL doesn't reach). -A few stops also include a small **"the number behind the view"** code cell that +A few steps also include a small **"the number behind the view"** code cell that computes what you're seeing, live, from the same public data files the Explorer itself reads. Unfold the code to see how — and to take your first step from *viewing* the data to *querying* it. @@ -73,7 +80,7 @@ VOCAB_LABELS = 'https://data.isamples.org/vocab_labels_202608.parquet' H3_RES4 = 'https://data.isamples.org/isamples_202608_h3_summary_res4.parquet' ``` -## Stop 1 — The world at a glance +## Step 1 — The world at a glance **[Open the Explorer →](https://isamples.org/explorer.html)** @@ -116,7 +123,7 @@ with no usable coordinates: they never appear on the globe or in the viewport table, but a world-scope search still finds them. Counting only what's visible would overstate the map and understate the archive. -## Stop 2 — Search that knows concepts +## Step 2 — Search that knows concepts **[Search "pottery Cyprus" →](https://isamples.org/explorer.html?search=pottery+Cyprus)** @@ -135,7 +142,7 @@ Search runs on a pre-built sharded index: your query fetches a few small files, not the whole dataset. The classic full-scan search still exists — add `&fts=off` to the URL to feel the difference. -## Stop 3 — Two sites, two sciences +## Step 3 — Two sites, two sciences **[Fly to central Anatolia →](https://isamples.org/explorer.html#v=1&lat=37.6619&lng=32.8334&alt=150000&mode=point)** @@ -170,7 +177,7 @@ html`

The five densest cells on the globe (click to fly there):

` ``` -## Stop 4 — Slicing by material +## Step 4 — Slicing by material **[Open the Explorer and expand the *Material* facet →](https://isamples.org/explorer.html)** @@ -204,7 +211,7 @@ Rock and mineral dominate (SESAR's size shows through), but biogenic and anthropogenic materials are each hundreds of thousands strong — this is genuinely a cross-domain collection, not a geology database with guests. -## Stop 5 — Density vs identity +## Step 5 — Density vs identity **[Toggle the heatmap →](https://isamples.org/explorer.html)** *(checkbox in the right panel)*, and try the **2D map** (globe button in the map toolbar → @@ -216,7 +223,7 @@ mode answers "which samples, exactly." The honest rule of thumb: heatmaps for patterns, points for identity, and never trust a color's *intensity* as a count — click and read the number instead. -## Stop 6 — From view to data-in-hand +## Step 6 — From view to data-in-hand Any view's sample table (below the globe) shows PID, place, date, and a **Source URL** linking each sample back to its home collection's record. @@ -225,7 +232,7 @@ carries the whole view state, **Copy Link to Current View** gives you a citation-grade pointer to *this exact slice* — paste it in a paper, a class assignment, or an issue report. -## Stop 7 — Under the hood (and doing this yourself) +## Step 7 — Under the hood (and doing this yourself) There is no server. The Explorer is a static page querying public [Parquet](https://parquet.apache.org/) files over HTTP range requests with From 62b3fc8fc307835ae338d3b4658d46df80e6d076 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 18:12:58 -0700 Subject: [PATCH 2/7] #343 follow-up: use decimal MB consistently in the How-It-Works callout too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The collapsed callout still read '494 KB' (KiB-derived) while the table below had been converted to decimal MB — reintroducing in miniature the mixed-units problem that made the original figures look self-consistent while being wrong. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- explorer.qmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/explorer.qmd b/explorer.qmd index 189f263..1493cc3 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -649,7 +649,7 @@ format: ::: {.callout-note collapse="true"} ## How It Works -1. **Instant** (<1s): Pre-aggregated H3 res4 summary (494 KB) → 38K colored circles +1. **Instant** (<1s): Pre-aggregated H3 res4 summary (0.5 MB) → 38K colored circles 2. **Zoom in**: Automatically switches to res6 (112K) then res8 (176K) clusters 3. **Zoom deeper** (<120 km): Individual sample points from 60 MB lite parquet 4. **Click**: Cluster info or individual sample card with full metadata From ea4bef46072f348a49b42d9f605eba2cafeaeb1f Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 18:52:17 -0700 Subject: [PATCH 3/7] #343 round 2: fix Codex's three factual findings + the current-app doc drift it surfaced MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex blocked the first cut on three inaccuracies in the new prose, all valid: 1. "21 objects" undercounted the release. The manifest directly inventories 21 files but separately summarizes the sharded search index (256 base shards plus hot-token sub-files, ~852 objects incl. sidecars per CANONICAL.md). Reworded to say exactly that rather than implying 21 is the whole published release. 2. A binary/decimal straggler survived at the How-It-Works callout: "60 MB lite parquet". 62,924,115 B is 60.0 MiB but 62.9 decimal MB, so under the stated convention it must read 63 MB — matching the table below it. 3. "Every figure above comes from the manifest" was false. The cluster counts come from parquet row counts, and <1s / <120 km / "Up to 5K" are properties of the app, not the data. Narrowed to "every FILE-SIZE figure", with the other sources named. Codex also caught that my HTML comment certified four other docs wholesale as fine, which overstated what I had checked. Corrected in two ways: - EXPLORER_QUERIES.md and index.qmd describe the CURRENT app and were genuinely drifted, so they are fixed here: map_lite_v2 -> _v3 and sample_facets_v3 -> _v4 (both obsolete filenames that would defeat anyone trying to reproduce a query), wide ~280 -> ~300 MB, lite ~50-60 -> ~63 MB, facets ~60 -> ~69 MB. - The comment now states the scope honestly: SERIALIZATIONS.md, how-to-use.qmd, data.qmd and tutorials/why_h3.qmd have CORRECT 202601 bytes and were left alone, but are explicitly NOT certified clean — their present-tense prose still calls those 202601 files the ones serving the Explorer. Rewording them as historical is recorded as open follow-up, not silently implied to be done. Also fixed a bug introduced while editing that comment: the block was closed early with a stray "-->", which would have rendered the units paragraph as visible body text plus a loose arrow. Verified the comment markers balance (5/5) and that none of the comment text appears outside a comment in the rendered HTML, while the intended "Sizes are decimal MB" caption does. Verified: quarto render clean; tests/test_smoke.py passes; test_frontend_derived 40 passed; no stale-unit strings remain outside the explanatory comment. Refs #343 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- EXPLORER_QUERIES.md | 28 ++++++++++++++-------------- explorer.qmd | 36 ++++++++++++++++++++++++------------ index.qmd | 2 +- 3 files changed, 39 insertions(+), 27 deletions(-) diff --git a/EXPLORER_QUERIES.md b/EXPLORER_QUERIES.md index dacd044..db4deeb 100644 --- a/EXPLORER_QUERIES.md +++ b/EXPLORER_QUERIES.md @@ -23,17 +23,17 @@ server. You can open any of these URLs directly, or point DuckDB at them one place, `explorer.qmd` around **line 800-864**, e.g.: ```js -lite_url = `${R2_BASE}/isamples_202608_samples_map_lite_v2.parquet` // map points + table +lite_url = `${R2_BASE}/isamples_202608_samples_map_lite_v3.parquet` // map points + table wide_url = `${R2_BASE}/isamples_202608_wide.parquet` // full sample detail -facets_url = `${R2_BASE}/isamples_202608_sample_facets_v3.parquet` // material/context/object_type + search text +facets_url = `${R2_BASE}/isamples_202608_sample_facets_v4.parquet` // material/context/object_type + search text h3_res4_url = `${R2_BASE}/isamples_202608_h3_summary_res4.parquet` // pre-counted globe dots (world zoom) ``` | File | Plain-English role | Roughly how big | |---|---|---| -| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~280 MB | -| `..._samples_map_lite_v2.parquet` | Slim version with just what the map/table need: coords, label, place, date | ~50-60 MB | -| `..._sample_facets_v3.parquet` | One row per sample: material/context(sampled feature)/object_type as plain URIs, plus a search-text blob | ~60 MB | +| `..._wide.parquet` | Full detail for every sample (one row each) — everything else is derived from this | ~300 MB | +| `..._samples_map_lite_v3.parquet` | Slim version with just what the map/table need: coords, label, place, date | ~63 MB | +| `..._sample_facets_v4.parquet` | One row per sample: material/context(sampled feature)/object_type as plain URIs, plus a search-text blob | ~69 MB | | `..._h3_summary_res{4,6,8}.parquet` | Pre-counted dots for the globe at 3 zoom tiers (continent / region / neighborhood), so zooming out never counts 6M rows live | tiny–few MB | | `..._facet_summaries.parquet`, `..._facet_cross_filter.parquet`, `..._facet_tree_*.parquet` | Pre-computed facet-checkbox counts at various levels of "how many filters are active" — the whole point of these is to avoid a live COUNT over millions of rows | KB–tens of MB | | `..._sample_facet_masks.parquet`, `..._sample_facet_index.parquet` | Bitmask tricks so 2+ facet filters at once are still fast (see `SERIALIZATIONS.md` §4.12 if you want the gory detail) | ~10 MB each | @@ -62,7 +62,7 @@ pre-counted hexagon summaries instead. As you zoom past a threshold, the Explorer swaps to res6, then res8 H3 tiles (same idea, finer hexagons), and eventually to individual points from -`samples_map_lite_v2.parquet` once there are few enough in view to draw +`samples_map_lite_v3.parquet` once there are few enough in view to draw directly. ### ...click a facet checkbox (Material / Sampled Feature / Object Type / Source) @@ -98,15 +98,15 @@ not tens of MB. *(`buildSearchFilterSubstrate()` + `assets/js/search_substrate.j index contract in `SEARCH_INDEX_V1.md`.)* **Fallback path (`?fts=off`, and automatically for identifier queries):** the -original `ILIKE`-style scan against `sample_facets_v3.parquet`'s description +original `ILIKE`-style scan against `sample_facets_v4.parquet`'s description column: ```sql -SELECT pid, label, source, place_name FROM read_parquet('sample_facets_v3.parquet') +SELECT pid, label, source, place_name FROM read_parquet('sample_facets_v4.parquet') WHERE description ILIKE '%pottery%' ``` *(`buildSearchFilter()` in `explorer.qmd`.)* This path downloads much more -data on first search (the scan touches most of the ~60 MB file) but handles +data on first search (the scan touches most of the ~69 MB file) but handles one thing the index cannot: **pasted identifiers** (ARK / IGSN / DOI — e.g. `ark:/28722/k2000hz7r`), which get exact-matched against the `pid` column. Identifier-looking queries are routed here automatically; you never need the @@ -119,20 +119,20 @@ mode. ### ...view the Samples table -The table pages through `samples_map_lite_v2.parquet` (coords/label/place/ -date) and, as of #311, joins in `sample_facets_v3.parquet` for +The table pages through `samples_map_lite_v3.parquet` (coords/label/place/ +date) and, as of #311, joins in `sample_facets_v4.parquet` for material/object type/sampled feature — one query per page (default page size), not the whole result set: ```sql WITH page AS ( SELECT pid, label, source, latitude, longitude, place_name, result_time - FROM read_parquet('samples_map_lite_v2.parquet') + FROM read_parquet('samples_map_lite_v3.parquet') WHERE ORDER BY pid LIMIT 50 OFFSET 0 ) SELECT page.*, f.material, f.context, f.object_type -FROM page LEFT JOIN read_parquet('sample_facets_v3.parquet') AS f ON f.pid = page.pid +FROM page LEFT JOIN read_parquet('sample_facets_v4.parquet') AS f ON f.pid = page.pid ``` *(`loadPage()`, `explorer.qmd` ~line 2755-2775.)* "Download CSV" (#312) runs the same shape without the `LIMIT`/`OFFSET` (capped at 50,000 rows so an @@ -170,7 +170,7 @@ GROUP BY n ORDER BY 2 DESC; -- the default path since 2026-07-17 probes the sharded search index instead -- (JS, not a single SQL statement — see SEARCH_INDEX_V1.md) SELECT pid, label, source -FROM read_parquet('https://data.isamples.org/isamples_202608_sample_facets_v3.parquet') +FROM read_parquet('https://data.isamples.org/isamples_202608_sample_facets_v4.parquet') WHERE description ILIKE '%pottery%' LIMIT 20; ``` diff --git a/explorer.qmd b/explorer.qmd index 1493cc3..82b36e9 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -651,7 +651,7 @@ format: 1. **Instant** (<1s): Pre-aggregated H3 res4 summary (0.5 MB) → 38K colored circles 2. **Zoom in**: Automatically switches to res6 (112K) then res8 (176K) clusters -3. **Zoom deeper** (<120 km): Individual sample points from 60 MB lite parquet +3. **Zoom deeper** (<120 km): Individual sample points from 63 MB lite parquet 4. **Click**: Cluster info or individual sample card with full metadata 5. **Search**: Find samples by name — results fly to the location on the globe @@ -7642,9 +7642,10 @@ Pre-aggregated H3 hexagonal indices achieve near-instant globe rendering, with s *Sizes are decimal MB (10⁶ bytes), matching the manifest's byte counts.* **Static files, zero backend.** All queries run in your browser via DuckDB-WASM -with HTTP range requests — only the bytes you need are transferred. The published -release is 21 objects (18 parquet + 3 small JSON search-index files); any one view -touches only a few of them. The authoritative inventory is +with HTTP range requests — only the bytes you need are transferred. The manifest +directly inventories 21 files and separately summarizes the sharded search index +(256 base shards plus hot-token sub-files, ~852 objects including sidecars); any +one view touches only a handful of them. The authoritative inventory is [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) (human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. @@ -7654,12 +7655,20 @@ touches only a few of them. The authoritative inventory is 38,462 / 112,019 / 176,669 (the manifest does not carry row counts). The previous figures (580 KB / 1.6 MB / 2.5 MB) were not invented — they are - correct for the **202601** snapshot, which SERIALIZATIONS.md, how-to-use.qmd, - data.qmd and tutorials/why_h3.qmd still legitimately document. This page - describes the LIVE app, which serves **202608**, so it had silently drifted a - generation behind. Beware: a "wrong number" here may be a right number for a - different snapshot — always check which release a doc is describing before - correcting it. + correct for the **202601** snapshot (verified live: 594,541 / 1,627,502 / + 2,543,412 B). This page describes the LIVE app, which serves **202608**, so it + had silently drifted a generation behind. Beware: a "wrong number" in this repo + may be a right number for a different release — always check which snapshot a + doc is describing before correcting it. + + Scope of this pass (Codex review): explorer.qmd, EXPLORER_QUERIES.md and + index.qmd describe the CURRENT app and were corrected. SERIALIZATIONS.md, + how-to-use.qmd, data.qmd and tutorials/why_h3.qmd quote 202601 objects whose + BYTES are correct — those were deliberately left alone. They are NOT certified + clean, though: their surrounding present-tense prose still describes those + 202601 files as the ones serving the Interactive Explorer, which is no longer + true. Rewording them as historical/archival is open follow-up work, not done + here. Units: decimal MB (10^6 B). The old row mixed conventions (580 KB was KiB while 1.6/2.5 MB were decimal), which is how it read as self-consistent. @@ -7668,8 +7677,11 @@ touches only a few of them. The authoritative inventory is ::: {.callout-tip collapse="true"} ## Check these numbers against the source -Every figure above comes from the published release manifest, which the Explorer -also cross-checks at boot. To re-derive them without taking this page's word for it: +Every **file-size** figure above comes from the published release manifest, which +the Explorer also cross-checks at boot. (The cluster counts come from the parquet +row counts — see below — and the timings, the 120 km threshold and the 5K point +budget are properties of the app, not the data.) To re-derive the sizes without +taking this page's word for it: ```bash curl -s https://data.isamples.org/isamples_202608_release_manifest.json \ diff --git a/index.qmd b/index.qmd index 82b9680..a4a0e26 100644 --- a/index.qmd +++ b/index.qmd @@ -51,7 +51,7 @@ iSamples integrates data from four major scientific repositories: The project uses **geoparquet files + DuckDB-WASM** for efficient, browser-based data access and analysis — no server required. -- **iSamples Full Dataset**: ~280 MB wide format, 6.7M samples +- **iSamples Full Dataset**: ~300 MB wide format, 6.7M samples - **Available via**: Cloudflare R2 with HTTP range requests - **Interactive tools**: [Interactive Explorer](/explorer.html) — search, filter, and explore 6.7M samples on a 3D globe or in a paginated table From 84625c9c13e490ee36315f5000c5d96aafacaa50 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 19:05:35 -0700 Subject: [PATCH 4/7] #343 round 3: Codex's four stragglers + query-spec drift banner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex round 2 caught four stale strings my grep missed, all in comments/prose (no executable code touched): - EXPLORER_QUERIES.md:154 'the 280 MB wide file' -> 300 MB - explorer.qmd:2682 comment naming sample_facets_v2.parquet -> _v4 - explorer.qmd:6045 and :6624 calling the facets parquet 63 MB -> 69 MB (63 MB is samples_map_lite_v3; the facets file is 69,384,530 B) Also adopted Codex's unambiguous object-count wording: the manifest directly inventories 21 objects INCLUDING five search-index sidecars, and summarizes the index's 847 shard files (256 base + 591 hot); the index directory holds 852 objects. The previous phrasing double-counted the sidecars implicitly. query-spec.qmd: Codex found deeper drift there — stale filenames AND a text-search narrative that still presents the ILIKE full-scan as current with the substrate index as future work (#169), when the sharded BM25 substrate shipped as the DEFAULT on 2026-07-17 and ILIKE survives only as ?fts=off. Correcting that properly means re-stating semantics, not renaming strings, so it now carries an explicit 'known drift' banner naming both problems rather than a silent partial fix. Recorded in explorer.qmd's follow-up inventory too. Verified: quarto render of explorer.qmd and query-spec.qmd clean; comment markers balance 5/5; test_smoke.py passes; test_frontend_derived 40 passed; repo-wide sweep shows remaining hits are only the deliberate CANONICAL.md supersession table, dated plans, and session logs. Refs #343 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- EXPLORER_QUERIES.md | 2 +- explorer.qmd | 19 ++++++++++++------- query-spec.qmd | 23 +++++++++++++++++++++++ 3 files changed, 36 insertions(+), 8 deletions(-) diff --git a/EXPLORER_QUERIES.md b/EXPLORER_QUERIES.md index db4deeb..93e7c2a 100644 --- a/EXPLORER_QUERIES.md +++ b/EXPLORER_QUERIES.md @@ -151,7 +151,7 @@ LEFT JOIN read_parquet('vocab_labels.parquet') mat_lbl ON mat_lbl.uri = mat.pid WHERE s.pid = '' ``` This is the one query that reads from `wide.parquet` on click (everything -above deliberately avoids touching the 280 MB wide file until you actually +above deliberately avoids touching the 300 MB wide file until you actually need full detail on one sample). ## Try it yourself diff --git a/explorer.qmd b/explorer.qmd index 82b36e9..08d3e39 100644 --- a/explorer.qmd +++ b/explorer.qmd @@ -2679,7 +2679,7 @@ phase1 = { // === Load facet summaries + SKOS prefLabels, populate filter checkboxes === // // Checkbox value = full URI (matches the URI strings stored in -// sample_facets_v2.parquet's material / context / object_type columns). +// sample_facets_v4.parquet's material / context / object_type columns). // Display label = SKOS prefLabel (en) when available, URI tail otherwise. // Default state: UNCHECKED — empty = no filter. facetFilters = { @@ -6042,7 +6042,7 @@ zoomWatcher = { // Compute the side-panel relevance score in the SAME scan that builds // the pid-set, and materialize the columns doSearch's results list // needs (label / source / place_name + score). This lets doSearch read - // the small search_pids table instead of RE-scanning the 63 MB facets + // the small search_pids table instead of RE-scanning the 69 MB facets // parquet — the old double-scan that pushed broad terms like "pottery" // past CI's 90s smoke budget. search_pids stays pid-keyed (one row per // unique pid), so every other surface's `pid IN (SELECT pid FROM @@ -6621,7 +6621,7 @@ zoomWatcher = { // The search-term match + relevance score are already materialized // in `search_pids` by buildSearchFilter (aliased `s` below), so the // results SELECT and the follow-up COUNT both read that small table - // instead of re-scanning the 63 MB facets parquet — the old + // instead of re-scanning the 69 MB facets parquet — the old // double-scan (one in buildSearchFilter, one here) that pushed // broad terms like `pottery` past CI's 90s smoke budget. The single // remaining facets scan lives in buildSearchFilter. @@ -7643,9 +7643,10 @@ Pre-aggregated H3 hexagonal indices achieve near-instant globe rendering, with s **Static files, zero backend.** All queries run in your browser via DuckDB-WASM with HTTP range requests — only the bytes you need are transferred. The manifest -directly inventories 21 files and separately summarizes the sharded search index -(256 base shards plus hot-token sub-files, ~852 objects including sidecars); any -one view touches only a handful of them. The authoritative inventory is +directly inventories 21 objects — including five search-index sidecars — and +summarizes the index's 847 shard files (256 base plus 591 hot); the index +directory holds 852 objects in all. Any one view touches only a handful of them. +The authoritative inventory is [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) (human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. @@ -7668,7 +7669,11 @@ one view touches only a handful of them. The authoritative inventory is clean, though: their surrounding present-tense prose still describes those 202601 files as the ones serving the Interactive Explorer, which is no longer true. Rewording them as historical/archival is open follow-up work, not done - here. + here. query-spec.qmd carries deeper drift still (stale filenames AND a + text-search narrative that predates the FTS substrate shipping as default on + 2026-07-17); it now carries its own "known drift" banner rather than a silent + partial correction, because fixing it properly means re-stating semantics, not + renaming strings. Units: decimal MB (10^6 B). The old row mixed conventions (580 KB was KiB while 1.6/2.5 MB were decimal), which is how it read as self-consistent. diff --git a/query-spec.qmd b/query-spec.qmd index 8b61290..548fb84 100644 --- a/query-spec.qmd +++ b/query-spec.qmd @@ -8,6 +8,29 @@ sidebar: false categories: [spec, architecture, query] --- +::: {.callout-important} +## Known drift as of 2026-08-05 — read before relying on the substrate bindings + +This spec has not kept pace with the shipped Explorer in two ways. Both are +recorded rather than silently corrected, because fixing them properly means +re-stating semantics, not renaming strings: + +- **Filenames are a generation stale.** The bindings below name + `sample_facets_v2.parquet`; the Explorer now reads + `isamples_202608_sample_facets_v4.parquet` (and `samples_map_lite_v3`). The + authoritative inventory is + [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json), + human twin `CANONICAL.md`. +- **Text search is described as the old full-scan path.** The `text MATCHES` + binding presents the `ILIKE` scan as what happens "currently" and the + substrate-backed index ([#169](https://github.com/isamplesorg/isamplesorg.github.io/issues/169)) + as future work. In fact the sharded BM25 substrate **shipped as the default on + 2026-07-17**; the `ILIKE` scan survives only as the `?fts=off` escape hatch. + +Treat the *shape* of the bindings as current and the *filenames and text-search +narrative* as historical until this is revised. +::: + ::: {.callout-warning} ## Draft — v0.2 From 6f66252a55301937a93aa5a5b8ba5b2b6b3fddb2 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 19:13:03 -0700 Subject: [PATCH 5/7] #343 round 4: broaden query-spec banner; label DATA_PROVENANCE and EXPLORER_STATE drift MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex round 3 found my query-spec banner understated the problem — it said the drift was 'in two ways' and that 'the shape of the bindings is current', both contradicted elsewhere in the same spec (objectType is described as not exposed when the Explorer ships it as the Specimen Type facet; lite is described as res8-only when lite_v3 carries res4/res6 too). Banner now says the drift is NOT limited to the listed examples, scopes 'current' to the substrate-neutral query model only, and marks anything describing what the Explorer actually does today as a record of the time. Added the two availability examples Codex found. Two more documents labeled rather than silently patched: - DATA_PROVENANCE.md — its version-skew bullets still asserted deployed derived files are 202601 and the live wide is 202604. Added a snapshot note: the live family is now 202608, but the REPRODUCIBILITY gap those bullets describe is still real and unresolved — only the version numbers are historical. - EXPLORER_STATE.md — the #178 two-button addendum describes current search SQL with sample_facets_v2 / samples_map_lite (two generations stale) and the ILIKE scan as the search path. Labeled historical; the scope DECISION still holds. Verified: query-spec renders clean; test_frontend_derived 40 passed. Refs #343 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- DATA_PROVENANCE.md | 9 +++++++++ EXPLORER_STATE.md | 8 ++++++++ query-spec.qmd | 18 ++++++++++++------ 3 files changed, 29 insertions(+), 6 deletions(-) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 063ba43..cba8bdf 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -70,6 +70,15 @@ Eric Kansa maintains OpenContext PQG **independently** on GCS (`storage.googleap ## Documentation / automation gaps (remaining) +> ⚠️ **Snapshot note (2026-08-05).** The version-skew bullets below were written +> when the deployed derived files were `202601` and the wide was `202604`. That is +> no longer the live state: the Explorer now serves the **`202608`** family +> (`sample_facets_v4`, `samples_map_lite_v3`, `wide` at 300,303,095 B). The +> *reproducibility* gap the bullets describe is still real and still unresolved — +> only the version numbers in them are historical. Authoritative current inventory: +> [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json), +> human twin `CANONICAL.md`. + - ⚠️ **The deployed `202601` derived files are NOT reproducible** from any available wide. A rebuild yields **528,983** root-material rows (pre-#271); the deployed `sample_facets_v2` has **346,768** — so the live files came from a different/unrecorded Stage-4 process, *and* the data has since rolled (wide is now `202604`). Treat a fresh `build_frontend_derived.py` run as the new source of truth, not as a bit-for-bit reproduction of the deployed files. - **Version skew:** the deployed derived files are `202601` while the wide they should derive from is `202604` (the popup reads `202604`). Rebuilding from `202604` resolves it (tracked in the pipeline epic). - **No R2 upload automation** — file upload to bucket `isamples-ry` + `current/manifest.json` update are manual `wrangler`/dashboard steps. diff --git a/EXPLORER_STATE.md b/EXPLORER_STATE.md index 3a537d8..c0088f9 100644 --- a/EXPLORER_STATE.md +++ b/EXPLORER_STATE.md @@ -328,6 +328,14 @@ compatible with all of them. ### Light-path addendum: two-button scope selection ([#178](https://github.com/isamplesorg/isamplesorg.github.io/issues/178), 2026-05-08) +> ⚠️ **Historical (2026-05-08).** This addendum records the decision as taken and +> the SQL as it stood then — including the `sample_facets_v2` / `samples_map_lite` +> filenames, which are two generations stale (now `_v4` / `_v3`), and the in-browser +> ILIKE scan, which is no longer the default search (the sharded BM25 substrate +> shipped as default 2026-07-17; ILIKE survives as `?fts=off`). The two-button scope +> *decision* still holds; treat the filenames and search mechanics here as a record +> of the time, not as current reference. + Hana's mockup ([Figma 213:394](https://www.figma.com/design/Nqkuqh3Z4aqVh0nmwUAgKg/iSamples-Wireframe-1.0?node-id=213-394)) proposed a two-button search UI: "Search Selected Areas" (viewport-scoped) and "Search Entire World" (full-corpus). Implemented as a Light extension diff --git a/query-spec.qmd b/query-spec.qmd index 548fb84..a224879 100644 --- a/query-spec.qmd +++ b/query-spec.qmd @@ -11,9 +11,14 @@ categories: [spec, architecture, query] ::: {.callout-important} ## Known drift as of 2026-08-05 — read before relying on the substrate bindings -This spec has not kept pace with the shipped Explorer in two ways. Both are -recorded rather than silently corrected, because fixing them properly means -re-stating semantics, not renaming strings: +This spec has not kept pace with the shipped Explorer. The drift is **not +limited to the examples below** — the Explorer-specific substrate bindings, the +availability notes and the open-questions section all contain additional +historical material. Treat the **substrate-neutral query model** as current +intent, and treat anything describing *what the Explorer actually does today* as +a record of the time until this is revised. + +Known instances (illustrative, not exhaustive): - **Filenames are a generation stale.** The bindings below name `sample_facets_v2.parquet`; the Explorer now reads @@ -26,9 +31,10 @@ re-stating semantics, not renaming strings: substrate-backed index ([#169](https://github.com/isamplesorg/isamplesorg.github.io/issues/169)) as future work. In fact the sharded BM25 substrate **shipped as the default on 2026-07-17**; the `ILIKE` scan survives only as the `?fts=off` escape hatch. - -Treat the *shape* of the bindings as current and the *filenames and text-search -narrative* as historical until this is revised. +- **Availability notes are stale too.** The spec says `objectType` is "not + currently exposed" — the Explorer ships it as the **Specimen Type** facet. It + also describes `lite` as carrying only H3 res8; `samples_map_lite_v3` carries + res4 and res6 as well. ::: ::: {.callout-warning} From c22a6c85636556fa93c42a1d7620ab9f736fb2c9 Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 19:22:45 -0700 Subject: [PATCH 6/7] #343 round 5: snapshot banners on the remaining current-facing 202601 docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codex round 4: several main-navbar pages present 202601 archival artifacts as current Explorer inputs — user-facing misinformation, and this content is about to be archived for the NSF report. Chose LABELING uniformly rather than rewriting the tables. Rationale: the 202601 URLs and byte counts on those pages are CORRECT and still resolve; the defect is purely the present-tense 'this is what the Explorer uses' framing. Rewriting them to 202608 would mean re-verifying a dozen sizes and counts against a different snapshot late in closeout — more error surface than the problem warrants, and it would destroy their value as a stable citable snapshot. Consistent 'Snapshot note (2026-08-05)' banner added to: how-to-use.qmd, data.qmd, tutorials/index.qmd, tutorials/why_h3.qmd, SERIALIZATIONS.md, README.md. Each says the examples pin 202601, that the live app serves the 202608 family, when to prefer each, and points at the release manifest / CANONICAL.md as authoritative. DATA_PROVENANCE.md also gets a document-level coverage caveat: its DAG documents the seven-file derived substrate as of the 2026-06-02 review and does NOT cover the facet index/masks/node-bits set (#304/#305) or the sharded search index (#171), both added later. Incomplete rather than wrong — the build chain and automation gaps it documents remain accurate for the files it covers. Verified: full 55-page quarto render clean; test_smoke.py passes; test_frontend_derived 40 passed; banners confirmed present in rendered HTML for all four qmd pages. Refs #343 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- DATA_PROVENANCE.md | 8 ++++++++ README.md | 8 ++++++++ SERIALIZATIONS.md | 8 ++++++++ data.qmd | 14 ++++++++++++++ how-to-use.qmd | 14 ++++++++++++++ tutorials/index.qmd | 14 ++++++++++++++ tutorials/why_h3.qmd | 14 ++++++++++++++ 7 files changed, 80 insertions(+) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index cba8bdf..384197b 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -3,6 +3,14 @@ How every parquet file the explorer uses is generated, from root to publish. *Reviewed 2026-06-02 (CC, via codebase audit). Complements `SERIALIZATIONS.md` (format/schema reference); this file is the end-to-end build chain + the automation gaps.* +> ⚠️ **Coverage caveat (2026-08-05).** The DAG below documents the **seven-file +> derived substrate as of the 2026-06-02 review**. It does *not* cover the whole +> live `202608` family — the facet index/masks/node-bits set (#304/#305) and the +> sharded search index (#171) were added afterwards and are not represented here. +> The *build chain and the automation gaps* it describes are still accurate for +> the files it does cover; treat it as incomplete rather than wrong. Authoritative +> current inventory: `isamples_202608_release_manifest.json` / `CANONICAL.md`. + > **Load-bearing constraint:** the **root export cannot be regenerated.** It was produced from the iSamples Central Solr API (`central.isample.xyz`), **offline since Aug 2025**. The Zenodo-archived export is a **frozen root**. Any *new* data (e.g. concept URIs, thumbnails) therefore must come from a **per-source supplementary file merged into the base by `pid`** — the "sidecar" pattern (see Stage 3) — not from re-exporting. ## Pipeline DAG diff --git a/README.md b/README.md index ee611b5..8ee678e 100644 --- a/README.md +++ b/README.md @@ -3,6 +3,14 @@ title: isamples.github.io subtitle: README for the isamples.github.io source --- +> ⚠️ **Snapshot note (2026-08-05).** The file examples below pin the **`202601`** +> archival snapshot. Those objects still exist and their byte counts are correct, +> but they are **not what the Interactive Explorer serves today** — the live app +> reads the **`202608`** family (`sample_facets_v4`, `samples_map_lite_v3`, +> `wide`, and the sharded search index). Use `202601` for a stable citable +> snapshot; for what the Explorer actually loads, see +> `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`). + # isamplesorg.github.io This repository provides the source for [SMR fork isamplesorg.github.io](https://smrgeoinfo.github.io/isamplesorg.github.io/). diff --git a/SERIALIZATIONS.md b/SERIALIZATIONS.md index a5e0a74..f4626b4 100644 --- a/SERIALIZATIONS.md +++ b/SERIALIZATIONS.md @@ -7,6 +7,14 @@ toc: true categories: [data, architecture, parquet] --- +> ⚠️ **Snapshot note (2026-08-05).** The file examples below pin the **`202601`** +> archival snapshot. Those objects still exist and their byte counts are correct, +> but they are **not what the Interactive Explorer serves today** — the live app +> reads the **`202608`** family (`sample_facets_v4`, `samples_map_lite_v3`, +> `wide`, and the sharded search index). Use `202601` for a stable citable +> snapshot; for what the Explorer actually loads, see +> `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`). + ## 1. Purpose and scope iSamples has roughly a dozen parquet files in circulation at any given diff --git a/data.qmd b/data.qmd index be0c075..eff593a 100644 --- a/data.qmd +++ b/data.qmd @@ -5,6 +5,20 @@ toc: true categories: [data, parquet, download] --- +::: {.callout-important} +## Snapshot note (2026-08-05) + +The file examples on this page pin the **`202601`** archival snapshot. Those +objects still exist and the byte counts are correct, but they are **not what the +Interactive Explorer serves today** — the live app reads the **`202608`** family +(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). + +Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +actually loading right now", the authoritative inventory is +[`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) +(human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. +::: + ::: {.callout-tip} **Quick start**: every file on this page is queryable directly from a URL — no bulk download needed. DuckDB's `httpfs` extension fetches only the diff --git a/how-to-use.qmd b/how-to-use.qmd index 1a0676c..23cf9f3 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -4,6 +4,20 @@ subtitle: "Get started exploring 6.7 million scientific samples" number-sections: false --- +::: {.callout-important} +## Snapshot note (2026-08-05) + +The file examples on this page pin the **`202601`** archival snapshot. Those +objects still exist and the byte counts are correct, but they are **not what the +Interactive Explorer serves today** — the live app reads the **`202608`** family +(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). + +Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +actually loading right now", the authoritative inventory is +[`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) +(human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. +::: + ## Quick Start {.unnumbered} 1. **Open the [Interactive Explorer](/explorer.html)** — a 3D globe loads with clustered sample data diff --git a/tutorials/index.qmd b/tutorials/index.qmd index d3db435..c672340 100644 --- a/tutorials/index.qmd +++ b/tutorials/index.qmd @@ -4,6 +4,20 @@ subtitle: "Learn to explore 6.7 million physical samples from scientific collect number-sections: false --- +::: {.callout-important} +## Snapshot note (2026-08-05) + +The file examples on this page pin the **`202601`** archival snapshot. Those +objects still exist and the byte counts are correct, but they are **not what the +Interactive Explorer serves today** — the live app reads the **`202608`** family +(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). + +Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +actually loading right now", the authoritative inventory is +[`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) +(human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. +::: + ## Start Here {.unnumbered} | Tutorial | What You'll Learn | diff --git a/tutorials/why_h3.qmd b/tutorials/why_h3.qmd index 70e2327..b77ce9a 100644 --- a/tutorials/why_h3.qmd +++ b/tutorials/why_h3.qmd @@ -9,6 +9,20 @@ format: toc-depth: 3 --- +::: {.callout-important} +## Snapshot note (2026-08-05) + +The file examples on this page pin the **`202601`** archival snapshot. Those +objects still exist and the byte counts are correct, but they are **not what the +Interactive Explorer serves today** — the live app reads the **`202608`** family +(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). + +Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +actually loading right now", the authoritative inventory is +[`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) +(human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. +::: + The progressive globe and the Interactive Explorer both render millions of samples by aggregating points into pre-computed [H3](https://h3geo.org/) cells at three resolutions. This page documents *why* H3, why *those* resolutions, and what we considered before adopting it. ::: {.callout-tip} From d44a64224dcd378601aaa914c5437aa615e4c9ff Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 19:32:46 -0700 Subject: [PATCH 7/7] #343 round 6: Codex's three must-fix items MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1. Banner first sentence narrowed. It claimed 'the file examples on this page pin 202601', which was systematically overstated — these pages mix generations (data.qmd uses the current/ alias and the 202512 narrow file, etc.). Now reads 'The 202601 file examples on this page pin a stable, versioned snapshot', with an explicit note that other generations may appear. Also dropped 'archival' in favour of 'stable, versioned' since the Zenodo deposit is not published yet, and added 'including' before the abbreviated 202608 list. 2. DATA_PROVENANCE coverage caveat was itself incomplete about its incompleteness. Now names every known omission: sample_facet_masks, facet_node_bits, sample_facet_index, sample_facet_index_meta, sample_facet_membership, facet_tree_summaries, facet_tree_cross_filter, and the sharded search index. 3. Three 'every file' completeness claims qualified, since they contradicted the new caveat: DATA_PROVENANCE's own opening line, CANONICAL.md's pointer to it, and the Guided Tour's 'how every derived file is built'. Verified: full 55-page quarto render clean; test_smoke.py passes; test_frontend_derived 40 passed. Refs #343 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- CANONICAL.md | 3 ++- DATA_PROVENANCE.md | 13 ++++++++++--- README.md | 15 ++++++++------- SERIALIZATIONS.md | 15 ++++++++------- data.qmd | 14 ++++++++------ how-to-use.qmd | 14 ++++++++------ tutorials/explorer_guided_tour.qmd | 2 +- tutorials/index.qmd | 14 ++++++++------ tutorials/why_h3.qmd | 14 ++++++++------ 9 files changed, 61 insertions(+), 43 deletions(-) diff --git a/CANONICAL.md b/CANONICAL.md index e03a544..60e4eba 100644 --- a/CANONICAL.md +++ b/CANONICAL.md @@ -36,7 +36,8 @@ Exactly what production `explorer.qmd` loads, all under `https://data.isamples.o The ~9-file facet family looks baroque but is load-bearing: it is the price of fast multi-filter counts with no server. See `EXPLORER_QUERIES.md` for how each -is queried and `DATA_PROVENANCE.md` for how each is built. +is queried and `DATA_PROVENANCE.md` for how the derived files are built (that +doc covers the pre-#290/#304 substrate; see its coverage caveat). ## 2. Superseded versions (still served; do not use) diff --git a/DATA_PROVENANCE.md b/DATA_PROVENANCE.md index 384197b..73ecf4c 100644 --- a/DATA_PROVENANCE.md +++ b/DATA_PROVENANCE.md @@ -1,12 +1,19 @@ # iSamples Explorer — Data Provenance -How every parquet file the explorer uses is generated, from root to publish. +How the explorer's derived parquet files are generated, from root to publish. +**Not exhaustive as of 2026-08-05 — see the coverage caveat below.** *Reviewed 2026-06-02 (CC, via codebase audit). Complements `SERIALIZATIONS.md` (format/schema reference); this file is the end-to-end build chain + the automation gaps.* > ⚠️ **Coverage caveat (2026-08-05).** The DAG below documents the **seven-file > derived substrate as of the 2026-06-02 review**. It does *not* cover the whole -> live `202608` family — the facet index/masks/node-bits set (#304/#305) and the -> sharded search index (#171) were added afterwards and are not represented here. +> live `202608` family. Known omissions: +> +> - `sample_facet_masks`, `facet_node_bits`, `sample_facet_index`, +> `sample_facet_index_meta` (the bitmask count path, #299/#304/#305/#313) +> - `sample_facet_membership` +> - `facet_tree_summaries`, `facet_tree_cross_filter` (the tree facet path, #290) +> - the sharded search index `isamples_202608_search_index_v1/` (#171) +> > The *build chain and the automation gaps* it describes are still accurate for > the files it does cover; treat it as incomplete rather than wrong. Authoritative > current inventory: `isamples_202608_release_manifest.json` / `CANONICAL.md`. diff --git a/README.md b/README.md index 8ee678e..8b472dc 100644 --- a/README.md +++ b/README.md @@ -3,13 +3,14 @@ title: isamples.github.io subtitle: README for the isamples.github.io source --- -> ⚠️ **Snapshot note (2026-08-05).** The file examples below pin the **`202601`** -> archival snapshot. Those objects still exist and their byte counts are correct, -> but they are **not what the Interactive Explorer serves today** — the live app -> reads the **`202608`** family (`sample_facets_v4`, `samples_map_lite_v3`, -> `wide`, and the sharded search index). Use `202601` for a stable citable -> snapshot; for what the Explorer actually loads, see -> `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`). +> ⚠️ **Snapshot note (2026-08-05).** The **`202601`** file examples below pin a +> stable, versioned snapshot. Those objects still exist and their byte counts are +> correct, but they are **not what the Interactive Explorer serves today** — the +> live app reads the **`202608`** family, including `sample_facets_v4`, +> `samples_map_lite_v3`, `wide`, and the sharded search index. (Other generations +> may also appear here, such as the `current/` alias or the `202512` narrow file.) +> Use `202601` for a stable citable snapshot; for what the Explorer actually +> loads, see `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`). # isamplesorg.github.io diff --git a/SERIALIZATIONS.md b/SERIALIZATIONS.md index f4626b4..f2afdc6 100644 --- a/SERIALIZATIONS.md +++ b/SERIALIZATIONS.md @@ -7,13 +7,14 @@ toc: true categories: [data, architecture, parquet] --- -> ⚠️ **Snapshot note (2026-08-05).** The file examples below pin the **`202601`** -> archival snapshot. Those objects still exist and their byte counts are correct, -> but they are **not what the Interactive Explorer serves today** — the live app -> reads the **`202608`** family (`sample_facets_v4`, `samples_map_lite_v3`, -> `wide`, and the sharded search index). Use `202601` for a stable citable -> snapshot; for what the Explorer actually loads, see -> `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`). +> ⚠️ **Snapshot note (2026-08-05).** The **`202601`** file examples below pin a +> stable, versioned snapshot. Those objects still exist and their byte counts are +> correct, but they are **not what the Interactive Explorer serves today** — the +> live app reads the **`202608`** family, including `sample_facets_v4`, +> `samples_map_lite_v3`, `wide`, and the sharded search index. (Other generations +> may also appear here, such as the `current/` alias or the `202512` narrow file.) +> Use `202601` for a stable citable snapshot; for what the Explorer actually +> loads, see `isamples_202608_release_manifest.json` (human twin: `CANONICAL.md`). ## 1. Purpose and scope diff --git a/data.qmd b/data.qmd index eff593a..d238026 100644 --- a/data.qmd +++ b/data.qmd @@ -8,12 +8,14 @@ categories: [data, parquet, download] ::: {.callout-important} ## Snapshot note (2026-08-05) -The file examples on this page pin the **`202601`** archival snapshot. Those -objects still exist and the byte counts are correct, but they are **not what the -Interactive Explorer serves today** — the live app reads the **`202608`** family -(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). - -Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +The **`202601`** file examples on this page pin a stable, versioned snapshot. +Those objects still exist and their byte counts are correct, but they are **not +what the Interactive Explorer serves today** — the live app reads the **`202608`** +family, including `sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the +sharded search index. (This page may also reference other generations, such as +the `current/` alias or the `202512` narrow file.) + +Use `202601` when you want a stable, citable reference. For "what the Explorer is actually loading right now", the authoritative inventory is [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) (human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. diff --git a/how-to-use.qmd b/how-to-use.qmd index 23cf9f3..84859d6 100644 --- a/how-to-use.qmd +++ b/how-to-use.qmd @@ -7,12 +7,14 @@ number-sections: false ::: {.callout-important} ## Snapshot note (2026-08-05) -The file examples on this page pin the **`202601`** archival snapshot. Those -objects still exist and the byte counts are correct, but they are **not what the -Interactive Explorer serves today** — the live app reads the **`202608`** family -(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). - -Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +The **`202601`** file examples on this page pin a stable, versioned snapshot. +Those objects still exist and their byte counts are correct, but they are **not +what the Interactive Explorer serves today** — the live app reads the **`202608`** +family, including `sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the +sharded search index. (This page may also reference other generations, such as +the `current/` alias or the `202512` narrow file.) + +Use `202601` when you want a stable, citable reference. For "what the Explorer is actually loading right now", the authoritative inventory is [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) (human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. diff --git a/tutorials/explorer_guided_tour.qmd b/tutorials/explorer_guided_tour.qmd index f9928da..72a1adf 100644 --- a/tutorials/explorer_guided_tour.qmd +++ b/tutorials/explorer_guided_tour.qmd @@ -242,7 +242,7 @@ further: - **[EXPLORER_QUERIES.md](https://github.com/isamplesorg/isamplesorg.github.io/blob/main/EXPLORER_QUERIES.md)** — plain-English walkthrough of every query the Explorer runs, with copy-paste DuckDB snippets - **[CANONICAL.md](https://github.com/isamplesorg/isamplesorg.github.io/blob/main/CANONICAL.md)** — which data files are canonical (this page uses only those) -- **[DATA_PROVENANCE.md](https://github.com/isamplesorg/isamplesorg.github.io/blob/main/DATA_PROVENANCE.md)** — how every derived file is built from the source export +- **[DATA_PROVENANCE.md](https://github.com/isamplesorg/isamplesorg.github.io/blob/main/DATA_PROVENANCE.md)** — how the derived files are built from the source export (covers the pre-#290/#304 substrate; see its coverage caveat) - The **[iSamples Zenodo community](https://zenodo.org/communities/isamples)** — the archived, citable snapshot of the data substrate diff --git a/tutorials/index.qmd b/tutorials/index.qmd index c672340..fc02fb3 100644 --- a/tutorials/index.qmd +++ b/tutorials/index.qmd @@ -7,12 +7,14 @@ number-sections: false ::: {.callout-important} ## Snapshot note (2026-08-05) -The file examples on this page pin the **`202601`** archival snapshot. Those -objects still exist and the byte counts are correct, but they are **not what the -Interactive Explorer serves today** — the live app reads the **`202608`** family -(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). - -Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +The **`202601`** file examples on this page pin a stable, versioned snapshot. +Those objects still exist and their byte counts are correct, but they are **not +what the Interactive Explorer serves today** — the live app reads the **`202608`** +family, including `sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the +sharded search index. (This page may also reference other generations, such as +the `current/` alias or the `202512` narrow file.) + +Use `202601` when you want a stable, citable reference. For "what the Explorer is actually loading right now", the authoritative inventory is [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) (human twin: `CANONICAL.md`), which the Explorer cross-checks at boot. diff --git a/tutorials/why_h3.qmd b/tutorials/why_h3.qmd index b77ce9a..cb05ab7 100644 --- a/tutorials/why_h3.qmd +++ b/tutorials/why_h3.qmd @@ -12,12 +12,14 @@ format: ::: {.callout-important} ## Snapshot note (2026-08-05) -The file examples on this page pin the **`202601`** archival snapshot. Those -objects still exist and the byte counts are correct, but they are **not what the -Interactive Explorer serves today** — the live app reads the **`202608`** family -(`sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the sharded search index). - -Use `202601` when you want a stable, citable snapshot. For "what the Explorer is +The **`202601`** file examples on this page pin a stable, versioned snapshot. +Those objects still exist and their byte counts are correct, but they are **not +what the Interactive Explorer serves today** — the live app reads the **`202608`** +family, including `sample_facets_v4`, `samples_map_lite_v3`, `wide`, and the +sharded search index. (This page may also reference other generations, such as +the `current/` alias or the `202512` narrow file.) + +Use `202601` when you want a stable, citable reference. For "what the Explorer is actually loading right now", the authoritative inventory is [`isamples_202608_release_manifest.json`](https://data.isamples.org/isamples_202608_release_manifest.json) (human twin: `CANONICAL.md`), which the Explorer cross-checks at boot.