Skip to content

fix(sitemap): guard catalog-only rebuilds from wiping /tutorials/ URLs - #2126

Merged
jung-thomas merged 1 commit into
DEVfrom
fix/sitemap-tutorials-guard
Sep 3, 2026
Merged

fix(sitemap): guard catalog-only rebuilds from wiping /tutorials/ URLs#2126
jung-thomas merged 1 commit into
DEVfrom
fix/sitemap-tutorials-guard

Conversation

@jung-thomas

Copy link
Copy Markdown
Contributor

Problem

The Intelligent Search Data Crawling team reported (2026-09-03) their Developers crawler started failing after Aug 6: crawled docs dropped from ~1,700 to ~1,500 and the live sitemap held only ~180 links.

Root cause: /sitemap.xml is served from the HANA page-sitemap.xml blob (srv/lib/page-key-map.js). publish-content.ts republishes that page-* blob on every non-slug rebuild (discoverPageFiles, gated if (!opts.slug)).

  • slug-targeted → blob not republished (carried forward). Safe.
  • full → "Fetch tutorials" runs, Hugo bakes the full ~1.4k tutorial <urlset>, republishes the good sitemap. Safe.
  • catalog-only → "Fetch tutorials" is skipped, so hugo/content/tutorials is empty, Hugo bakes a sitemap with no /tutorials/ URLs, and republishing that tutorial-less blob wipes the live sitemap to ~180 links.

Same wipe class as the /browse/ (2026-08-07) and /authors/ (#1659 Phase C) catalog-only wipes.

PROD/DEV live fix is a separate mode=full rebuild (already triggered) — regenerates the full sitemap. This PR prevents recurrence.

Changes

  • scripts/seed-sitemap-from-deployed.ts — catalog-only preserve step, mirroring seed-browse/seed-authors. Runs after the Hugo build (the sitemap is a build output, not an input) and overwrites the tutorial-less sitemap with the one the approuter is currently serving. Fails the run if the deployed sitemap is unreadable or names zero tutorials (ALLOW_EMPTY_SITEMAP=1 escape hatch for seed envs).
  • scripts/check-sitemap-tutorials.cjs — fail-closed guard, mirroring check-verb-shelves. Runs in full + catalog-only, before publish; fails if the baked sitemap has zero /tutorials/ <loc> (the exact wipe signature).
  • .github/workflows/rebuild-content.yml — wire both steps in after "Build Hugo site".
  • test/smoke/seo-files.test.js — smoke assertion that /sitemap.xml contains a /tutorials/ <loc>.
  • scripts/__tests__/check-sitemap-tutorials.test.ts — 12 cases covering both tutorial-<loc> counters (shared logic run through both impls so they can't diverge).

Verification

  • vitest run --project unit scripts/__tests__/check-sitemap-tutorials.test.ts → 12/12 pass.
  • Guard CLI exercised directly: exit 1 on 0-tutorial and missing-file sitemaps, exit 0 with tutorials present.

/sitemap.xml is served from the HANA page-sitemap.xml blob and republished
by publish-content.ts on every non-slug rebuild (discoverPageFiles, gated
`if (!opts.slug)`). catalog-only skips "Fetch tutorials", so Hugo bakes a
sitemap with NO /tutorials/ URLs and republishing it WIPES the live sitemap
down to ~180 links (reported 2026-09-03 by the Intelligent Search Data
Crawling team: crawled docs ~1700 -> ~1500). Same wipe class as the
/browse/ and /authors/ catalog-only wipes.

- scripts/seed-sitemap-from-deployed.ts: catalog-only preserve step. Runs
  AFTER the Hugo build (the sitemap is a build output, not an input) and
  overwrites the tutorial-less sitemap with the one the approuter is
  currently serving. Fails the run if the deployed sitemap is unreadable or
  names zero tutorials (ALLOW_EMPTY_SITEMAP=1 escape hatch for seed envs).
- scripts/check-sitemap-tutorials.cjs: fail-closed guard. Runs in full +
  catalog-only before publish; fails if the baked sitemap has zero
  /tutorials/ <loc> (the exact wipe signature).
- rebuild-content.yml: wire both steps in after "Build Hugo site".
- seo-files.test.js: smoke assertion that /sitemap.xml contains /tutorials/.
- unit tests cover both tutorial-<loc> counters (shared logic, run through
  both impls so they can't diverge).
@jung-thomas
jung-thomas merged commit 62e870a into DEV Sep 3, 2026
5 checks passed
@jung-thomas
jung-thomas deleted the fix/sitemap-tutorials-guard branch September 3, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant