Skip to content

Make sitemap lastmod accurate and stable - #883

Merged
richarddushime merged 3 commits into
mainfrom
worktree-issue-859-sitemap-lastmod
Sep 2, 2026
Merged

Make sitemap lastmod accurate and stable#883
richarddushime merged 3 commits into
mainfrom
worktree-issue-859-sitemap-lastmod

Conversation

@LukasWallrich

@LukasWallrich LukasWallrich commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closes #859

Root causes

  • 740 URLs without lastmod: Hugo takes page dates from git log --name-only, and Git quotes paths with non-ASCII characters by default ("content/glossary/turkish/a\303\247\304\261k_bilim.md"). Hugo's parser never unquotes them, so no glossary term or curated resource with an accented or non-Latin filename got a date. Verified locally: setting core.quotepath=false takes the count from 738 to 1.
  • 1,495 URLs sharing 2026-06-11T06:17:48+08:00: that is the commit that first added the curated-resources seed copy (Update built resources #795). The deployed files come from the daily data artifact, but Hugo still looks their paths up in Git, so every resource carried the seed's commit date.

Changes

  • deploy.yaml and staging-aggregate.yaml: git config core.quotepath false after checkout (staging also gets fetch-depth: 0, matching production).
  • config.toml: [frontmatter] lastmod = ["lastmod", ":git", "date", "publishDate"], so an explicit front-matter lastmod wins over Git.
  • content/glossary/_create_glossaries.py and content/resources/resource.py now write a lastmod key (date only, UTC) and keep the previous value while an entry's content is unchanged. Only a real change to the entry moves the date. Shared logic in scripts/generated_lastmod.py. Bootstrap for entries that predate the key: the file's last commit date (glossary) or the sheet submission timestamp (curated resources).
  • data-processing.yml: full-history checkout, and the curated generator compares against the build-resources branch, so a changed resource does not get re-dated daily until the next seed merge.
  • scripts/check_sitemap_lastmod.py: reports undated URLs, any timestamp stamped on more than half the site, and (with --compare-source) pages whose sitemap date disagrees with their front matter or Git history. Runs warning-only in deploy on a 200-page sample.

Verification

Full local pipeline (both generators, Hugo build with quotepath off, checker on every mapped page):

Check Result
URLs without lastmod 1 of 3,178 (the taxonomy root /publication_types/, which has no dated members)
Pages whose date disagrees with source 0 of 2,889
Generators re-run on unchanged data byte-identical output

Things to be aware of

  • One-cycle transient: the first deploy after merge still downloads the previous data artifact, so curated dates stay on the seed commit until the next daily data-processing run. Glossary dates come from Git until the next manual regenerate_glossary run is merged.
  • Glossary bootstrap puts all ~1,300 terms on 2026-08-05, the commit that added en_title to every file. That is the last verifiable revision of each file, and it persists for terms that do not change afterwards.
  • 297 curated resources have the placeholder submission timestamp 01/01/2020 in the sheet and will show 2020-01-01 as their date until their content changes. The previous value (the June 2026 seed commit) was equally uninformative, but this one is visible on the page.
  • 28 hand-maintained pages (pedagogies, educators corner, summaries) set an explicit lastmod in front matter and now use it instead of their Git date. Their last commits were all housekeeping (link fixes, webp conversion), so the explicit values are the better ones.

🤖 Generated with Claude Code

Hugo reads page dates from `git log --name-only`, and Git quotes non-ASCII
paths unless core.quotepath is off; the quoted paths never matched, so every
glossary term or curated resource with a non-ASCII filename (737 pages) had
no lastmod. The build workflows now set core.quotepath=false.

Generated collections carry their own `lastmod`: the glossary and curated
resource generators keep an entry's previous date while its content is
unchanged and set today's date only when it changes. Existing entries
bootstrap from the file's last commit (glossary) or the sheet submission
timestamp (curated resources). The data-processing workflow compares curated
resources against the build-resources branch so dates do not drift between
seed merges. Hugo now prefers an explicit front-matter lastmod over Git.

scripts/check_sitemap_lastmod.py reports undated URLs, timestamps stamped
on most of the site, and pages whose date disagrees with their source; the
deploy workflow runs it as a warning-only step.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CAq4MsBnm8dNGe1zDqZwgn
@LukasWallrich
LukasWallrich requested a review from a team as a code owner September 2, 2026 16:54
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

👍 All image files/references (if any) are in webp format, in line with our policy.

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

✅ Spell Check Passed

No spelling issues found when checking 8 changed file(s)! 🎉

@LukasWallrich

LukasWallrich commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

Staging Deployment Status

This PR has been successfully deployed to staging as part of an aggregated deployment.

Deployed at: 2026-09-02 22:22:19 UTC
Staging URL: https://staging.forrt.org

The staging site shows the combined state of all compatible open PRs.

@richarddushime richarddushime left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TO BE MONITORED LATER

@richarddushime
richarddushime merged commit 1573414 into main Sep 2, 2026
5 checks passed
@richarddushime
richarddushime deleted the worktree-issue-859-sitemap-lastmod branch September 2, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make sitemap lastmod values accurate and stable

2 participants