fix: stop naming the operator view in robots.txt - #9
Merged
Merged
Conversation
robots.txt is a public file that scanners fetch first, so a Disallow line advertises a path rather than protecting it. Naming the operator view there published the exact thing the line was meant to keep quiet. The canonical link already consolidates the operator view onto the board for any engine that honors it, and discloses nothing to do so. Nothing links to the view, so crawlers have no way to reach it in the first place. Adds a test asserting no public document names it, so this cannot regress. Note this changes what is advertised, not what is reachable: the view is toggled by client-side JavaScript that ships in the page, so anyone reading the source can still find it. Making it genuinely private needs a real mechanism, not a robots directive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The mistake
PR #8 added
Disallow: /*?opstorobots.txt. That's backwards.robots.txtis a public file that scanners fetch first, so aDisallowline advertises a path rather than protecting it — it published the exact thing the line was meant to keep quiet.The fix
Remove it. The canonical link already consolidates the operator view onto the board for any engine that honors canonical, and discloses nothing to do so. Nothing links to the view, so crawlers have no route to it regardless.
Disallow: /api/stays — that endpoint is already documented inllms.txtand serves public data, so there's nothing to disclose.A new test asserts that no public document (
robots.txt,sitemap.xml,llms.txt) contains the string, so this can't regress.What this does not change
The operator view is toggled by client-side JavaScript that ships in the page, so anyone reading the source can still find it. This change stops broadcasting it to every scanner; it does not make it private. Genuinely private needs a real mechanism — a token, or an edge rule — which is a separate decision.
🤖 Generated with Claude Code