[ABB65]

An opt-in switch in Tools > Contentrain Bridge: "Let Contentrain Migrate read my pages and media". It is off by default and turned on only by an explicit owner click.

What it does

When on, the robots.txt WordPress serves (the robots_txt filter, which runs last) gains one group for the ContentrainMigrate crawler:

• the rules that User-agent: * has, minus only the Disallows that close the whole site (any pattern matching /: /, /*, /$);
• then Allow: /wp-json/, Allow: <the site's REST prefix>, Allow: <the uploads path> and Allow: /.

Why that keeps wp-admin closed: by RFC 9309 longest match (an Allow wins a tie), those Allows open the REST API and the media under any shorter Disallow (/wp-, /wp-content/). Every other path such a Disallow closes stays closed, wp-admin included.

What else stays as it was:

• Other crawlers see robots.txt exactly as before: the original output stays first, byte for byte.
• The owner's own group wins: if robots.txt already names ContentrainMigrate, nothing is added (/about reports owner_group: true).
• Turning it off removes the group. Uninstall deletes the option.

When WordPress can't change robots.txt:

• A robots.txt file on disk (in WordPress's directory or in the site root via get_home_path()): the switch stays off. The card shows the exact lines to add by hand, built from that file, and /about reports file: 'physical'.
• A CDN-served robots.txt (e.g. Cloudflare managed): the card's "served by Cloudflare or another CDN?" section shows the lines to add there. The readme FAQ says the same.

/about gets capabilities: ['robots_allow'] and robots: { allow, file: 'virtual' | 'physical', owner_group }. t8's S3 adds 'counts'; whoever merges second keeps both.

The form is a plain admin-post form (no JS needed), behind Admin::permitted() (export + manage_options) and its own nonce.

readme.txt (WP.org)

• External services: a "robots.txt (only when you turn it on)" paragraph saying exactly what is added and removed, including the longest-match note. It contacts no host.
• FAQ: why to turn it on, plus the CDN line.
• Description: "not modified" names this one opt-in exception.

No version bump: 0.7.0 follows in a separate PR together with S3.

Tests

tests/robots.php, 41 checks (npm run test:robots; also in tests/coverage.sh), each judged by an RFC 9309 verdict (merged groups, longest match, Allow on a tie):

• off by default; /about reports off, virtual, no owner group;
• on: wp-admin is closed and admin-ajax open; the * group is unchanged; the original bytes come first;
• ork2's fixtures:
  • the fixtures: Disallow: /, /*, /wp-, /wp-content/, uploads + plugins closed with a Googlebot group, /wp-json/ + /private/, and Cloudflare's managed robots.txt (migrate's fixture);
  • for each one, Migrate reads /, a post, /wp-json/…/about, /wp-json/wp/v2/posts and an upload, and wp-admin stays closed;
  • *, Googlebot, GPTBot and ClaudeBot have the same verdict on every path as with the switch off.
• the owner's closed paths outside the reads stay closed; only a whole-site closure is left out; /wp- is kept;
• the owner's own group is left as written, with owner_group set;
• a file on disk: physical and off, and the lines to add by hand open the reads;
• off again: the group is removed and the option deleted;
• the handler refuses a visitor without permission.

tests/robots-parity.mjs (npm run test:robots-parity -- <migrate packages/crawler/dist/index.mjs>) reads the same off/on pairs with Contentrain Migrate's own parser (@migrate/crawler, RFC 9309 group merge since migrate#679). The same three claims hold: Migrate reads, wp-admin is closed, and other agents' verdicts are unchanged.

tests/robots.sh takes BRIDGE_TEST_PROJECT (-p) for parallel sessions.

Local gate (feat/robots-allow @ 5319eb8, on origin/main 88b5e8e)

Check	Result
tests/robots.sh (WP 7.1, php8.3)	41/41
tests/robots-parity.mjs through migrate main f3cf2a82's crawler	7/7
Plugin Check 2.1.0	0 errors, 26 warnings, the same 26 as 0.6.5 (none new)

npm test is 12/13. The failing handoff.test.mjs needs tests/.out/store from the full acceptance run; that's pre-existing, and the full acceptance runs in the slot after t8.

[ABB65]

Read-review (ork2), head 2d4d0bb: CHANGES REQUESTED. One MUST, plus ORK's media item.

MUST: dropping a Disallow can open more than / and /wp-json/. group() drops every * Disallow that matches / or /wp-json/. A prefix rule also covers other paths:

• Disallow: /wp- closes /wp-json/, so it is dropped, and that also opens /wp-admin/. The same happens with /wp.
• Owner * rules Disallow: /wp- + Disallow: /wp-admin/: /wp-admin/ stays closed through the explicit rule. With only /wp-, /wp-admin/ opens to us, which goes against "everything else as the owner set it".

Suggested shape (RFC 9309 longest match; on a tie, Allow wins):

1. Drop only the Disallows that close the whole site, i.e. the ones that match path / (/, /*, /$).
2. Keep every other owner rule as is, and add targeted Allows: Allow: /wp-json/, Allow: /wp-content/uploads/ (ORK's media item), then Allow: /.
   • An Allow is longer than a shorter prefix Disallow (/wp-, /wp-content/), so it wins for those paths.
   • On a tie with an exact Disallow: /wp-json/, Allow wins.
   • /wp-admin/ and other private paths stay closed through the owner's own rules.
3. A longer, specific owner Disallow under /wp-json/ (for example /wp-json/wp/v2/users) stays: the owner's choice.

Parity test (ork1's suggestion): run the robots.txt Bridge produces through our #679 parser (@migrate/crawler) and assert the verdicts:

• ContentrainMigrate is allowed on /, /wp-json/ and /wp-content/uploads/x.jpg;
• /wp-admin/ stays closed;
• other UAs are byte-identical.

The fixtures need to cover /wp-, /wp-content/, /*, Disallow: /, and a Cloudflare-managed block.

Notes (non-blocking):

• physical() looks at ABSPATH . 'robots.txt'. With WordPress in a subdirectory (home at root, core in /wp/), robots.txt sits at the web root, not ABSPATH, so detection misses it. Look at get_home_path() as well.
• If the owner has their own ContentrainMigrate group, with_group() changes nothing, while /about still reports allow: true. Expose that case in /about (for example owner_group: true) so web doesn't promise access the toggle doesn't give.
• With a Cloudflare-managed robots.txt, /about says virtual but the served file differs. Web's "Check again" will catch that; mention it in the readme FAQ.

[ABB65]

Delta read-review (ork2) @ 5319eb8: the MUST is resolved, read-review OK.

group() now drops only the Disallows that match / (/, /*, /$) and keeps every other owner rule as written. It then appends targeted Allows: /wp-json/, the REST prefix, the uploads path, and /. Under longest match with Allow winning ties, /wp- and /wp-content/ keep wp-admin and plugins closed, while REST and uploads open. That also covers ORK's media item.

Tests:

• 41 checks with my fixtures (/, /*, /wp-, /wp-content/, Googlebot, /private/, Cloudflare-managed);
• other UAs return the same verdicts as with the switch off;
• parity through @migrate/crawler (#679): 7/7.

The non-blocking items are done too: get_home_path(), owner_group, and the Cloudflare FAQ line.

Remaining for the ONAY: t3's full Docker acceptance, which runs in the lane B slot after t8's S3.

[ABB65]

Gate on head 5319eb8, run from down -v in compose project t3-robots with KEEP=1.

• npm ci: rc 0
• test:wordpress: 394 checks passed
• Plugin Check: 0 errors, 26 warnings. These are the same 26 that main shows (jobs.php direct queries, source.php:386, 4 WPML hooks); the robots code adds none.
• test:store: valid, 14 models / 62 entries; byte parity on 55 files
• npm test: 13/13
• test:robots: 41 checks
• robots-parity: 7/7 through @migrate/crawler
• uninstall.php: rc 0
• package: rc 0
• Stack taken down afterwards.

[ABB65]

On hold: the CI acceptance job is red on 5319eb8 (robots.php, wp-admin check). Looking into the environment difference; the local gate above does not stand until that is fixed.

[ABB65]

ork2 delta read @c331669b: OK. The group ALWAYS adds Disallow: <admin_url path> and Allow: <admin-ajax path>, and under longest match wp-admin stays closed against Allow: /. It's subdir-safe. The fixtures cover a Yoast-style rewritten robots.txt, one with no * group, and blog_public=0. The ONAY waits for PR CI acceptance.

[ABB65]

Hold lifted. Head 97a468e, all CI green: acceptance, PHP 7.4/8.3 syntax, metadata, perf small/large.

The CI red came from Yoast. It stays active during CI's coverage step and rewrites robots.txt.

Fixes:

• Product: the Migrate group now always closes wp-admin. It adds Disallow on admin_url's path plus Allow on admin-ajax.php, whatever the * group says.
• Tests:
  • a Yoast-style replaced output;
  • output with no * group;
  • output that ends without a newline;
  • blog_public 0;
  • every fixture now asserts that the Migrate group stands as a group of its own.

Local run, own compose project, in CI's order: robots 61/61 with Yoast active and 61/61 without it; parity 10/10 through @migrate/crawler; Plugin Check 0 errors, the same 26 warnings as main.