Skip to content

[Docs] Pin v1.0-paper as the reproducibility tag in the README - #79

Merged
aaronsmulktis merged 2 commits into
facebookresearch:mainfrom
aaronsmulktis:aaronsmulktis/repro-pin
Sep 14, 2026
Merged

aaronsmulktis merged 2 commits into
facebookresearch:mainfrom
aaronsmulktis:aaronsmulktis/repro-pin

Conversation

@aaronsmulktis

Copy link
Copy Markdown
Contributor

main is about to diverge from the tree the arXiv:2511.20766 results were produced against -- PR #72 removes the per-app appearance config group in favour of a shared theme plus per-app layout, so the paper's variation overrides stop resolving. Point anyone reproducing our numbers at the v1.0-paper tag instead, and say plainly that everything after it is free to move.

`main` is about to diverge from the tree the arXiv:2511.20766 results were
produced against -- PR facebookresearch#72 removes the per-app `appearance` config group in
favour of a shared theme plus per-app layout, so the paper's variation
overrides stop resolving. Point anyone reproducing our numbers at the
`v1.0-paper` tag instead, and say plainly that everything after it is free
to move.
@aaronsmulktis
aaronsmulktis requested review from marksibrahim and a lite review from Copilot September 9, 2026 14:42
@aaronsmulktis aaronsmulktis self-assigned this Sep 9, 2026
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new README text contains time-dependent/inaccurate statements about the current docs/config surface and includes a small typo that should be corrected.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the README to direct paper reproducibility efforts to the v1.0-paper tag, clarifying that main may diverge from the configuration surface used for arXiv:2511.20766.

Changes:

  • Adds a “Reproducing the paper” section with checkout and install steps for v1.0-paper.
  • Explains that post-tag changes may break old variation overrides and that results may differ slightly due to post-publication fixes.
File summaries
File Description
README.md Adds reproducibility guidance pointing to v1.0-paper and describes expected divergence from main.
Review details

Suppressed comments (2)

README.md:101

  • This sentence describes the appearance→theme/layout refactor as already true on main and claims the docs include a migration mapping table, but the repo currently still documents and uses apps/<app>/appearance=... (e.g., site/index.md:40-115) and there is no mapping table. This makes the README section inaccurate/time-dependent; consider switching to future tense and linking to the PR instead of asserting current docs content.
Everything after that tag is free to diverge. The first such change is the appearance refactor: the per-app `appearance` config group is replaced by a shared `apps/theme` (look) plus a per-app `apps/<app>/layout` (structure), so overrides written as `apps/todo/appearance=dark_theme` no longer resolve on `main`. See [App variations](https://facebookresearch.github.io/OpenApps/#app-variations) in the docs for the current axes, which include a table mapping every old `appearance` value onto its replacement.

README.md:103

  • “map tasks” reads like a typo for the Maps app/tasks (consistent with the rest of the repo referring to the app as “maps”).
The tag is the config surface the paper used, not a byte-exact snapshot of the runs: it carries the app, task and harness fixes landed since publication, some of which move rewards (map tasks now match coordinates by ground distance, for example). Expect small differences from the published tables.
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread README.md Outdated
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@aaronsmulktis

Copy link
Copy Markdown
Contributor Author

Whoops, accidentally closed temporarily.

@marksibrahim marksibrahim left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thank you Aaron!

@aaronsmulktis
aaronsmulktis merged commit c7e08cb into facebookresearch:main Sep 14, 2026
2 checks passed
@aaronsmulktis
aaronsmulktis deleted the aaronsmulktis/repro-pin branch September 14, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants