fix: GAE ends an episode at the step that ended it, not one later

[tactino]

GAE has been ending every episode one step late since 48042e5 (2026-04-03). FPO, DPPO and anything else built on GAEBuffer have been affected.

The calling convention. Both servers call feedback for step t with:

• terminated / truncated = step t−1's outcome
• next_terminated / next_truncated = step t's outcome
• prev_node = step t−1's node

prev_node is never reset at an episode boundary. So a frame carrying terminated is the first frame of a new episode, and it is chained to the last frame of the episode before.

What went wrong. Before 48042e5, compute_advantages_and_returns cut the recursion at a linked frame with the successor's dones[next_idx]. Under this convention that is step t's own outcome, which is correct. 48042e5 switched to the frame's own dones[step] (and terminated[step] / truncated[step]), which is step t−1's outcome. Since then:

• The step that ended an episode (Hopper falling, a LIBERO success) bootstrapped from the value of the next episode's first state and kept accumulating that episode's advantages.
• Each episode's first step was cut off from everything after it.

The same commit also made a chain end that carries dones (a frame that begins an episode and then meets the end of the buffer) count as terminal, so it dropped its bootstrap value.

Worked by hand: one env, reward 1, values 0, γ = λ = 1, three 3-step episodes.

	advantages
correct	3 2 1 · 3 2 1 · 3 2 1
main	4 3 2 · 1 3 2 · 1 2 1

The fix. Whether a frame's own transition ended its episode is now read from next_* in both branches. For linked frames this is what the code did before 48042e5. For chain ends it drops the extra sub-branch. The masking of a truncated step when treat_truncated_as_done is off is unchanged.

tests/test_gae_episode_boundary.py feeds the buffer exactly as the servers do and checks against advantages worked by hand. It covers:

• terminated episodes under both treat_truncated_as_done settings
• truncated episodes under each setting
• a chain that runs off the buffer, both on an episode's first step and mid-episode

All six fail on main. The full suite passes (242 passed, 3 skipped).

One side effect showed up in the suite. test_fpo_holds_what_it_learns is FPO's known-defect test, marked xfail. It checks that FPO keeps what it learned on the bandit, whose episodes are three steps long. On this machine it held in 4 of its 10 cases before the fix and holds in 7 after. The test stays marked xfail.

What this does to earlier results. Every experiment since April ran with this defect. The coverage figure's green cells learned in spite of it. Their numbers were measured on the defective code and stay as measured. E36 and E37, both running now, are also on it. HalfCheetah never terminates, so it is affected only at its 1000-step truncations. Hopper and Walker2d, which end on a fall, are affected at every fall.

What it does on a real task. This is a diagnostic, not a registered experiment: gaussian-policy under ppo (#85) on Hopper-v5, 100 iterations of 2,048 steps, seeds 0–2. Both arms ran the same code except for this fix. The runs are ab6e32f and 274929c on guangzhao. Mean return:

seed	41–50 before	41–50 after	91–100 before	91–100 after
0	267	674	432	1005
1	589	673	1512	868
2	367	454	910	949
mean	407	600	951	941

With the fix, learning was faster early on and the three seeds were closer together. By iteration 100 the two arms cannot be told apart on three seeds. The defect is established by the tests above, not by this table, and on this task its cost was modest.

One test tightened its claim. test_scaling_the_value_loss_changes_nothing asserted that value-loss coefficients 0.25, 1.0 and 4.0 give bit-identical weights. After the fix changed the toy's returns, one parameter came out a few bits apart on CI, while it stayed identical on Windows and on a Linux workstation (all torch 2.7.1). The factors are powers of two, so the scaled gradients and moments are exact. The difference comes from Adam's eps, which is added unscaled. The test now asserts equality to float32 rounding, and the docstring says why. Any real effect of the coefficient would still fail it by orders of magnitude.