E39: with FPO++'s full square fine-tuning, FPO lifts the clone on every seed - short of the bar at 4.8M steps - #95
Merged
Conversation
…er CFM loss, as options
E37 ran FPO++'s policy loss on square from a behaviour-cloned start and the
policy fell from 0.50 to 0.24-0.42 over fifty evaluation episodes, while its
randomly initialised critic never fit the returns. FPO++'s square fine-tuning
(amazon-far/fpo-control, manipulation_experiments/finetune_online_rl.py) has
more than the loss, and this adds the rest, each off by default:
critic_learning_rate, adam_eps, weight_decay, actor_adam_beta2
two AdamW groups (FPO++: actor 1e-5 betas (0.9, 0.99), critic 1e-4,
eps 1e-5, weight decay 1e-6); left at their defaults, FPO's one Adam.
During a critic warmup the actor's group gets no gradient at all, so
weight decay cannot move it.
max_grad_norm
the actor's and the critic's gradients clipped separately (FPO++: 25);
their pre-clip maxima are logged.
normalize_advantage_per_minibatch
as FPO++ does, instead of over the buffer.
cfm_loss_huber_delta
FPO++'s Huber on the flow-matching error, d^2 within delta and
2 delta |d| - delta^2 beyond (FPO++: 1).
FPO's learn step now also reports the critic's explained variance, which
DPPO always has and E37 could not read.
…e-tuning (before the pilot)
… clone on every seed, 1 of 3 past +0.2 at 4.8M steps
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #91 (FPO++'s optimizer, clipping, advantage and Huber options), whose commit this branch contains: merge #91 first.
E39 was pre-registered at 5c6ccd7, after a three-iteration pilot. It runs E37's
fpo-squareagain, this time with FPO++'s square fine-tuning in full rather than only its loss. Same clone, three seeds, 100 iterations of 48,000 steps (4.8M), onguangzhao.The clone itself evaluates at 0.50.
fpo-policy· FPO · square stays "runs" by the rule.The settings were added together, so FINDINGS does not claim which one did the work. The E37 fall was our defect: the platform ran FPO without the rest of the fine-tuning it was published with.