gaussian-policy and ppo: CleanRL's continuous-action PPO

[tactino]

Stacked on #86 (the GAE episode-boundary fix), whose commit this branch contains: merge #86 first.

The coverage figure's rows pair expressive policies (flow, diffusion, pi0.5) with algorithms built for them. It has no row for the baseline those are measured against: a Gaussian MLP trained by PPO. This adds the pair, written to CleanRL's ppo_continuous_action.py with every default, so that if it fails on MuJoCo the platform failed, not an unfamiliar algorithm.

• gaussian-policy: tanh MLPs of 64×64 for the mean and the value, and a log std that does not depend on the observation. Weights are initialised orthogonally at gain √2, with 0.01 on the mean's last layer and 1.0 on the value's. --policy.deterministic acts with the mean, for evaluation, since the coverage clips act without training's sampling noise; ppo refuses a deterministic policy.
• ppo: rollouts of 2048 steps, 10 epochs of 64-sample minibatches, and a 0.2 clip on both the ratio and the value. Advantages are normalised per minibatch. One Adam (eps 1e-5) covers the actor and critic, the gradient norm is clipped at 0.5, and the learning rate of 3e-4 anneals linearly to zero over 488 iterations (one million steps).

CleanRL clips actions and normalises observations and rewards in gymnasium wrappers. PlugRL's env client wraps nothing, so all of that happens on the server:

• The environment gets the clipped sample. The buffer keeps the unclipped sample whose density the ratio is taken of.
• Observation statistics are policy buffers, updated after each learn the way DPPO updates fpo-policy's. One iteration therefore collects and learns under a single normalisation, and the ratio is exactly 1 before the first update. CleanRL updates its statistics every step; ours lag by one rollout.
• PPOBuffer divides rewards by the running deviation of the discounted return and clips at ±10. The discounted return restarts with each episode, as in NormalizeReward and DPPO's RunningRewardScaler. DPPOBuffer's return carries across episodes because it follows the server's chain of frames. This PR leaves that alone.

tests/test_gaussian_ppo.py (26 tests) covers:

• CleanRL's initialisation
• clipped action versus stored sample
• recomputed log-probability equal to the stored one
• observation normalisation and the ±10 clip
• the return restarting per episode, and reward scaling
• statistics updated only after learning
• the annealing schedule
• one Adam with gradient clip 0.5
• checkpoint resume, including the reward scale and the schedule's position
• the deterministic flag, and PPO refusing it
• CLI registration in a fresh interpreter

It also runs the pair on the bandit that test_fpo_learns_anything gives FPO, on five seeds. The pair goes from about -0.78 to between -0.120 and -0.160, and final over best is 1.00–1.14. FPO's hold test fails some of its ten cases on the same bar. That test uses lr 3e-3: at 3e-4, thirty small iterations are too few for the deviation to narrow.

The full suite passes locally (261 passed, 3 skipped, before #86 was merged in; the PPO tests pass after it too). The MuJoCo run is next: E38, pre-registered, on HalfCheetah, Hopper and Walker2d with the figure's bars.