[flyingdoubleG]

What

For adv_estimator: gigpo, compute_response_level_rewards normalizes episode rewards with the user's reward_normalization, then builds the step-level normalization from the deprecated method only:

reward_normalization=RewardNormalizationConfig(grouping="state_group_id", method=pipeline_config.reward_normalization.method)

method defaults to "identity", and RewardNormalizationConfig asks users to set norm_mean_type / norm_std_type instead. With a config that does that, e.g.

reward_normalization:
  grouping: traj_group_id
  norm_mean_type: group
  norm_std_type: group

episode rewards are normalized per group, but the step config resolves method="identity" to norm_mean_type=None, norm_std_type=None, so step rewards are not normalized at all before being mixed with step_reward_weight. GiGPO normalizes both levels the same way (in the reference implementation episode_norm_reward and step_norm_reward share remove_std). The bundled gigpo examples set method, so they are not affected.

Change

A small helper, step_reward_normalization, builds the step config from the resolved norm_mean_type / norm_std_type (each one mapped to "group", i.e. the anchor-state group, or kept None) and passes method along. This reproduces what method gives today for every method value and grouping (checked in the tests), and fixes configs that only use the new fields.

Tests

tests/utils/test_agentic_reward_norm.py: the step config for the deprecated method values is unchanged, the new-field configs get group normalization, and agentic_reward_norm with the step config normalizes within each state group (CPU).

🤖 Generated with Claude Code