fix: revert preset reward function deletion from hyperparams dict

[mujtaba1747]

RLVR training needs a reward signal. The old code accepted preset_reward_function on the hyperparameters object, then silently deleted it during processing — leaving jobs to fail downstream (or run without a valid reward).

This PR:

1. Preserves the preset_reward_function hyperparameter through _process_hyperparameters so it actually reaches the job.
2. Fails fast, client-side, with a clear message when no reward source is configured, instead of relying on a later server-side failure.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[github-actions]

Claude finished @mujtaba1747's task in 1m 40s —— View job

---

Review: PR #6181 — revert preset reward function deletion from hyperparams dict

• Read PR diff
• Analyze rlvr_trainer.py changes
• Check test coverage & other affected paths
• Post review

The change is small and correct. It stops _process_hyperparameters from stripping preset_reward_function, so a Hub-supplied or user-set preset now actually reaches the training job, and it adds a fail-fast guard in train() requiring at least one reward signal. This is a genuine bug fix (previously a preset was silently deleted before submission).

Observations

✅ Correctness

• The validation is placed before the compute-type dispatch, so it applies to serverless, SMTJ, and HyperPod paths — good.
• getattr(self.hyperparameters, "preset_reward_function", None) safely handles both a missing attribute and self.hyperparameters is None (beta-session fallback). Good.
• The error message is clear and actionable, and the docstring Raises: note is accurate.

⚠️ Backward compatibility (minor, likely intended)

• train() now raises ValueError when neither custom_reward_function nor preset_reward_function is set. This is stricter than before. It's an improvement (previously such a job would submit with no reward signal and fail server-side), but it is a behavioral change — worth a line in release notes if any consumer relied on Hub defaults populating the preset. If the Hub spec provides a default preset_reward_function, getattr returns it (truthy) and validation passes, so that path is unaffected.

⚠️ Test coverage gap (main feedback)

• Only an integ test line was added (test_rlvr_trainer_integration.py:96), which is @pytest.mark.gpu_intensive and won't run in normal CI.
• There's no unit test for the new guard. Note that the existing unit tests in tests/unit/train/test_rlvr_trainer.py set self.hyperparameters to a bare Mock(), so getattr(..., "preset_reward_function", None) returns an auto-created (truthy) Mock, which means the new validation passes only by accident and is effectively untested at the unit level.
• Suggest adding a unit test that asserts train() raises ValueError when neither reward signal is configured — e.g. a hyperparameters object whose preset_reward_function is None/absent and custom_reward_function=None. This also guards the regression from reappearing:

def test_train_raises_when_no_reward_signal(self, ...):
    mock_hyperparams = Mock(spec=[])  # no preset_reward_function attr
    mock_hyperparams.to_dict.return_value = {}
    mock_finetuning_options.return_value = (mock_hyperparams, "model-arn", False)
    trainer = RLVRTrainer(model="test-model", model_package_group="test-group",
                          training_dataset="s3://bucket/train")  # no custom_reward_function
    with pytest.raises(ValueError, match="requires a reward signal"):
        trainer.train(wait=False)

Nit

• test_rlvr_trainer_integration.py:95 introduces a blank line with trailing whitespace; linters may flag it.

Overall this looks good to merge; the only substantive ask is a unit test for the new validation so it's actually exercised in CI.
• master-rlvr-hyperparam-bugfix