Skip to content

Detector silently keeps checkpoint hyper-parameters when config values update at resume. #136

Description

@rz4

PR #135; src/apeiron/drift_detection/detectors/base.py

On resume, load_state_dict replaces the wrapped river object wholesale, so river-level hyper-parameters silently keep their checkpointed values and ignore any changed config. The wrapper-level thresholds do pick up the new config values. Here's the problem summarized as a test function:

    @pytest.mark.parametrize(
        ("make_detector", "param", "old", "new"),
        [
            pytest.param(ADWINDetector, "delta", 0.002, 0.05, id="adwin"),
            pytest.param(KSWINDetector, "alpha", 0.005, 0.001, id="kswin"),
            pytest.param(
                PageHinkleyDetector, "threshold", 50.0, 10.0, id="page_hinkley"
            ),
        ],
    )
    def test_river_params_come_from_checkpoint_not_config(
        self, make_detector, param, old, new
    ):
        """Restoring replaces the river object wholesale, so
        river-level hyperparameters silently keep their checkpointed values
        even when the detector was rebuilt with new ones, while wrapper-level
        params (the thresholds) take the new config."""
        state = make_detector(**{param: old}, minor_threshold=0.3).state_dict()

        resumed = make_detector(**{param: new}, minor_threshold=0.9)
        resumed.load_state_dict(state)

        assert getattr(resumed.detector, param) == old  # stale checkpoint value
        assert resumed.minor_threshold == 0.9  # new config value
        # wrapper now disagrees with its own river object
        assert getattr(resumed, param) == new # new config value

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions