Spatial morphing by gavinevans · Pull Request #2441 · metoppv/improver

gavinevans · 2026-08-28T11:03:07Z

Related to https://github.com/metoppv/mo-blue-team/issues/1268
improver_test_data PR: metoppv/improver_test_data#146

Description
This PR seeks to address weaknesses with the current use of temporal interpolation as implemented using the ForecastTrajectoryGapFiller plugin. The main weaknesses are:

As Google FILM only knows about the frames at the start and end of a time period, it doesn't have a sense of the general meteorological flow. Therefore, although the morphing from one feature to another is reasonable, the direction of the movement during the temporal interpolation period can be perpendicular to the general direction of the meteorological flow. To alleviate this issue, this PR creates a new plugin to use Google FILM in a different way i.e. as a spatial morphing tool, rather than for strictly doing temporal interpolation. This means that this plugin uses Google FILM to blend together two forecast sources at the same validity time, where the timestep between source A and source B re-interpreted as a weight.
The temporal interpolation can result in an overabundance of low precipitation amounts. This plugin therefore includes the application of Quantile Mapping, which helps to reduce amount of non-zero precipitation introduced as intermediate timesteps.

The SpatialMorphing plugin effectively combines two existing plugins i.e. RealizationSelection and ForecastTrajectoryGapFiller, as this plugin itself takes the clustering cube as an input, so that it can select the relevant realizations to use as input to Google FILM.

Testing:

Ran tests and they passed OK
Added new tests for the new feature(s)

…ility.

…ransition point between forecast sources.

…els.

…ight precipitation.

…ons sources is absent.

mo-jbeaver

I've added some comments below, majority are docstring related. The unit tests passed successfully but the acceptance tests failed with assertion errors

mo-jbeaver · 2026-09-02T13:02:42Z

+            nearest_fp, use_secondary, secondary_map, primary_map, cluster_cube
+        )
+        if self.cluster_number not in full_cluster_to_selection:
+            raise ValueError(


Value Error should possibly be added to the docstring

bayliffe

Tiny comments, nearly there.

bayliffe · 2026-09-16T12:00:24Z

+                for template_slice in template_slices
+            ]
+        elif np.isscalar(self.interpolation_fractions):
+            fractions = [float(self.interpolation_fractions)] * len(template_slices)


I remain confused. Why do we have multiple template slices if we are simply morphing between two input fields at a fixed validity time?

bayliffe · 2026-09-16T12:10:03Z

+                If supplied, cube1 and cube2 may have the same validity time,
+                allowing source morphing at a fixed validity time. A scalar applies
+                the same fraction to every output slice; a sequence provides one
+                fraction per output slice.


I think I'm confused again (along with the query about multiple template_slices previously) do we ever morph to multiple steps between two fields valid at the same time?

I've modified this text. From the SpatialMorphing plugin, we only ever take two inputs and morph to a single intermediate point. We never morph to multiple points. Just to clarify, the GoogleFilmInterpolation plugin could morph to multiple points, if we wanted to support it, but the GoogleFILMInterpolation plugin is never called in that way from the SpatialMorphing plugin. I think that the confusion arises because the GoogleFILMInterpolation plugin could do things in a more general way than we want to support specifically for the SpatialMorphing plugin.

…learly within the GoogleFILMInterpolation changes.

…filtering.

gavinevans · 2026-09-17T07:24:48Z

I've now added these two commits: c8381f8 and e4dbf6d. These are all related to this comment: #2441 (comment). I previously tried to address this by making the the selection of the bounding pair of forecast sources cluster-aware, but after testing, I realised that this only worked for "nc_det uk_det" => "uk_det" and "nc_det uk_det" => "uk_ens" and wasn't sufficient for a cluster defined with "nc_det uk_det" => "uk_det" => "uk_ens", because "nc_det uk_det", "uk_det", "uk_ens" were all present in one cluster, so the selection of the bounding pair of forecast sources also needed to be forecast_period aware.

…sed robustly.

gavinevans · 2026-09-17T10:25:18Z

The latest commit (bd2c3b0), I think relates to changes made in relation to this comment: #2441 (comment), where I removed cube.copy()-type lines. In some command line running, I noticed that I was getting some screwy output following this change, as the source cubes were being updated as part of the Google FILM processing, so when these same source cubes were used for the suppression steps in #2451, the output was incorrect.

bayliffe

Thanks for the changes Gavin. Happy to leave this alone now...

…sion * upstream/master: Spatial morphing (metoppv#2441) Stochastic noise extensions (metoppv#2442) Forecast capping if calibrate forecast is too far from the uncalibrated (metoppv#2449) Vicinity metadata from the threshold plugin (metoppv#2439) Update cluster sources bookkeeping, so that forecast periods are only stored in one forecast source. (metoppv#2437) Correct realization numbering bug in temporal interpolation (metoppv#2436) Modify precedence ordering when clustering (metoppv#2435)

* master: Spatial Morphing - refinement of Google FILM output (#2451) Spatial morphing (#2441) Stochastic noise extensions (#2442) Forecast capping if calibrate forecast is too far from the uncalibrated (#2449) Vicinity metadata from the threshold plugin (#2439) Update cluster sources bookkeeping, so that forecast periods are only stored in one forecast source. (#2437)

gavinevans added 15 commits August 28, 2026 11:44

Re-work Google FILM temporal interpolation into a spatial morphing ut…

8a72126

…ility.

Minor CLI edits.

fef7376

Reemove pdb.

5a75cd6

Correct argument handling.

ebbeb1b

Support smoothstep weights.

0eaea28

Add blending sanitisation.

a453590

Alter spatial morphing interface to more easily define the intended t…

c913b72

…ransition point between forecast sources.

Update to correct handling of transition dictionary.

6f59c5a

Add active pixel refinement to avoid increasing the number of wet pix…

9eb3e18

…els.

Add quantile mapping into the spatial morphing to control spread of l…

4079883

…ight precipitation.

Modifcation to ensure result cube has a name.

333db2c

Refactoring including provide support if one of the expected transiti…

df3855e

…ons sources is absent.

Acceptance test-related edits.

5d5a236

Correct rebasing error.

f255bdd

Update checksums.

c7da7fa

gavinevans mentioned this pull request Aug 28, 2026

Add spatial morphing inputs and kgos metoppv/improver_test_data#146

Merged

gavinevans added 3 commits August 29, 2026 21:21

Provide fallback.

f78286c

Strengthen fallback logic.

a35c1a1

Correct tion to fallback logic and addition of provenance metadata.

26f334e

gavinevans assigned mo-jbeaver Sep 1, 2026

Extend docstrings.

72c4058

mo-jbeaver requested changes Sep 3, 2026

View reviewed changes

mo-jbeaver assigned gavinevans and unassigned mo-jbeaver Sep 3, 2026

Review edits.

5a500a7

gavinevans assigned mo-jbeaver and unassigned gavinevans Sep 3, 2026

gavinevans mentioned this pull request Sep 4, 2026

Spatial morphing - quantile mapping extensions #2448

Closed

2 tasks

gavinevans mentioned this pull request Sep 14, 2026

Spatial Morphing - refinement of Google FILM output #2451

Merged

2 tasks

bayliffe self-requested a review September 14, 2026 09:09

gavinevans assigned bayliffe and unassigned gavinevans Sep 16, 2026

bayliffe reviewed Sep 16, 2026

View reviewed changes

bayliffe assigned gavinevans and unassigned bayliffe Sep 16, 2026

Edits to try to simplify and support the SpatialMorphing usage more c…

ed69c11

…learly within the GoogleFILMInterpolation changes.

gavinevans assigned bayliffe and unassigned gavinevans Sep 16, 2026

Make cluster source identification more accurate.

c8381f8

gavinevans assigned gavinevans and unassigned bayliffe Sep 17, 2026

Add unit test to demonstrate why the forecast period is required for …

e4dbf6d

…filtering.

gavinevans assigned bayliffe and unassigned gavinevans Sep 17, 2026

Add copies of source A and source B, so that these inputs can be re-u…

bd2c3b0

…sed robustly.

bayliffe previously approved these changes Sep 17, 2026

View reviewed changes

bayliffe assigned mo-jbeaver and unassigned bayliffe Sep 17, 2026

mo-jbeaver assigned gavinevans and unassigned mo-jbeaver Sep 17, 2026

gavinevans added 2 commits September 17, 2026 14:40

Merge branch 'master' into mobt_1244_spatial_morphing

2f8b08d

Amendments after merging.

811aa95

gavinevans dismissed bayliffe’s stale review via 811aa95 September 17, 2026 17:50

bayliffe approved these changes Sep 18, 2026

View reviewed changes

gavinevans merged commit e3991ba into metoppv:master Sep 18, 2026
13 checks passed

gavinevans deleted the mobt_1244_spatial_morphing branch September 18, 2026 07:25

Conversation

gavinevans commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mo-jbeaver left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mo-jbeaver Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

gavinevans Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bayliffe left a comment

Choose a reason for hiding this comment

Uh oh!

bayliffe Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

bayliffe Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

gavinevans Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

gavinevans commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gavinevans commented Sep 17, 2026

Uh oh!

bayliffe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gavinevans commented Aug 28, 2026 •

edited

Loading

gavinevans commented Sep 17, 2026 •

edited

Loading