[Fixes #14377] Honor skip_existing_layers flag

[ostyhar]

Fixes #14377.

The upload API exposes skip_existing_layers, while the importer checks skip_existing_layer. This change normalizes the flag before the execution request is created and adjusts the expected layer count when existing layers are skipped.

It also fixes the raster and 3D Tiles skip paths and adds regression tests for full and partial skips.

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

• Confirm you have read the contribution guidelines
• You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
• Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

• There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
• The issue connected to the PR must have Labels and Milestone assigned
• PR for bug fixes and small new features are presented as a single commit
• PR title must be in the form "[Fixes #<issue_number>] Title of the PR"
• New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @ostyhar on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[ostyhar]

The CLA was sent to info@osgeo.org

[mattiagiupponi]

Hi @ostyhar
I dont agree much with the implementation, i would suggest to approach it in a different way:

1. the main issue since the parameters is skip_existing_layer instead of the one in the serializer skip_existing_layers i agree that we should do it, but I think the change should be done on the extract_params_from_data for each handler rather than an IF in the view. Understandable that at least is common for everyone, but goes againts the principle to have each handler responsible for the preparation of the parameters required for the execution
2. Rather than leaving the funciton as it is and move logic in the datastore to evaluate the execution, since we want to skip some dataset, i suggest to move the control should_be_imported into the _select_valid_layers which checks if the layers are valid. Ofc if i need to skip it is not valid, then should be removed from the selected valid layers. We can add a log into the function to evaluate it. In this way when it reaches the import_resource there are not valid layers, so then the execution stops.

[ostyhar]

@mattiagiupponi , got it. I moved the flag handling into extract_params_from_data and did the vector filtering in _select_valid_layers.

Just one thing I'm not sure about: when skip_existing_layers=true and all valid layers already exist, what should happen? After moving the check into _select_valid_layers, the list will be empty, so currently it looks like it would end with No valid layers found.

Should this case finish successfully with 0 imported resources, or should it fail? And should raster and 3D Tiles behave the same way when everything is skipped?

[mattiagiupponi]

Hey @ostyhar!
Since messages only pop up when things fail right now, we can actually make the execution fail on purpose here. We just need to parse the response to show this message to the user:

“The skip_existing_layers was enabled, and no new layers were found in the upload. The process is skipped.”

What do you think?

[giohappy]

I would rather display a more informative message rather the criptic "The skip_existing_layers was enabled". Something like:

No new layers were detected in your upload. Existing layers were left unchanged, so no updates were made.

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @ostyhar on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @ostyhar on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[ostyhar]

@mattiagiupponi , @giohappy , I've updated my branch. Check it please.

[giohappy]

Thanks @ostyhar . I'm not sure we'll be able to review it very soon, but I've started the test runs meanwhile.

[mattiagiupponi]

Overall looks good, just a note on my side @ostyhar

[mattiagiupponi]

The filter_existing should not come from the extracted parameter?

[cla-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have the users @ostyhar on file. In order for us to review and merge your code, please contact the project maintainers to get yourself added.

[ostyhar]

@mattiagiupponi , I've updated the branch. Check it please.