[Fixes #14651] Management command for metadata validation by etj · Pull Request #14653 · GeoNode/geonode

etj · 2026-09-24T16:05:12Z

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

Confirm you have read the contribution guidelines
You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
The issue connected to the PR must have Labels and Milestone assigned
PR for bug fixes and small new features are presented as a single commit
PR title must be in the form "[Fixes #<issue_number>] Title of the PR"
New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

(cherry picked from commit d27537e)

(cherry picked from commit 1d1f43f)

Bumps [django-allauth](https://github.com/sponsors/pennersr) from 65.16.1 to 65.17.0. - [Commits](https://github.com/sponsors/pennersr/commits) --- updated-dependencies: - dependency-name: django-allauth dependency-version: 65.17.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> (cherry picked from commit 92a21b0)

Bumps [requests](https://github.com/psf/requests) from 2.33.1 to 2.34.2. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.33.1...v2.34.2) --- updated-dependencies: - dependency-name: requests dependency-version: 2.34.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> (cherry picked from commit 440d2ae)

(cherry picked from commit e2e365c)

…sponse (#14265) (#14272) * Do not return WWW-Authenticate inside API exception response * Test WWW-Authenticate stripped * Fix test docstring (cherry picked from commit 3c9b610) Co-authored-by: Giovanni Allegri <giohappy@gmail.com>

…ad (#14263) (#14273) (cherry picked from commit e9553e8) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com>

…ly (#14187) * Add autoload subcommand to thesaurus management command, task in tasks.py, and entrypoint call * Add documentation for thesaurus autoload command and boot-time thesauri loading * Move autoload_thesauri into its own thesaurus_subcommands/autoload.py * Log improvements, lang selection * Apply suggestions from code review * Fix mutable default argument for langs parameter in load_thesaurus * Fix patch apps.get_app_configs in correct autoload module * Fix code formatting * Improve load final log Co-authored-by: etj <717359+etj@users.noreply.github.com> Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it> (cherry picked from commit 016fdd9)

(cherry picked from commit 08bd4c7)

* Add parameter for beat DB file location * Cleaner location for BEAT_DB (review comment from Gpetrak) (cherry picked from commit d295a18) Co-authored-by: Wolfgang Kaltz <jwkaltz70@gmail.com>

Bumps [django](https://github.com/django/django) from 5.2.14 to 5.2.15. - [Commits](django/django@5.2.14...5.2.15) --- updated-dependencies: - dependency-name: django dependency-version: 5.2.15 dependency-type: direct:production update-type: version-update:semver-patch ... (cherry picked from commit c9ef9d1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…14312) * Management command to clear gwc cache * GWC Module to handle gwc related stuffs * Updated gwc truncate command to use gwc module * Improve GWC client and truncate command error handling * Fix masstruncate URL * Improvements and fixes * Improvements --------- (cherry picked from commit b4a695d) Co-authored-by: Niraj Adhikari <41701707+nrjadkry@users.noreply.github.com> Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

…and of gwc (#14313) (cherry picked from commit 820b5a8) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

(cherry picked from commit 929ad55) Co-authored-by: Wolfgang Kaltz <jwkaltz70@gmail.com>

…) (#14317) (cherry picked from commit 77f4f51) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

(cherry picked from commit 2c0fdfd) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

…#14323) (#14324) * [Fixes #14321] copy_with_dump is missing from copy_table_with_ogr2ogr --------- (cherry picked from commit a0f50e5) Co-authored-by: mattiagiupponi <51856725+mattiagiupponi@users.noreply.github.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

(cherry picked from commit 0803857)

(cherry picked from commit 6b1d8db) Co-authored-by: Mattia Giupponi <mattia.giupponi@gmail.com>

Bumps [cryptography](https://github.com/pyca/cryptography) from 48.0.0 to 49.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@48.0.0...49.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-version: 49.0.0 dependency-type: direct:production update-type: version-update:semver-major ... (cherry picked from commit 2b7c0ff) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Upgrade of setupttols to 82 with fixes to dependencies * Sanitize metadata input * build(deps): bump cryptography from 48.0.0 to 49.0.0 Bumps [cryptography](https://github.com/pyca/cryptography) from 48.0.0 to 49.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@48.0.0...49.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-version: 49.0.0 dependency-type: direct:production update-type: version-update:semver-major ... * [Fixes #14339] Drop support Dropbox storage manager --------- (cherry picked from commit 3d929fc) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Giovanni Allegri <giohappy@gmail.com> Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mattia Giupponi <mattia.giupponi@gmail.com>

(cherry picked from commit 109bc4b) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

* Squashed PR #14001 (cherry picked from commit a9ade43) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

(cherry picked from commit 0e09ec4) Co-authored-by: Giovanni Allegri <giohappy@gmail.com>

…14357)

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved correctness, performance, error-handling, and test-coverage issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 · 2 · 1

Open (4)

Use polymorphic queryset to preserve resource-specific metadata · New
Stream reports to prevent unbounded memory use · New
Handle schema errors during validation · New
Add tests for metadata validation command behavior · New

What changed in this PR

Adds a Django management command for validating GeoNode resource metadata against JSON Schema and generating reports.

Changes:

Supports resource filtering, language selection, and validation options.
Generates JSON or CSV validation reports.
Reports schema and instance validation errors.

File	Summary
`geonode/metadata/management/commands/validate_metadata.py`	Implements metadata validation, filtering, error formatting, and reporting.
`geonode/metadata/management/commands/__init__.py`	Reviewed; no functional changes noted.
`geonode/metadata/management/__init__.py`	Reviewed; no functional changes noted.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+        return json.loads(json.dumps(data, cls=JSONEncoder))
+
+    def select_resources(self, ids, resource_types):
+        queryset = ResourceBase.objects.all()


+            if not record["valid"]:
+                invalid_count += 1
+            if not record["valid"] or options["include_valid"]:
+                records.append(record)


+        errors = self.drop_redundant(validator.iter_errors(instance))
+        errors = sorted(errors, key=lambda err: ([str(p) for p in err.absolute_path], str(err.validator)))
+        reported = errors[:max_errors] if max_errors > 0 else errors


+
+        parser.add_argument("--debug", dest="debug", action="store_true", help="Set log level to debug")
+
+    def handle(self, *args, **options):


…int coordinates (#14655) * [Fixes #14639] Add test case (cherry picked from commit 1b21fe1) * [Fixes #14639] Add lon to accepted CSV fields for point coordinates (cherry picked from commit 638026a) --------- Co-authored-by: sijandh35 <sumoseason35@gmail.com>

* Improvements in Asset * Move logic into permissions registry --------- (cherry picked from commit c63fa4f) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com> Co-authored-by: mattiagiupponi <mattia.giupponi@gmail.com>

…r local thumbnails fails

(cherry picked from commit de34307) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com>

(cherry picked from commit 83ea1c0)

…rom CSV (#14642) (#14667) (cherry picked from commit cac1717) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com>

cla-bot · 2026-10-01T14:19:19Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Fabian Fischer.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

gitguardian · 2026-10-01T14:20:47Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
37692361	Triggered	Username Password	`dbc4e59`	geonode/base/api/tests.py	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

gitguardian · 2026-10-01T14:20:48Z

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
37692361	Triggered	Username Password	`dbc4e59`	geonode/base/api/tests.py	View secret

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secret safely. Learn here the best practices.
Revoke and rotate this secret.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

Copilot

Copilot review overview

🔵 Needs a closer look

Unresolved critical security, data-loss, and correctness findings remain in this broad change set.

Review effort: Lite
Findings: 5 · 6 · 3

Open (14)

Preserve oversized metadata before destructive deletion · New
Merge duplicate GroupForm clean methods · New
Recursively validate nested serializer values · New
Use safe path containment checks during extraction · New
Use polymorphic queryset to preserve resource-specific metadata
Combine metadata filters conjunctively · New
Handle layers with no identifiable CRS · New
Handle null GDAL layers before accessing methods · New
Use validated JSON candidate for base asset · New
Handle schema errors during validation
Stream reports to prevent unbounded memory use
Fix misspelling in security-policy link · New
Use cloned directory consistently in installation instructions · New
Add tests for metadata validation command behavior

+        if len(value) > 1024:
+            logger.warning(
+                f"ExtraMetadata pk={extra_meta.pk} for resource {extra_meta.resource.id}:{extra_meta.resource.title} "
+                f"skipped during migration to SparseField: "
+                f"serialized value exceeds 1024 characters"
+            )
+            continue


+    def clean(self):
+        cleaned_data = self.cleaned_data
+        for field_name, value in list(cleaned_data.items()):
+            if contains_disallowed_template_tokens(value):
+                raise forms.ValidationError(
+                    _("Field %(field_name)s contains characters that are not allowed."),
+                    params={"field_name": field_name},
+                )
+        return cleaned_data


+def contains_disallowed_template_tokens(value):
+    values = value if isinstance(value, (list, tuple)) else [value]
+    return any(isinstance(item, str) and any(token in item for token in DISALLOWED_TEMPLATE_TOKENS) for item in values)


+                    if not os.path.realpath(os.path.join(extract_dir, member)).startswith(
+                        os.path.realpath(extract_dir)
+                    ):
+                        raise ImportException(f"Invalid zip entry: {member}")


+                # Compare values in a type-agnostic way.
+                if str(stored_value.get(field_name, None)) == str(filter_value):
+                    batch_pks.add(sf.resource.pk)
+
+            filtered_pks.update(batch_pks)


+                logger.error(
+                    f"The following layer {layer.GetName()} does not have a Coordinate Reference System (CRS) and will be skipped."
+                )
        return f"{_name}:{_code}"


            raise InvalidGeopackageException("The geopackage provided is invalid")

-        layers_count = len(layers)
+        layers_count = layers[0].GetLayerCount()


+        if js_file:
+            files = {"base_file": files.get("base_file")}


+1. The reported vulnerability has been verified by working with the GeoNode PSC
+2. GitHub [security advisory](https://github.com/geonode/geonode/security) is used to reserve a CVE number by the GeoNode Organization
+3. A fix or documentation clarification is accepted and backported to active branches
+4. A fix is included for the active branches release downloads ([reelases](https://github.com/GeoNode/geonode/releases), or issued via emergency update)


+Go inside the `geonode-project` folder and create the .env file by using the `create-envfile` script:

 ```bash
 cd my_geonode


cla-bot · 2026-10-01T14:39:34Z

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Fabian Fischer.
This is most likely caused by a git client misconfiguration; please make sure to:

check if your git client is configured with an email to sign commits git config --list | grep email
If not, set it up using git config --global user.email email@example.com
Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

etj · 2026-10-01T14:44:00Z

That branch was supposed to be merged on 5.1.x.

New master PR: [Fixes #14651] Management command for metadata validation #14680
51.x PR is at [Fixes #14651][5.1.x] Management command for metadata validation #14679

giohappy and others added 30 commits May 15, 2026 10:27

Upgrade to NGINX 1.31.0 (#14226)

163abf3

(cherry picked from commit d27537e)

extende docs validation

02a2544

(cherry picked from commit 1d1f43f)

Add read DEFAULT_MAX_UPLOAD_SIZE from environment

bd69c1d

(cherry picked from commit e2e365c)

backport pillow upgrade (#14269)

32a8636

[Fixes #14262] Avoid returning all permissions in dataset owner paylo…

c9c657b

…ad (#14263) (#14273) (cherry picked from commit e9553e8) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com>

adding improvements in the new docs

c860a30

(cherry picked from commit 08bd4c7)

Upgrade to Geoserver 2.28.4 (#14289)

08d8e88

[Fixes #14291] Add parameter for beat DB file location (#14292) (#14303)

a36ec4a

* Add parameter for beat DB file location * Cleaner location for BEAT_DB (review comment from Gpetrak) (cherry picked from commit d295a18) Co-authored-by: Wolfgang Kaltz <jwkaltz70@gmail.com>

Update django_geonode_mapstore_client version

918429c

[Backport 5.1.x][Fixes #14306] Refact create_tile_layers as a subcomm…

96cfb17

…and of gwc (#14313) (cherry picked from commit 820b5a8) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

Fix #14305 (#14307) (#14316)

8eadd6f

(cherry picked from commit 929ad55) Co-authored-by: Wolfgang Kaltz <jwkaltz70@gmail.com>

[Fixes #11920] Remove call to command rebuild_index in tasks.py (#14304…

4fa63e8

…) (#14317) (cherry picked from commit 77f4f51) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

[Fixes #14318] Improve logging (#14319) (#14320)

026505b

(cherry picked from commit 2c0fdfd) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

Sanitize metadata input

2168d41

(cherry picked from commit 0803857)

[Fixes #14339] Drop support Dropbox storage manager (#14347)

2263d0b

(cherry picked from commit 6b1d8db) Co-authored-by: Mattia Giupponi <mattia.giupponi@gmail.com>

Update lxml dependency version to 6.1.1

1ca8488

[Fixes #14344] Pack rdf files in build distribution (#14345) (#14350)

2f11330

(cherry picked from commit 109bc4b) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

[Fixes #14000] Refact extraMetadata (#14286) (#14351)

c78706c

* Squashed PR #14001 (cherry picked from commit a9ade43) Co-authored-by: Emanuele Tajariol <etj@geo-solutions.it>

Fix to exception handling for XLSX (#14138) (#14352)

2592448

(cherry picked from commit 0e09ec4) Co-authored-by: Giovanni Allegri <giohappy@gmail.com>

[Backport Fixes #14354] Cannot upload multiple layers from GPKG file (#…

de298ee

…14357)

increase pyjwt-version

1b8ffd1

cla-bot Bot added the cla-signed CLA Bot: community license agreement signed label Sep 24, 2026

etj self-assigned this Sep 24, 2026

etj requested a review from mattiagiupponi September 24, 2026 16:05

Copilot started reviewing on behalf of etj September 24, 2026 16:05 View session

etj added enhancement metadata backport 5.1.x labels Sep 24, 2026

etj linked an issue Sep 24, 2026 that may be closed by this pull request

Management command for metadata validation #14651

Open

Copilot AI reviewed Sep 24, 2026

View reviewed changes

github-actions Bot and others added 8 commits September 25, 2026 13:02

Improvements in Asset (#14615) (#14657)

43bb3f8

* Improvements in Asset * Move logic into permissions registry --------- (cherry picked from commit c63fa4f) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com> Co-authored-by: mattiagiupponi <mattia.giupponi@gmail.com>

[Backport 5.1.x][Fixes #14637] The check in is_monochromatic_image fo…

6e43814

…r local thumbnails fails

Fixed typo in dependency

6090afa

[Fixes #14660] XLSX upload fix (#14661) (#14662)

5ae706f

(cherry picked from commit de34307) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com>

Fixed search fields lookup

dbc4e59

(cherry picked from commit 83ea1c0)

[Fixes #14659][5.1.x] Error in uploaded metadata in multilang instances

9ddcc4f

[Fixes #14638] Extent is not assigned correctly to datasets created f…

aa36588

…rom CSV (#14642) (#14667) (cherry picked from commit cac1717) Co-authored-by: Sijan Dhungana <sumoseason35@gmail.com>

etj force-pushed the 14651_metadata_validation branch from cad3b88 to a5bcca6 Compare October 1, 2026 14:19

cla-bot Bot removed the cla-signed CLA Bot: community license agreement signed label Oct 1, 2026

etj requested a lite review from Copilot October 1, 2026 14:19

Copilot started reviewing on behalf of etj October 1, 2026 14:20 View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

[Fixes #14651] Management command for metadata validation

4038798

etj force-pushed the 14651_metadata_validation branch from a5bcca6 to 4038798 Compare October 1, 2026 14:39

etj closed this Oct 1, 2026

Navigation Menu

[Fixes #14651] Management command for metadata validation - #14653

[Fixes #14651] Management command for metadata validation#14653
etj wants to merge 98 commits into
masterfrom
14651_metadata_validation

etj commented Sep 24, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

cla-bot Bot commented Oct 1, 2026

Uh oh!

gitguardian Bot commented Oct 1, 2026

Uh oh!

gitguardian Bot commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

cla-bot Bot commented Oct 1, 2026

Uh oh!

etj commented Oct 1, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants


		parser.add_argument("--debug", dest="debug", action="store_true", help="Set log level to debug")

		def handle(self, args, *options):

Conversation

etj commented Sep 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

cla-bot Bot commented Oct 1, 2026

Uh oh!

gitguardian Bot commented Oct 1, 2026

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Uh oh!

gitguardian Bot commented Oct 1, 2026

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🔵 Needs a closer look

Uh oh!

cla-bot Bot commented Oct 1, 2026

Uh oh!

etj commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

etj commented Sep 24, 2026 •

edited

Loading

etj commented Oct 1, 2026 •

edited

Loading