Skip to content

[Fixes #14651][5.1.x] Management command for metadata validation - #14679

Merged
etj merged 4 commits into
5.1.xfrom
14651_metadata_validation
Oct 2, 2026
Merged

etj merged 4 commits into
5.1.xfrom
14651_metadata_validation

Conversation

@etj

@etj etj commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Add a validate_metadata management command, and report the schema bugs it found

Walks the resources, builds each one's jsonschema instance via metadata_manager, and validates it against the metadata schema. Filters by id and resource type, reports to JSON or CSV, exits non-zero when anything is invalid.

Writing it surfaced three genuine bugs in the schema generation, fixed here:

  • sparse.py emitted "title": null for thesaurus keywords with no label in the requested language, so the fallback to the default label was missing.
  • Optional fields whose subschema rejected a null flagged every resource that left them empty. Nullable fields now get an explicit - choice in their oneOf (_add_oneof_null_option, shared by the base subhandlers and the sparse thesauri).
  • A thesaurus with card_min > 0 was never enforced: minItems alone only constrains a list already present. The mandatory thesauri now go in tkeywords.required, and geonode:required promotes tkeywords itself.

The report separates the two failure kinds: an empty mandatory field is reported against the resource, while an optional field the schema won't let be null is a schema defect, warned about once with the number of resources affected. Errors carry both the JSON path and the resolved field titles, so a /tkeywords/3-2-4-1-gemet-inspire-themes row is readable.

Fixing the null/oneOf schema problem (nullable properties restricted to oneOf list needed an entry withl key null) uncovered a problem client side, addressed by GeoNode/geonode-mapstore-client#2668

Checklist

Reviewing is a process done by project maintainers, mostly on a volunteer basis. We try to keep the overhead as small as possible and appreciate if you help us to do so by completing the following items. Feel free to ask in a comment if you have troubles with any of them.

For all pull requests:

  • Confirm you have read the contribution guidelines
  • You have sent a Contribution Licence Agreement (CLA) as necessary (not required for small changes, e.g., fixing typos in the documentation)
  • Make sure the first PR targets the master branch, eventual backports will be managed later. This can be ignored if the PR is fixing an issue that only happens in a specific branch, but not in newer ones.

The following are required only for core and extension modules (they are welcomed, but not required, for contrib modules):

  • There is a ticket in https://github.com/GeoNode/geonode/issues describing the issue/improvement/feature (a notable exemption is, changes not visible to end-users)
  • The issue connected to the PR must have Labels and Milestone assigned
  • PR for bug fixes and small new features are presented as a single commit
  • PR title must be in the form "[Fixes #<issue_number>] Title of the PR"
  • New unit tests have been added covering the changes, unless there is an explanation on why the tests are not necessary/implemented

Submitting the PR does not require you to check all items, but by the time it gets merged, they should be either satisfied or inapplicable.

@etj etj self-assigned this Oct 1, 2026
@etj etj added the enhancement label Oct 1, 2026
Copilot AI balanced review requested due to automatic review settings October 1, 2026 14:42
@cla-bot cla-bot Bot added the cla-signed CLA Bot: community license agreement signed label Oct 1, 2026
@gitguardian

gitguardian Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Empty choices remain invalid, schema failures can be omitted or return success, and the new command lacks coverage and bounded reporting memory.

Review effort: Balanced
Findings: 4 Medium severity · 1 Low severity

Open (5)
What changed in this PR

Adds metadata validation and reporting for resources, addressing issue #14651.

Changes:

  • Adds JSON/CSV metadata validation command with filtering.
  • Improves nullable schema choices and thesaurus labels.
  • Enforces mandatory thesauri and adds related tests.
File Description
geonode/​metadata/​management/​__init__.py Initializes management package.
geonode/​metadata/​management/​commands/​__init__.py Initializes command package.
geonode/​metadata/​management/​commands/​validate_metadata.py Implements validation and reporting command.
geonode/​metadata/​handlers/​abstract.py Adds nullable-choice helpers.
geonode/​metadata/​handlers/​base.py Adds null options to enumerated fields.
geonode/​metadata/​handlers/​sparse.py Fixes labels and sparse-field nullability.
geonode/​metadata/​handlers/​thesaurus.py Enforces mandatory thesauri.
geonode/​metadata/​schemas/​base.json Marks optional fields nullable.
geonode/​metadata/​tests/​test_handlers.py Tests thesaurus requiredness.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread geonode/metadata/handlers/abstract.py
Comment thread geonode/metadata/management/commands/validate_metadata.py
Comment thread geonode/metadata/management/commands/validate_metadata.py Outdated
Comment thread geonode/metadata/management/commands/validate_metadata.py Outdated
Comment thread geonode/metadata/management/commands/validate_metadata.py Outdated
@codecov

codecov Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.89671% with 26 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (5.1.x@aa36588). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff            @@
##             5.1.x   #14679   +/-   ##
========================================
  Coverage         ?   75.37%           
========================================
  Files            ?      987           
  Lines            ?    62224           
  Branches         ?     8549           
========================================
  Hits             ?    46901           
  Misses           ?    13409           
  Partials         ?     1914           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@etj etj changed the title [Fixes #14651] Management command for metadata validation [Fixes #14651][5.1.x] Management command for metadata validation Oct 1, 2026
@etj
etj merged commit fb92702 into 5.1.x Oct 2, 2026
13 checks passed
@etj
etj deleted the 14651_metadata_validation branch October 2, 2026 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed CLA Bot: community license agreement signed enhancement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants