Add Open Pentest Format (OPF) parser

[Su1ph3r]

Adds a parser for the Open Pentest Format (OPF), a JSON format for pentest
findings (spec: https://cairnsecurity.com/opf). It reads a .opf.json file and
maps each finding to a DefectDojo finding, so an OPF export imports directly with
no conversion step.

Mapping

OPF	DefectDojo Finding
severity	severity (informational becomes Info)
cvssScore / cvssVector	cvssv3_score / cvssv3
first cweIds / cweId	cwe
cveIds	unsaved_vulnerability_ids
recommendation	mitigation
impact	impact
stepsToReproduce	steps_to_reproduce
references	references
URL affectedAssets	endpoints (other assets go in the description)
testType, owaspCategory, mitreTechniques	tags
id	unique_id_from_tool / vuln_id_from_tool

OPF text is sometimes HTML (textFormat: "html"), so the parser flattens it to
plain text.

Included

• dojo/tools/opf/ parser
• Unit test (unittests/tools/test_opf_parser.py) and two sample scans (unittests/scans/opf/)
• Docs page (docs/content/supported_tools/parsers/file/opf.md)
• Dedupe config for the OPF Scan scan type in settings.dist.py

Testing

python manage.py test unittests.tools.test_opf_parser --keepdb covers an empty
document and a four-finding document: severity mapping, CWE, CVSS score and
vector, HTML flattening, tags, and findings that carry no CVSS.

[Su1ph3r]

The two red checks are the Unit Tests Complete gate and one UI test shard (login_test / alerts_test / system_settings_test). That shard failed on relation "dojo_system_settings" does not exist, which looks like a DB/migration setup race in the test container rather than anything from this PR, and the gate just mirrors that single failure.

This change is a file parser that doesn't touch the UI, and the rest-framework unit tests pass, so nothing that exercises the parser is red. Could a maintainer re-run the failed jobs when you have a moment? Happy to rebase or push a change if you'd prefer.

[Su1ph3r]

Correction to my note above: I misread the log. The relation "dojo_system_settings" does not exist line comes from postgres during migration and is not the failure.

The shard actually failed on tests/engagement_extended_test.py::test_close_engagement_for_reopen, a Selenium engagement test that errored on a WebDriver call. The Unit Tests Complete gate is just mirroring that one shard.

The conclusion is unchanged, but the evidence I cited for it was wrong. This PR adds a file parser under dojo/tools/opf/ plus a three-line entry in settings.dist.py, and touches nothing in the engagement UI. The parser unit tests pass.

Happy to rebase onto current master so the shard re-runs, or to make any changes you would prefer.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[Su1ph3r]

Rebased onto current dev, conflicts resolved. The only conflict was in settings.dist.py, where recent dev added new dedupe entries alongside this PR's OPF Scan entries; I kept both. History is now a single parser commit on top of dev, and the diff is unchanged (7 files, parser + tests + docs). Ready for review whenever a maintainer has a moment.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[Su1ph3r]

Rebased onto current dev again, conflict resolved. As before it was only settings.dist.py, where #15611's new parser entries sit in the same dedupe dicts as this PR's OPF Scan entries; I kept both. Still one parser commit on top of dev, diff unchanged (7 files: parser + tests + docs).

Since settings.dist.py re-conflicts with nearly every parser PR, this branch tends to drift back into conflict within a day or two. It's clean and green-able right now, so it'd be great to get a review in this window while it applies cleanly. @valentijnscholten @Maffooch @blakeaowens whenever one of you has a moment, happy to make any changes you'd like.

[github-actions]

Conflicts have been resolved. A maintainer will review the pull request shortly.

[valentijnscholten]

Thank you @Su1ph3r . Currently the test suite is broken. Once it passes we can review/approve/merge.

[valentijnscholten]

Reviewed the OPF parser — nice, self-contained addition with solid input guarding (titleless-entry skip, type checks, URL-vs-other asset split). Two things worth addressing, both in the field mapping:

1. dojo/tools/opf/parser.py:80 — CVSS vector stored raw in cvssv3 with no version handling

vector = entry.get("cvssVector")
if isinstance(vector, str) and vector:
    finding.cvssv3 = vector
score = entry.get("cvssScore")
if isinstance(score, (int, float)):
    finding.cvssv3_score = float(score)

OPF's cvssVector isn't pinned to CVSS v3, and pentest exports increasingly use v4 (and legacy v2). As written, a v4 vector lands mislabeled in the v3 field and cvssv4 is never populated; a v2 vector is likewise misfiled; and even a v3 vector isn't normalized (clean_vector()), with the score trusted from the tool's self-report rather than derived from the vector.

The repo already has dojo.utils.parse_cvss_data(), which detects the version and routes v4 → cvssv4, v3 → cvssv3, v2 → cvssv2 with authoritative scores. It's the established path in recent parsers (openvas, auditjs, cyberwatch). Suggest routing cvssVector through it, e.g.:

from dojo.utils import parse_cvss_data
...
cvss = parse_cvss_data(entry.get("cvssVector") or "")
if cvss.get("cvssv3"):
    finding.cvssv3 = cvss["cvssv3"]
if cvss.get("cvssv4"):
    finding.cvssv4 = cvss["cvssv4"]

(falling back to cvssScore only when the vector yields no score).

2. dojo/tools/opf/parser.py:181 — _html_to_text uses a hand-rolled, incomplete HTML-entity map

replacements = {
    "&nbsp;": " ", "&amp;": "&", "&lt;": "<", "&gt;": ">", "&quot;": '"',
    "&#39;": "'", "&rsquo;": "'", "&lsquo;": "'", "&ldquo;": '"', "&rdquo;": '"',
}

This decodes only ~10 entities. Common ones that OPF HTML can contain — ', ', numeric refs like ’, §, etc. — pass through as literal text into description/impact/mitigation. Python's stdlib html.unescape() decodes the full named + numeric set and can replace the dict wholesale — keep the <br>/<p>/<li> → newline regexes, then run html.unescape(text) after tag stripping (same order as now):

import html
...
text = re.sub(r"<[^>]+>", "", text)
text = html.unescape(text)
return re.sub(r"\n{3,}", "\n\n", text).strip()

Neither blocks import; #1 is the more important since it's a silent data-quality loss on v4 CVSS vectors. The endpoint handling, dedupe config, and guards otherwise look good.

[valentijnscholten]

Correction to my previous comment: both findings are blocking and should be fixed before merge.

• parser.py:80 — CVSS handling: storing cvssVector raw in cvssv3 silently mislabels v4/v2 vectors (and leaves cvssv4 empty), which is a data-integrity problem on import, not a nicety. Please route it through dojo.utils.parse_cvss_data() as noted above.
• parser.py:181 — HTML entity decoding: the partial entity map leaks literal entity codes (', ', ’, …) into description/impact/mitigation. Please switch to stdlib html.unescape() after tag stripping.

Details and suggested snippets are in the review comment above. Happy to re-review once these are addressed.

[Su1ph3r]

Thanks for the review @valentijnscholten, both addressed in 340a47b:

1. CVSS versioning — cvssVector now routes through dojo.utils.parse_cvss_data(), so v4 lands in cvssv4 and v3 in cvssv3, each with the library-derived score. The reported cvssScore is used only as a fallback when the vector yields no score (no vector, unparseable, or a v2 vector with no field to hold it). Added a test asserting a CVSS:4.0 vector populates cvssv4 (not cvssv3).
2. HTML entities — _html_to_text now uses stdlib html.unescape() after tag stripping, replacing the partial map. Added a test covering ', ', ’, é.

Ran ruff clean and verified the CVSS routing against the cvss library (v4 → cvssv4 score 9.3; the existing v3.1 sample still resolves to 9.8). Ready for another look.