Repository navigation
feat(integrations): Splunk data source + anomaly forwarding (#99) - #145
Merged
Merged
Conversation
Adds a bidirectional Splunk integration, reusing the existing connector / data-source / integration-settings patterns: - SplunkConnector (core/integrations): REST export search (input) + HTTP Event Collector (output), bearer or basic auth, env + integration_settings config - model runner: 'splunk' data_source branch runs an SPL search and returns a DataFrame (self-contained, mirrors the elasticsearch branch) - models train/execute: splunk_search / splunk_index params + dispatch - orchestrator: best-effort forward of each run's anomalies to Splunk HEC when enabled (forward_anomalies), never blocking detection - settings: register 'splunk', mask token/hec_token/password, /test probe - settings UI: Splunk card under Data Engines - docs/SPLUNK.md - dedicated test subsuite (26 unit tests, fully mocked, runs in the CI unit job) Closes #99
There was a problem hiding this comment.
Pull request overview
This PR adds a bidirectional Splunk integration to OpenUBA: querying Splunk as a model data source (via the REST export endpoint) and optionally forwarding detected anomalies back to Splunk (via HEC), along with settings UI/API wiring and a dedicated unit test suite.
Changes:
- Added a
SplunkConnector(REST export search + HEC output) and registered it in integrations and settings test-probe/masking/whitelist plumbing. - Extended the model runner and model train/execute API to support a
splunkdata source withsplunk_search(and exposedsplunk_index). - Added orchestrator-side best-effort anomaly forwarding to Splunk HEC plus unit tests and user documentation.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| interface/src/components/settings/settings-tabs.tsx | Adds a Splunk integration card/fields to the settings UI. |
| docs/SPLUNK.md | Documents Splunk configuration, SPL search usage, and anomaly forwarding. |
| docker/model-runner/runner.py | Adds a splunk branch in _load_data_for_model to run Splunk export searches and build a DataFrame. |
| core/integrations/splunk.py | Introduces SplunkConnector for search + HEC event/anomaly forwarding + connectivity probe. |
| core/integrations/init.py | Exposes SplunkConnector from the integrations package. |
| core/api_routers/settings.py | Whitelists splunk, adds /test dispatch to _test_splunk, and masks Splunk secrets. |
| core/api_routers/models.py | Adds splunk_search/splunk_index request parameters and input-data dispatch for train/execute. |
| core/services/model_orchestrator.py | Adds best-effort forwarding of anomalies to Splunk HEC after inference persistence (Docker mode only as written). |
| core/tests/test_services/test_splunk_connector.py | Unit tests for connector normalization, auth, search parsing, HEC output, and connectivity probe. |
| core/tests/test_services/test_splunk_wiring.py | Unit tests for settings whitelist/masking/test probe and orchestrator forwarding gating. |
Suppressed comments (1)
core/api_routers/models.py:488
- Same issue as the train endpoint:
splunk_indexis forwarded intoinput_databut ignored by the model-runner Splunk loader, and there is no wiring here to provide the required Splunk host/auth into the runner container for Splunk searches.
elif _data_source == "splunk" and _splunk_search:
input_data["splunk_search"] = _splunk_search
if _splunk_index:
input_data["splunk_index"] = _splunk_index
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+288
to
+292
| async def _test_splunk(config: dict) -> dict: | ||
| '''verify Splunk connectivity via the shared SplunkConnector''' | ||
| from core.integrations.splunk import SplunkConnector | ||
| return SplunkConnector.from_config(config).test_connection() | ||
|
|
Comment on lines
175
to
+178
| db.commit() | ||
|
|
||
| # optional: forward anomalies to Splunk (HEC) if configured | ||
| self._forward_anomalies_to_splunk(result["anomalies"]) |
Comment on lines
+373
to
+376
| elif _data_source == "splunk" and _splunk_search: | ||
| input_data["splunk_search"] = _splunk_search | ||
| if _splunk_index: | ||
| input_data["splunk_index"] = _splunk_index |
Comment on lines
+40
to
+44
| -d '{ | ||
| "data_source": "splunk", | ||
| "splunk_search": "index=proxy sourcetype=access_combined", | ||
| "splunk_index": "proxy" | ||
| }' |
Comment on lines
+412
to
+415
| headers = {"Content-Type": "application/x-www-form-urlencoded"} | ||
| if token: | ||
| headers["Authorization"] = f"Bearer {token}" | ||
| auth = (username, password) if (not token and username) else None |
Comment on lines
+198
to
+204
| def send_anomalies(self, anomalies: List[Dict[str, Any]], index: Optional[str] = None) -> int: | ||
| '''forward a batch of anomalies; returns the count successfully sent''' | ||
| sent = 0 | ||
| for a in anomalies: | ||
| if self.send_anomaly(a, index=index): | ||
| sent += 1 | ||
| return sent |
two audit findings on the Splunk integration: 1. HEC anomaly-forwarding was DEAD in kubernetes (prod) mode. the forward call sat inside the 'execution_mode != kubernetes' block, but in k8s the runner persists anomalies directly to the DB and result["anomalies"] is empty — so the feature never fired in the mode it's meant for. add a both-modes path: docker forwards the in-memory result (as before); k8s re-reads this run's anomalies by run_id in bounded batches (reusing the rule-eval read pattern) and forwards them. factor the config/gating into _splunk_forward_target() shared by both paths. 2. SSRF: host/hec_url are admin/input_data-controlled (models:write reach). add _assert_safe_endpoint() — require http/https scheme and refuse hosts resolving to link-local (169.254.0.0/16, fe80::/10, incl. cloud instance metadata 169.254.169.254). internal/private/loopback Splunk is still allowed (the normal deployment). wired into search/send_event/test_connection. tests: +5 SSRF (scheme, link-local reject, private allow, send_event/search refuse) and +3 k8s-forward (re-read-by-run_id sends, skip-when-disabled, never-raises). 34 splunk tests pass. NOTE (deferred, needs k8s e2e): the runner's splunk branch still reads creds from env only, not integration_settings — UI-configured Splunk won't auth in k8s. tracked for a follow-up that can be exercised end-to-end. local draft.
adversarial-audit follow-ups on the splunk hardening: - SSRF bypass: _assert_safe_endpoint only checked is_link_local, so an IPv4-mapped IPv6 (::ffff:169.254.169.254) resolved to metadata unblocked. unwrap ipv4_mapped before classifying and also block the unspecified address (0.0.0.0/::). - perf regression: send_anomalies -> send_event resolved DNS (getaddrinfo) once PER anomaly — 195k resolutions for a large run. memoize validation per unique endpoint on the connector (_ensure_safe_endpoint); 195k -> 1. - redirect bypass: add allow_redirects=False to every splunk REST/HEC call (Splunk never legitimately redirects; prevents validate-then-follow-to- metadata). - the model-runner's elasticsearch/splunk search path (docker/model-runner/ runner.py) built requests inline with a model-input-controlled host and NO guard — the exact SSRF on the ingest side, in k8s/prod. add an inlined _assert_safe_url (runner can't import core) + allow_redirects=False on both. - verify_ssl=false now logs a warning. - k8s anomaly forward gated on run_type == 'infer' (train has no anomalies). tests: +4 (ipv4-mapped-IPv6 block, unspecified block, hostname->metadata block, getaddrinfo-once perf lock-in). 38 splunk tests pass; runner compiles. NOTE still deferred (needs k8s e2e): runner reads splunk creds from env only. local draft, not pushed.
address the deferred audit items: - runner creds: the runner reads SPLUNK_* from its env but nothing bridged the UI-configured integration_settings creds to it, so UI-configured Splunk did not authenticate. add ModelOrchestrator._splunk_runner_env() which surfaces the enabled splunk config as SPLUNK_* and inject it as EPHEMERAL container env for splunk-source docker runs (never persisted to input_data/ DB). runner unchanged (already reads env). NOTE: k8s injection still needs the UBAInference CRD + operator to forward these from a Secret — that piece genuinely needs cluster e2e and is the remaining deployment step. - fail-open SSRF: add opt-in SPLUNK_SSRF_STRICT=true to fail CLOSED when a host can't be resolved for validation (default stays fail-open for best-effort egress + to not break transient DNS). tests: +runner-env (collects creds / empty when disabled) and +strict-mode (fail-open default vs fail-closed when strict). 41 splunk tests pass. local draft.
) the goal audit flagged that Splunk-as-INPUT-source in kubernetes can't get credentials (the UBAInference/UBATraining operator doesn't read integration settings). docker mode is now bridged (ephemeral SPLUNK_* env, prior commit); the k8s input path requires operators to supply SPLUNK_* to the runner pod (e.g. a Secret via envFrom) until the operator learns to sync them — a planned follow-up. document this explicitly so it is a known, loud-failing deployment requirement rather than a silent gap. HEC anomaly FORWARDING runs backend-side and already works in both modes. local draft, not pushed.
audit flagged that docker/model-runner/runner.py's _assert_safe_url was verified by code-reading only — the two SSRF-guard copies (core vs runner) could silently drift. load the runner module directly and assert it blocks link-local/metadata, IPv4-mapped IPv6 (::ffff:169.254.169.254), and non-http schemes while allowing internal/private hosts and failing open on unresolvable. 46 splunk/runner tests pass. local draft.
Jovonni
added a commit
that referenced
this pull request
Oct 16, 2026
…merge macOS sync-conflict copies (byte-identical to the old pre-hardening sources); not importable, but dead weight / a trap. drop them from the staging branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Integrates Splunk in both directions (requested in #99):
How it reuses existing infrastructure
SplunkConnector(core/integrations/splunk.py) sits alongsideElasticsearchConnector/SparkConnector— REST export search for input, HTTP Event Collector for output, bearer or basic auth. Config viaintegration_settings(from_config) orSPLUNK_*env.splunkbranch in_load_data_for_model, mirroring the elasticsearch branch (self-containedrequests, returns a DataFrame with numeric coercion).splunk_search/splunk_indexparams + a dispatch branch, exactly like the existingspark/elasticsearch/source_groupsources.forward_anomaliesis enabled — best-effort, so a Splunk outage never affects detection or persistence.splunkregistered in the whitelist, secrets (token/hec_token/password) masked,/testconnectivity probe, and a Splunk card in the Data Engines settings panel.docs/SPLUNK.md.Tests (dedicated subsuite, CI-wired)
core/tests/test_services/test_splunk_connector.py+test_splunk_wiring.py— 26 unit tests, all mocked (no live Splunk): SPL normalization, search parsing + bearer/basic auth, error handling, HEC output + auth header, batch send counts, connectivity probe, settings whitelist/masking/_test_splunk, and orchestrator forwarding gating (enabled/disabled/flag-off/error-safe). Placed undertest_services/so they run in the CI unit job (the-kfilter excludes anything undertest_integrations/).Local: 26/26 green, runner compiles,
tsc/next lint/next buildall clean.Follow-up (not in this PR)
Splunk-as-a-physical-source inside
source_groups(so a source group can mix Splunk with other sources) is a natural next step.Closes #99