Repository navigation
Conversation
The startup lifespan retried init_db() with a blocking time.sleep() inside an async function, which stalls the event loop during db reconnection — delaying readiness/health probes that share the loop. Switch to await asyncio.sleep() and add a regression test suite. Closes #140
Linux servers couldn't run 'make reset-dev' cleanly: start-dev.sh launched port-forwards via macOS-only 'open -a Terminal.app'. Guard that behind an OS check and background the port-forwards on Linux (logged to port-forward.log). Add an official installer (scripts/install-ubuntu.sh) and an Ubuntu Server guide (docs/INSTALL_UBUNTU.md), adapted from the community writeup in #116, and link both from the README and docs/INSTALL.md. Align the README Python prerequisite with CI (3.9+). Closes #116
Adds an SMTP email channel and in-app notifications for detections, wired through the existing rule engine, rule canvas, settings, and SDK — no new tables (reuses integration_settings + notifications). - notification_service: EmailService (config from integration_settings with SMTP_* env fallback, à la ChatService) + AlertNotifier (email + per-user in-app notification), best-effort so delivery never blocks alerting - rule_engine._fire_alert dispatches notifications when the alert node action is a notify action, threading the node's optional recipients - settings: register 'smtp' integration, mask the password, add a /test probe - rule canvas: 'fire alert + notify' action + optional recipients field - settings UI: Notifications section with an Email (SMTP) integration card - POST /api/v1/alerts to raise alerts directly (owns them under a system 'SDK Alerts' rule) with optional notify - SDK: openuba.send_alert(...) for models - dedicated test subsuite (28 unit tests, no container needed) Closes #25
Adds a bidirectional Splunk integration, reusing the existing connector / data-source / integration-settings patterns: - SplunkConnector (core/integrations): REST export search (input) + HTTP Event Collector (output), bearer or basic auth, env + integration_settings config - model runner: 'splunk' data_source branch runs an SPL search and returns a DataFrame (self-contained, mirrors the elasticsearch branch) - models train/execute: splunk_search / splunk_index params + dispatch - orchestrator: best-effort forward of each run's anomalies to Splunk HEC when enabled (forward_anomalies), never blocking detection - settings: register 'splunk', mask token/hec_token/password, /test probe - settings UI: Splunk card under Data Engines - docs/SPLUNK.md - dedicated test subsuite (26 unit tests, fully mocked, runs in the CI unit job) Closes #99
The workspace image build failed generating pyspark metadata ('setuptools is
not available in the build environment'). Upgrade build tooling right after the
base image so isolated sdist builds resolve setuptools. Unblocks the workspace
CI triggered by the sdk/ change in this branch.
) First slice of the Data Ingestion & Testing work, on existing rails: - core/services/detection_eval.py: pure, dependency-light entity-level scoring (precision/recall/F1/FPR + per-scenario recall) shaped for ExperimentRun.metrics - anomalies read-by-run: AnomalyRepository.list_all(run_id=...), a run_id query param on GET /api/v1/anomalies, and query_anomalies(run_id=) in the SDK — the read path evaluation needs to score a single run's output - tests: 10 unit tests for the scorer (no infra) + a repo test for the run_id filter
Replace the hard toy_1-only whitelist in ingest_all_datasets with a 'looks like a dataset' check (directory with >=1 log-type subdir). toy_1 behavior is unchanged; additional datasets now fan out to their <name>_<logtype> Spark tables + openuba-<logtype>-<name> ES indices with no code change. Junk/hidden dirs and stray files are still skipped. Tests: dataset-scan unit tests (temp tree, mocked ingest — no infra).
- POST /api/v1/evaluate/run: reads a run's anomalies (by run_id) and scores them with the shared core.services.detection_eval scorer; returns JSONB- friendly metrics ready for ExperimentRun.metrics - SDK: openuba.evaluate_run(run_id, malicious_entities, ...) + query_anomalies now accepts run_id - tests: API test (seed run+anomalies, POST, check metrics) + 3 SDK unit tests
- inject an admin caller (override get_current_user) so the evaluation API test exercises the real require_permission path and returns 200 - pass the required slug to ModelRepository.create in the new tests - docs/DATASETS_AND_EVALUATION.md: adding datasets, ground truth, and evaluating a run (precision/recall/f1) via the SDK, reusing experiments
This branch touches sdk/, which triggers the workspace image build; it fails generating pyspark metadata without modern build tooling. Same fix already validated on the alerts branch.
) raising an alert is a detection-output action, not rule-definition management. change create_alert's guard from require_permission("rules", "write") to require_permission("anomalies","write") so emitting alerts (e.g. from a model via the SDK) doesn't require rule-editing privileges. add the missing POST /alerts coverage (audit found ZERO): admin happy path (201, get-or-create SDK rule), unknown rule_id -> 404, and unauth -> 401. also add the _as_admin autouse override + unauthenticated_client fixtures, which fixes the 3 pre-existing auth-less failures in this file (test_create_rule/list_rules/get_rule were red because they never authenticated) — net no regression. local draft, not pushed.
two audit findings on the Splunk integration: 1. HEC anomaly-forwarding was DEAD in kubernetes (prod) mode. the forward call sat inside the 'execution_mode != kubernetes' block, but in k8s the runner persists anomalies directly to the DB and result["anomalies"] is empty — so the feature never fired in the mode it's meant for. add a both-modes path: docker forwards the in-memory result (as before); k8s re-reads this run's anomalies by run_id in bounded batches (reusing the rule-eval read pattern) and forwards them. factor the config/gating into _splunk_forward_target() shared by both paths. 2. SSRF: host/hec_url are admin/input_data-controlled (models:write reach). add _assert_safe_endpoint() — require http/https scheme and refuse hosts resolving to link-local (169.254.0.0/16, fe80::/10, incl. cloud instance metadata 169.254.169.254). internal/private/loopback Splunk is still allowed (the normal deployment). wired into search/send_event/test_connection. tests: +5 SSRF (scheme, link-local reject, private allow, send_event/search refuse) and +3 k8s-forward (re-read-by-run_id sends, skip-when-disabled, never-raises). 34 splunk tests pass. NOTE (deferred, needs k8s e2e): the runner's splunk branch still reads creds from env only, not integration_settings — UI-configured Splunk won't auth in k8s. tracked for a follow-up that can be exercised end-to-end. local draft.
) adversarial-audit follow-ups: - the previous anomalies:write guard was an access no-op (only analyst has write on rules OR anomalies by default, same set) AND inconsistent with GET /alerts which uses the 'alerts' page. switch create_alert to require_permission("alerts","write") — consistent with list_alerts, and 'alerts' write is admin-only by default (appropriate for direct alert injection; grantable per-role via the permissions UI). - the headline permission was previously UNTESTED (every test authenticates as admin, which short-circuits require_permission). add discriminating tests via a real non-admin role + seeded role_permissions reached through get_db_context: an analyst WITH anomalies:write but WITHOUT alerts:write is denied 403 (proves it's keyed on alerts, not anomalies), and an analyst granted alerts:write gets 201. - update SDK send_alert docstring: rules:write -> alerts:write. test_rules.py: 8 passed. local draft, not pushed.
adversarial-audit follow-ups on the splunk hardening: - SSRF bypass: _assert_safe_endpoint only checked is_link_local, so an IPv4-mapped IPv6 (::ffff:169.254.169.254) resolved to metadata unblocked. unwrap ipv4_mapped before classifying and also block the unspecified address (0.0.0.0/::). - perf regression: send_anomalies -> send_event resolved DNS (getaddrinfo) once PER anomaly — 195k resolutions for a large run. memoize validation per unique endpoint on the connector (_ensure_safe_endpoint); 195k -> 1. - redirect bypass: add allow_redirects=False to every splunk REST/HEC call (Splunk never legitimately redirects; prevents validate-then-follow-to- metadata). - the model-runner's elasticsearch/splunk search path (docker/model-runner/ runner.py) built requests inline with a model-input-controlled host and NO guard — the exact SSRF on the ingest side, in k8s/prod. add an inlined _assert_safe_url (runner can't import core) + allow_redirects=False on both. - verify_ssl=false now logs a warning. - k8s anomaly forward gated on run_type == 'infer' (train has no anomalies). tests: +4 (ipv4-mapped-IPv6 block, unspecified block, hostname->metadata block, getaddrinfo-once perf lock-in). 38 splunk tests pass; runner compiles. NOTE still deferred (needs k8s e2e): runner reads splunk creds from env only. local draft, not pushed.
address the two deferred audit items: - _get_or_create_sdk_rule race: give the synthetic rule a deterministic primary key and INSERT inside a SAVEPOINT. concurrent first-time callers now collide on the (already-unique) PK — one wins, the other catches IntegrityError and re-reads the winner — so no duplicate 'SDK Alerts' rows accumulate. no schema migration / no risky global-unique-on-name needed. - unbounded in-app notification fan-out: cap the per-alert fan-out (OPENUBA_MAX_INAPP_NOTIFICATIONS, default 500) so one alert can't write unboundedly many rows synchronously in the request path; report in_app_truncated and log when capped. tests: +sdk-rule idempotency (deterministic id, exactly one row) and +fan-out cap (truncates at the limit). 33 tests pass. local draft, not pushed.
address the deferred audit items: - runner creds: the runner reads SPLUNK_* from its env but nothing bridged the UI-configured integration_settings creds to it, so UI-configured Splunk did not authenticate. add ModelOrchestrator._splunk_runner_env() which surfaces the enabled splunk config as SPLUNK_* and inject it as EPHEMERAL container env for splunk-source docker runs (never persisted to input_data/ DB). runner unchanged (already reads env). NOTE: k8s injection still needs the UBAInference CRD + operator to forward these from a Secret — that piece genuinely needs cluster e2e and is the remaining deployment step. - fail-open SSRF: add opt-in SPLUNK_SSRF_STRICT=true to fail CLOSED when a host can't be resolved for validation (default stays fail-open for best-effort egress + to not break transient DNS). tests: +runner-env (collects creds / empty when disabled) and +strict-mode (fail-open default vs fail-closed when strict). 41 splunk tests pass. local draft.
verification found the race-recovery branch untested. re-read the winner by the fixed id (the conflict is on the PK, not the name) and add a test that genuinely drives the IntegrityError path: pre-occupy the fixed id under a different name so the create flush conflicts inside the SAVEPOINT and the except branch re-reads the occupant by id (no raise, no duplicate). test_rules.py: 10 passed. local draft.
) the goal audit flagged that Splunk-as-INPUT-source in kubernetes can't get credentials (the UBAInference/UBATraining operator doesn't read integration settings). docker mode is now bridged (ephemeral SPLUNK_* env, prior commit); the k8s input path requires operators to supply SPLUNK_* to the runner pod (e.g. a Secret via envFrom) until the operator learns to sync them — a planned follow-up. document this explicitly so it is a known, loud-failing deployment requirement rather than a silent gap. HEC anomaly FORWARDING runs backend-side and already works in both modes. local draft, not pushed.
audit flagged that docker/model-runner/runner.py's _assert_safe_url was verified by code-reading only — the two SSRF-guard copies (core vs runner) could silently drift. load the runner module directly and assert it blocks link-local/metadata, IPv4-mapped IPv6 (::ffff:169.254.169.254), and non-http schemes while allowing internal/private hosts and failing open on unresolvable. 46 splunk/runner tests pass. local draft.
Signed-off-by: pushnanashi2 <relomeru.y@gmail.com>
…#147/#148) secret at startup (_load_secret_key fails closed if unset/placeholder/too short); reject forged tokens that carry role:admin without a DB-validated user_id; make require_permission fail CLOSED on errors (except the first-boot missing-table case). Wire JWT_SECRET_KEY into docker-compose + k8s secret/deployment; set it for tests in conftest before any core import. cases (list/get, mirrors external #148), rules (list/get), alerts (list), anomalies (list/get). source_groups left to #127. Tests use the dependency-override auth pattern (robust to the #149 hardening) + explicit 401 tests; also repairs pre-existing case/rule/anomaly tests that never authenticated. NOTE: applied + py_compile clean; local test-suite run pending (venv/disk issue).
audit follow-up: the invalid-user_id rejection test's label overstated that it exercises a missing DB row; in the harness it fires via the fail-closed path when the user validation query cannot succeed. reword the docstring/comment to reflect the real (broader) guarantee — a well-formed token whose DB validation does not succeed is rejected, whether the row is absent or the DB is unreachable. no behavior change.
…127) source_groups CRUD was fully unauthenticated, incl. create/update writes — the last unauthenticated write surface in the cases/rules/anomalies neighborhood. add require_permission("source_groups", read|write) to all four routes (reads for manager/analyst, writes admin-only) and seed the new 'source_groups' page in DEFAULT_PERMISSIONS so non-admin roles resolve correctly instead of 403-ing. adds test_source_groups.py (unauth reads AND writes -> 401, admin happy paths). NOTE: supersedes the stalled external #127 (which used a bespoke require_source_group_write_access instead of the standard require_permission); reconcile/close #127 when this is opened as a PR. local draft, not pushed.
…127) adversarial-audit follow-ups on the auth-hardening workstream: - HIGH: seed_defaults() used an all-or-nothing 'count()==0' gate, so adding the source_groups page only helped FRESH installs — already-seeded (upgraded) deployments never got the rows, so manager/analyst would 403 on source_groups reads. make BOTH seed paths (fastapi_app.seed_defaults and core/db/seed._seed_permissions) idempotent PER (role, page) so new pages are backfilled on upgrade; existing (admin-edited) rows are untouched. add source_groups to core/db/seed DEFAULT_PERMISSIONS too. - add 'source_groups' to ALL_PAGES (core/api_routers/auth.py) so admins can actually edit its permissions in the UI (update_permissions skips unknown pages). - test(c) was near-vacuous (it 401'd because the module-engine DB was unreachable, not because the user row was missing). add a genuine test that wires get_current_user's get_db_context to the testcontainer and asserts a token with an absent user_id is rejected via the row-is-None path. - add test_db/test_seed_permissions.py proving idempotent backfill + no dup. - conftest test_client teardown: pop(get_db) instead of clear() (don't wipe other fixtures' overrides). - source_groups SourceGroupRead: orm_mode -> from_attributes (pydantic 2.5). 42 tests pass (auth_security/cases/rules/anomalies/source_groups/seed). local draft.
the goal-coverage audit found two unauthenticated read surfaces the session's 'close the auth gaps' objective should cover (beyond #147/#148): - core/api_routers/display.py: BOTH routes were fully unauthenticated. the /display/{type} proxy exposes get_all_users / get_all_entities / get_system_log and /display/home/summary leaks aggregate counts — a strictly worse leak than #147. guard both with require_permission("home","read"). - core/api_routers/models.py: list_models / search_models / get_model lacked require_permission; add ("models","read") to match the write routes. add _as_admin + unauthenticated_client fixtures and 401 read tests to test_display.py and test_models.py (which also fixes their pre-existing auth-less write-route failures). 50 tests pass across the security suite. NOTE: per-display-type RBAC (e.g. user-listing admin-only) is a further refinement; this closes the unauthenticated hole. local draft, not pushed.
the independent local audit found auth gaps the earlier passes missed, all
within the 'close the read/write auth gaps' mandate:
- POST /chat was a FULLY UNAUTHENTICATED LLM proxy (billable, accepts page
context) — cost-abuse / prompt-injection / data-exposure risk. require
authentication via require_permission("home","read").
- GET /feedback was unauthenticated while POST /feedback was guarded (the
read-open/write-locked asymmetry #147 is about). add get_current_user,
mirroring the POST.
- core/db/seed.py (standalone CLI seeder) had NO 'manager' rows and set
analyst alerts can_write=True, diverging from the authoritative
fastapi_app.seed_defaults. under the new fail-closed require_permission a
CLI-seeded DB would 403 every manager everywhere. add the manager rows
(read-only, mirroring fastapi_app) and align analyst alerts to write=False.
tests: +test_chat.py (401 unauth), +feedback read-401 (and the _as_admin
fixture repairs feedback's pre-existing auth-less failures), seed test updated
for the manager source_groups row. security suite 54 pass. local draft.
) the cases read-auth change (identical to PR #148) makes the e2e case-flow calls return 401; thread the existing e2e auth_headers fixture (real admin login -> token with user_id) through every models/anomalies/cases request. closes the gap the Copilot review flagged on PR #148. our dependency-override unit tests are kept as-is (robust to the #149 no-user_id rejection that PR #148's create_access_token(sub,role) headers would trip). Co-authored-by: pushnanashi2 <relomeru.y@gmail.com>
the app now fails fast at import if JWT_SECRET_KEY is unset (#149 hardening), so the verify-imports job needs a throwaway secret, same as compose/k8s do.
add develop to the push + pull_request branch filters in ci-backend/docker/ frontend/sdk/workspace so PRs targeting develop run the same checks. also set the JWT_SECRET_KEY throwaway on the import-check step (matches the #149 hardening that lands via #150). publish workflows (docker-publish, publish-sdk) stay master-only.
Jovonni
added
bug
# Conflicts: # core/api_routers/settings.py # interface/src/components/settings/settings-tabs.tsx
# Conflicts: # core/tests/test_api_routers/test_rules.py
Jovonni
changed the title
This was referenced Oct 16, 2026
Open
Open
…merge macOS sync-conflict copies (byte-identical to the old pre-hardening sources); not importable, but dead weight / a trap. drop them from the staging branch.
- blocking: saving an integration wrote the MASKED secret back over the real one (GET returns secrets masked, the UI echoes them on save) — corrupting the splunk token/hec_token (and smtp password) so auth died. keep the stored secret when the incoming value is just its mask (_merge_preserving_secrets). - docker: rewrite localhost/127.0.0.1 -> host.docker.internal for SPLUNK_HOST/ SPLUNK_HEC_URL (we already do this for db/es) so a host/compose splunk is reachable from the runner's bridge network. - splunk_index was accepted everywhere but never applied to the search; scope the SPL to it when the query doesn't already pin an index. tests: secret-merge round-trip + index scoping. 52 splunk/runner tests pass.
splunk_index is user-controlled (models:write) and was interpolated into the SPL unescaped. restrict it to [A-Za-z0-9_-] and reject anything else; add rejection tests and a docs note on the index field.
fix(splunk): make the integration actually work (secret-save, docker net, index)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
staging branch collecting the current batch before it goes to master.
git checkout developto run it all together and kick the tires.credits: