feat(oie): RQ A→B→C fan-out and Admin OIE queue tab by northdpole · Pull Request #1142 · OWASP/OpenCRE

northdpole · 2026-10-09T08:36:59Z

Summary

Add RQ queue oie fan-out for Module A (per repo) → B/C (per artifact_id), with --rq / CRE_OIE_RQ, start_oie_workers, and harvest_input.artifact_id migration.
Admin OIE queue tab + shared RQ dashboard snapshot; wire import-all / workers for the oie queue.
Fix B2 eval harvester LangGraph repos_yaml TypeError so top10/llm/api score arms can run; keep promoted librarian/safety knobs usable on RQ workers.

Test plan

python -m unittest application.tests.oie_rq.fanout_test application.tests.oie_rq.stage_filter_test application.tests.oie_b2_gold_loader_test.TestOieB2HarvesterLanggraphKwargs
make alembic-guardrail (new revision a9b0c1d2e3f4)
Local: make start-oie-workers + scripts/run_oie_pipeline.py --rq (or full_pipeline --rq) drains queue without silent B2 TypeError
Admin OIE tab loads /admin/oie/rq/status with queue depth

Stacked on #1137 (admin-panel-showcase).

Made with Cursor

coderabbitai · 2026-10-09T08:37:10Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review: 1

main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: OWASP/OpenCRE/.coderabbit.yml
Review profile: CHILL
Plan: Advanced
Run ID: 5c3d1040-8518-4406-a1ec-fbdc78afdbf9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Add LOGIN_ALLOWED_EMAILS so only Spyros/Rob/Rock can establish sessions on internet-facing staging.

Record staging.opencre.org tunnel/DNS layout and LOGIN_ALLOWED_EMAILS env for the three OWASP operators.

Ship deploy/staging with own Postgres/Redis/web/worker, skip-sync entrypoint knobs, and pathspec so the image boots behind the Cloudflare tunnel.

Workers, Admin ingest, and the full-pipeline harness now share apply_promoted_defaults() (pgvector, margin γ=0.85, judge, safety, leaf drilldown). Staging Compose inherits the same knobs; image builds can pull requirements-dev for C.2.

Mint/verify keys for allowlisted operators, audit log + migrations, and stop caching SPA bundles so admin deploys pick up new JS.

Move shortlist/edition/neighbor/safety caches into Postgres, keep edition remap live-LLM off by default, batch safety checks, and gate C.1/C.2 for ablation. Includes accuracy-lever modules, staging compose tweaks, and admin upstream-sync+ingest helpers.

When both fail-closed levers are on, Module C issues a single combined judge call instead of two. Stop oie.env from overwriting worker DEV_DATABASE_URL so parallel accuracy-matrix scoring hits cre_al_wN instead of empty cre.

Keep edition-remap live LLM off, safety batching on, and experimental fail-closed levers off unless an eval explicitly enables them.

Lets operators recover stalled Module C work via /admin/oie/rq without docker or redis CLI.

Ship sidebar shell, needs-attention/review-pulse dashboard, paginated staged-link review with title/content, workbench targets, and GET /admin/config/diff — all still gated by login + CRE_ALLOW_IMPORT.

Staging monitors need API-key access to /admin/oie and related ops routes; also resolve CRE links by uuid or external_id and align ingest test asserts with skip_c1/max_repos=0. Co-authored-by: Cursor <cursoragent@cursor.com>

Operators can see per-run start flags, live HI/KQ/RQ remaining, ETA, and stuck detection on the Pipeline tab (and a strip on Overview).

Remaining/ETA now track KQ+RQ only and use snap-to-snap drain rate so old ImportRuns stop inventing multi-day estimates.

Main landed MCP v2 (f3a91c07d24b) on 967016ee10fa; keep a single Alembic head by parenting the stack's oie_run revision after mcp_grants.

login_or_mcp_required now accepts allowlisted API JWTs as well as session/MCP identity, and API keys carry ALL_SCOPES for require_scopes.

Route auto/security/owasp_meta to structured CRE and OWASP meta facts with optional MyOpenCRE standards filtering; no OpenCRE narrative LLM.

Avoid promoting Standard composite search keys into get_cre_by_id.

Expose remote MCP at /mcp behind OPENCRE_MCP_REMOTE (staging Compose only). Public tools stay open; remote ask requires allowlisted Admin API JWT. Stdio ask remains ungated. Document tunnel /mcp* → :5002 and Cursor config. Co-authored-by: Cursor <cursoragent@cursor.com>

MCP stdio has no Flask request context, so the OWASP sim gate never saw pgvector. Also try meta/evidence before cre_normative short-circuit so AIX cards are not swallowed by empty CRE search.

Cherry-pick conflict resolution had taken older fanout agent paths and dropped meta hit / normalize / picker wiring already on admin-panel-showcase.

Cherry-pick conflict resolution had overwritten admin's word-boundary intent fixes and sim-first router paths.

validate_cron_line was dropped from harvester schemas during the scheduled-OIE restack, breaking openapi-guardrail imports. Restore it (with SourceConfig) and exempt admin-only /rest/v1/api-keys routes from the public OpenAPI coverage check. Co-authored-by: Cursor <cursoragent@cursor.com>

…baselines The shipped default is pgvector, which CI (SQLite) does not have, so the factory tests failed closed before reaching what they test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Formatting only. Fold into the originating commits during the cleanup pass. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Jobs read their interval and enabled flag from admin runtime settings (DB), falling back to the built-in defaults. Adds GET /admin/schedule and PUT /admin/schedule/<job>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

…s follows the admin-set interval Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

… in the bearer test Several tests set NO_LOAD_GRAPH_DB (admin_authz_hardening, admin_runtime_settings) or the librarian promoted defaults (evaluate_harness) process-wide and never restored them, so later db_test gap-analysis and parser tests failed in CI. The bearer test hit Redis, which CI does not have. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

northdpole requested review from Pa04rth, paoga87 and robvanderveer as code owners October 9, 2026 08:37

northdpole force-pushed the admin-panel-showcase branch from e114292 to 4178ecf Compare October 11, 2026 23:04

northdpole force-pushed the feat/oie-rq-fanout branch from 723d697 to cba71ba Compare October 11, 2026 23:05

northdpole mentioned this pull request Oct 11, 2026

Fix OWASP agent probe gaps (chapters, people, membership, CRE facts) #1127

Open

4 tasks

northdpole force-pushed the admin-panel-showcase branch from 4178ecf to aad929a Compare October 12, 2026 07:05

northdpole force-pushed the feat/oie-rq-fanout branch from cba71ba to 2a1c75d Compare October 12, 2026 07:05

northdpole force-pushed the admin-panel-showcase branch from aad929a to 475945f Compare October 12, 2026 07:17

northdpole force-pushed the feat/oie-rq-fanout branch from 2a1c75d to 545522b Compare October 12, 2026 07:17

northdpole force-pushed the admin-panel-showcase branch from 475945f to 48e9d4f Compare October 12, 2026 07:19

northdpole force-pushed the feat/oie-rq-fanout branch 3 times, most recently from d68ddbc to 7aac2f5 Compare October 12, 2026 22:19

northdpole force-pushed the admin-panel-showcase branch from 48e9d4f to 30a2a40 Compare October 12, 2026 22:19

northdpole mentioned this pull request Oct 13, 2026

Admin panel redesign (CRE blues) on OIE RQ fanout tip #1146

Merged

4 tasks

northdpole force-pushed the feat/oie-rq-fanout branch from 404293e to 1d39a8b Compare October 14, 2026 08:42

northdpole force-pushed the admin-panel-showcase branch 6 times, most recently from e41d93b to a948365 Compare October 15, 2026 00:32

northdpole force-pushed the feat/oie-rq-fanout branch from b460148 to 4a876cc Compare October 15, 2026 00:34

northdpole force-pushed the admin-panel-showcase branch from a948365 to 3c45bf3 Compare October 15, 2026 12:16

northdpole force-pushed the feat/oie-rq-fanout branch from 4a876cc to 1371926 Compare October 15, 2026 12:17

northdpole force-pushed the admin-panel-showcase branch 2 times, most recently from 023a4be to 18ed280 Compare October 15, 2026 17:43

northdpole mentioned this pull request Oct 16, 2026

fix(mcp): use chatbot embedding retrieval for ask security facts #1155

Open

2 tasks done

northdpole and others added 27 commits October 20, 2026 09:41

Restrict staging login to an explicit email allowlist.

0738c73

Add LOGIN_ALLOWED_EMAILS so only Spyros/Rob/Rock can establish sessions on internet-facing staging.

Document homeserver staging behind Cloudflare Tunnel.

edb15fd

Record staging.opencre.org tunnel/DNS layout and LOGIN_ALLOWED_EMAILS env for the three OWASP operators.

Add dedicated Compose stack for staging.opencre.org.

0bec9b5

Ship deploy/staging with own Postgres/Redis/web/worker, skip-sync entrypoint knobs, and pathspec so the image boots behind the Cloudflare tunnel.

Add admin API tokens and JWT request auth.

74b7440

Mint/verify keys for allowlisted operators, audit log + migrations, and stop caching SPA bundles so admin deploys pick up new JS.

Pin cost-free Module C path in promoted defaults and staging.

107820a

Keep edition-remap live LLM off, safety batching on, and experimental fail-closed levers off unless an eval explicitly enables them.

Add Admin REST + UI to requeue/cancel failed OIE RQ jobs.

53583ab

Lets operators recover stalled Module C work via /admin/oie/rq without docker or redis CLI.

Redesign admin panel with CRE blues, env/.env diff, and review widgets.

f607a21

Ship sidebar shell, needs-attention/review-pulse dashboard, paginated staged-link review with title/content, workbench targets, and GET /admin/config/diff — all still gated by login + CRE_ALLOW_IMPORT.

Allow Bearer on Admin OIE REST and fix staging filing lookup.

939b70e

Staging monitors need API-key access to /admin/oie and related ops routes; also resolve CRE links by uuid or external_id and align ingest test asserts with skip_c1/max_repos=0. Co-authored-by: Cursor <cursoragent@cursor.com>

Add Admin active ingestions status with config, progress, and ETA.

d40ea13

Operators can see per-run start flags, live HI/KQ/RQ remaining, ETA, and stuck detection on the Pipeline tab (and a strip on Overview).

Fix Admin ingest ETA to use Module C remaining, not HI backlog.

f8b29a2

Remaining/ETA now track KQ+RQ only and use snap-to-snap drain rate so old ImportRuns stop inventing multi-day estimates.

chore(db): chain oie_run migration after mcp_grants

73b07ba

Main landed MCP v2 (f3a91c07d24b) on 967016ee10fa; keep a single Alembic head by parenting the stack's oie_run revision after mcp_grants.

fix(auth): admit admin API keys on MyOpenCRE after MCP rebase

3b46713

login_or_mcp_required now accepts allowlisted API JWTs as well as session/MCP identity, and API keys carry ALL_SCOPES for require_scopes.

feat(mcp): add fact-only ask tool for Cursor multi-call synthesis

9f1fa16

Route auto/security/owasp_meta to structured CRE and OWASP meta facts with optional MyOpenCRE standards filtering; no OpenCRE narrative LLM.

fix(mcp): only treat CRE-shaped ids as ask security hits

e69cad0

Avoid promoting Standard composite search keys into get_cre_by_id.

fix(mcp): run ask meta router in app context before CRE defer

a51d895

MCP stdio has no Flask request context, so the OWASP sim gate never saw pgvector. Also try meta/evidence before cre_normative short-circuit so AIX cards are not swallowed by empty CRE search.

fix(owasp-agent): restore sim-first router after fanout restack

103394e

Cherry-pick conflict resolution had taken older fanout agent paths and dropped meta hit / normalize / picker wiring already on admin-panel-showcase.

fix(owasp-agent): restore intent/sim-first agent after fanout restack

0fea7f6

Cherry-pick conflict resolution had overwritten admin's word-boundary intent fixes and sim-first router paths.

test(librarian): pin the in-memory retriever backend in factory test …

5544638

…baselines The shipped default is pgvector, which CI (SQLite) does not have, so the factory tests failed closed before reaching what they test. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

style: format files flagged by CI (black 24.4.2)

2c9d39c

Formatting only. Fold into the originating commits during the cleanup pass. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(admin): Schedules tab to change job intervals and switch jobs off

6ec609e

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

feat(oie-scheduler): owasp job runs daily by default; health stalenes…

26a507d

…s follows the admin-set interval Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

northdpole force-pushed the feat/oie-rq-fanout branch from 1183b05 to 26a507d Compare October 20, 2026 08:48

northdpole mentioned this pull request Oct 20, 2026

feat(feedback): thumbs up/down on answers (MCP + chat) with an answer_log table #1162

Draft

Navigation Menu

Repository navigation

feat(oie): RQ A→B→C fan-out and Admin OIE queue tab - #1142

feat(oie): RQ A→B→C fan-out and Admin OIE queue tab#1142
northdpole wants to merge 38 commits into
admin-panel-showcasefrom
feat/oie-rq-fanout

northdpole commented Oct 9, 2026

Uh oh!

coderabbitai Bot commented Oct 9, 2026 •

edited

Loading

Review skipped

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

northdpole commented Oct 9, 2026

Summary

Test plan

Uh oh!

coderabbitai Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coderabbitai Bot commented Oct 9, 2026 •

edited

Loading