Repository navigation
Conversation
Preserve in-progress storage, scheduling, claims, credentials and native API/CLI work. The orchestration backend remains incomplete and is not ready for delivery.
Add claim-scoped execution inputs and native snapshot drift validation. Incomplete orchestration backend checkpoint; worker launch and the remaining control-plane integration are not implemented yet. Targeted SQLite/PostgreSQL tests pass; full static readiness is pending.
Verify pinned inputs before start and expose retryable run reports and cancellation heartbeats over the authenticated API. Disable redirects for credentialed requests and cover the lifecycle against SQLite and PostgreSQL. Keep the orchestration PR explicitly incomplete pending actual launchers and the remaining backend scope.
Separate lease expiry, replica connections, terminal run fixtures and fault cases so each integration binary imports only the support it uses. Preserve SQLite and PostgreSQL transaction coverage.
Observe heartbeat loss in fenced, bounded transactions while retaining physical write claims. Recover through existing worker heartbeats and continue full batches without an idle delay.
Own worker heartbeats on a dedicated native runtime and connection pool. Propagate cancellation and revoked credentials without cancelling work on transient API failures; stop and join with the execution owner.
Move bounded client construction, credential redaction, URL validation and JSON response handling to the API owner so worker and location flows can share them. Keep independent heartbeat pools and worker-facing error categories.
Add privileged native administration with JSON results, bounded location credentials and atomic security audit. Detect lost result delivery without claiming committed mutations rolled back.
Keep the public Python API limited to authoring records and native delegation. Resolve selectors and validate schedule data in Rust before publication. Preserve stable shorthand trigger names across cron edits. Add the explicitly approved narrow Python facade boundary entries without increasing movable Python or native crossing budgets.
Collect exported jobs and nested factory containers natively, deduplicating aliases and terminating cyclic containers. Explicit empty declarations skip user module execution. Remove failed imports from the module cache so publication retries cannot reuse partial definitions.
Verify retained compiler baselines before worker startup and join native heartbeat work on interpreter-side close. Preserve claim identity and report retry keys without treating resource teardown as physical warehouse completion.
Restrict forward and reverse cron expansion and pure tick planning to the scheduler. Preserve scheduling bodies and all existing fault, DST and simulation coverage.
Expose owner-only cancellation with stable retry identity. Commit lifecycle facts, projections and security audit together while retaining active physical claims until termination.
Resolve manual requests against the current published job inside one write transaction. Preserve the accepted snapshot and execution inputs across retries after redeployment, and commit queue publication with the trusted owner audit.
Compile in the project interpreter and retain native snapshots through publication. Keep JSON results separate from progress and distinguish uncertain acknowledgements from failed result delivery.
Reconstruct current publications, jobs, run outcomes and structural catalog from retained sources under serialized writer transactions. Keep security, operational ownership and source facts intact, with paged causal replay and atomic failure handling. Exercise SQLite and PostgreSQL corruption repair, protected-row equality, idempotence, rollback and concurrent readers/writers. Preserve recorded times across backward clock steps. Keep the incomplete full rebuild command unexposed.
sqb server --statistics-retention-days N (default 90) sets how long hourly warehouse statistics are kept. Person reads hide samples older than the horizon, and the existing bounded cleanup deletes them with the same cutoff. The value is validated before the control store is opened.
kvlonge
force-pushed
the
feat/orchestration-backend
branch
from
November 3, 2026 23:42
5b44eff to
ef69e50
Compare
sqb server --audit-retention-days N (default 400) sets how long security audit records are kept. A record older than the horizon is deleted only when every configured audit webhook has received it, no retained run's retry lookup depends on it, and it is not the newest record, so storage IDs are never reused. Records kept past the horizon are reported on stderr as "retained past horizon: N" whenever the count changes.
The scheduler leader deletes skipped and covered ticks older than 30 days in fenced batches, keeping each trigger's latest tick and every tick a run points at. Evaluation never starts before the same horizon, so a new schedule, a resumed trigger or a cursor reset catches up from 30 days ago at most and deleted ticks are never recreated. Additive indexes keep the cleanup bounded.
sqb server --snapshot-retention-count N (default 10) keeps the newest N published snapshots per location and target. An older snapshot is deleted only when no run, job, trigger, trigger change, location request or latest trigger progress fact references it; superseded progress facts, node definitions, SQL text and documents that nothing else shares go with it. Collection runs periodically in small batches and is safe to repeat.
Published custom roles narrow at once. A widened definition takes effect only after a different administrator approves the activation request, or after a sole administrator explicitly confirms a self-approval. Approvals are recorded unsigned and audited.
A cleanup pass now deletes at most ten batches. A longer backlog resumes on the next scheduler cycle, after the lease renews, so a large backlog of old covered ticks cannot outlast the scheduler lease.
Worker and recovery fixtures used an annual schedule starting in 2000 and expected an immediate catch-up run. The scheduler no longer creates ticks older than the 30-day tick horizon, so the latest annual occurrence was never ticked and each test waited out its 30-second run deadline. A daily schedule keeps one catch-up run inside the horizon.
With catch_up="latest", a trigger always gets exactly one tick for its latest occurrence, even when that occurrence is older than the 30-day tick horizon. A yearly schedule published with a start years back therefore catches up once, as before the horizon, instead of waiting for its next occurrence. The horizon still bounds scheduler work and still stops re-deciding old ticks: nothing older than the latest occurrence is created, an occurrence that already has a tick is not decided again after a cursor reset, and bounded catch-up keeps the horizon clamp. The worker fixtures keep their daily schedule.
…r occurrence A never-ticked trigger starts searching from its sub-second start time, which leaked the fraction into the first tick (00:00:00.999999). Forward and backward cron searches now start from the whole second. A cursor left just past an occurrence still resumes at the next occurrence, so state written before this fix does not re-tick.
Two schedulers on separate store handles run seeded random steps of crash-restarts, lease lapses, injected transaction faults and single or concurrent cycles on SQLite and PostgreSQL. After recovery every due minute has exactly one tick, one queued run and one request event, every frozen leader's write is fenced, and the database rejects a duplicate tick. The pure simulation now expands real cron schedules across both New York transitions under random crashes and backward and forward clock jumps, and compares the ticks with occurrences derived day by day from wall times.
A real scheduler loop queues each calendar time within 1.5 s of it falling due, and after a stopped leader's lease lapses a second scheduler queues the overdue work within 1.5 s, on SQLite and PostgreSQL.
Measured tick-to-queued and post-lease catch-up stay under 20 ms. A scheduler that stops waking at the next due time and falls back to its one-second poll queues ticks 0.7-0.9 s late; 1.5 s missed that, 500 ms catches it.
The randomized two-scheduler run now drains a two-hour backlog, so armed faults land while ticks are still being written, and it asserts that every crash point in the tick transaction fires across the seeds run (five by default). The failover check starts the second scheduler before the old lease lapses, so a standby that waits too long before retrying leadership fails; its budget is 1.5 s to allow the one-second retry.
…sses A store left by the build before trigger progress, with that build's fractional ticks and cursor and its lease still live, upgrades in place on SQLite and PostgreSQL. The upgraded scheduler waits for the old lease, resumes every later minute once, fences the old holder, and an operator repair succeeds afterwards. Installed sqb scheduler processes are SIGKILLed at seeded random moments while two of them drain a backlog; a survivor then finishes it, and every due minute of every trigger has exactly one tick, one queued run and one request event.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Native orchestration, Python workers, browser CLI authentication.
Changes
--statistics-retention-days(90).--audit-retention-days(400) drops audit rows all webhooks have and no retained retry uses; skipped/covered ticks expire after 30 days (latest/run-linked kept); none older is created except onelatestcatch-up tick;--snapshot-retention-count(10) keeps newest/referenced snapshots.Verification