Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
183 commits
Select commit Hold shift + click to select a range
1d53f7e
docs: artifact plane design spec
juicycleff Aug 11, 2026
8370d0f
docs: artifact plane implementation plan
juicycleff Aug 12, 2026
af03d47
feat(id): add artifact TypeID prefix
juicycleff Aug 12, 2026
18007db
feat(artifact): add entity types, errors, and Store interface
juicycleff Aug 12, 2026
a4f5f05
feat(artifact): add store conformance suite and memory implementation
juicycleff Aug 12, 2026
deccd59
feat(artifact): add postgres store implementation
juicycleff Aug 12, 2026
412b064
feat(artifact): add sqlite store implementation
juicycleff Aug 12, 2026
58567c2
feat(artifact): add mongo store implementation
juicycleff Aug 12, 2026
d06d8e0
feat(artifact): add redis store implementation
juicycleff Aug 12, 2026
f178cc6
feat(artifact): add Backend interface and in-memory test double
juicycleff Aug 12, 2026
1cea6c1
feat(artifact): add Service for register, create, commit, and link
juicycleff Aug 12, 2026
9e375db
feat(artifact): add Trove backend adapter
juicycleff Aug 12, 2026
f59087b
feat(artifact): add content-addressed staging cache
juicycleff Aug 12, 2026
fd7471c
feat(artifact): add input declarations, accessor, and staging middleware
juicycleff Aug 12, 2026
aea17df
feat(artifact): wire the artifact plane into the engine
juicycleff Aug 12, 2026
8320bb9
feat(artifact): resolve the artifact backend from Forge DI
juicycleff Aug 12, 2026
8c91af5
feat(artifact): add leader-only two-phase lifecycle sweeper
juicycleff Aug 12, 2026
85897ab
docs: document the artifact plane
juicycleff Aug 12, 2026
52f15ee
test(postgres): repair the integration test harness
juicycleff Aug 12, 2026
0a96443
fix(artifact): use $N placeholders in postgres raw queries
juicycleff Aug 12, 2026
32d2af5
fix(postgres): repair leader election scan
juicycleff Aug 12, 2026
22e258d
test(postgres): stop dequeue test racing the database clock
juicycleff Aug 12, 2026
f22057b
refactor(postgres): rename stale dispatch/bun error prefix
juicycleff Aug 12, 2026
52911fa
refactor(postgres): extract error prefix into a package-level const
juicycleff Aug 12, 2026
71c04ac
chore(deps): drop bun modules left over from the grove migration
juicycleff Aug 12, 2026
8364628
docs: execution isolation design spec
juicycleff Aug 12, 2026
633ede2
docs: resource model and resource-aware scheduling design spec
juicycleff Aug 12, 2026
422460c
docs: reconcile execution isolation spec with tracks A and B
juicycleff Aug 12, 2026
caf2973
docs: implementation plan for execution isolation phase 1
juicycleff Aug 12, 2026
d6e10bf
docs: declare execution policy via job.WithExecution
juicycleff Aug 12, 2026
5506d30
chore: stop tracking superpowers working artifacts
juicycleff Aug 12, 2026
0869d52
test(artifact): scope store-suite setup to the subtest T
juicycleff Aug 12, 2026
9d26fc9
feat(job): add the lease model and the LeaseStore capability
juicycleff Aug 12, 2026
73a73f9
feat(store): add the lease conformance suite and memory backend
juicycleff Aug 12, 2026
5f016f2
chore: ignore the .superpowers scratch directory
juicycleff Aug 12, 2026
3bd1888
fix(storetest): add a concurrent reclaim-exclusivity case
juicycleff Aug 12, 2026
2028da6
feat(postgres): implement job.LeaseStore
juicycleff Aug 12, 2026
2a0155f
feat(resource): Set vector with canonical-unit arithmetic
juicycleff Aug 12, 2026
17da0e2
feat(resource): Spec and enqueue-time resolution precedence
juicycleff Aug 12, 2026
309604c
feat(sqlite): implement job.LeaseStore
juicycleff Aug 12, 2026
878199e
feat(resource): Manager generalizing the cache budget to N keys
juicycleff Aug 12, 2026
1169eff
feat(mongo): implement job.LeaseStore
juicycleff Aug 12, 2026
e9870e4
fix(resource): return reclaimed units via lease release, not a credit
juicycleff Aug 12, 2026
4aca353
feat(resource): capacity detection preferring cgroup limits
juicycleff Aug 12, 2026
6644972
fix(mongo): reclaim index on the path Migrate() runs, limit guards, g…
juicycleff Aug 12, 2026
0e0c924
test(resource): resourcetest fakes mirroring artifacttest
juicycleff Aug 12, 2026
ca07fd4
fix(resource): release already-acquired leases on FakeReclaimer const…
juicycleff Aug 12, 2026
946fe08
feat(job): resource declaration options
juicycleff Aug 12, 2026
f298688
feat(redis): implement job.LeaseStore
juicycleff Aug 12, 2026
5afd3c6
feat(cluster): advertise per-worker resource capacity
juicycleff Aug 12, 2026
67ae0c4
feat(engine): resolve resource requirements at enqueue
juicycleff Aug 12, 2026
deeef46
fix(engine): pin every resolution source and drop crashed workers
juicycleff Aug 12, 2026
2b085cb
fix(store/memory): deep-copy jobs instead of shallow struct copy
juicycleff Aug 12, 2026
3c6edd8
fix(store/memory): deep-copy leased jobs in lease.go too
juicycleff Aug 12, 2026
183abf9
feat(store/postgres): job resource columns and covering index
juicycleff Aug 12, 2026
862abb1
test(store/postgres): cover encodeCustomKeys and encodeSet/decodeSet …
juicycleff Aug 12, 2026
2bc7a75
feat(store/sqlite): job resource columns and index
juicycleff Aug 12, 2026
9ee0b85
refactor(resource): extract EncodeSet/DecodeSet/EncodeCustomKeys
juicycleff Aug 12, 2026
73d27a0
feat(store): persist job resource fields on redis and mongo
juicycleff Aug 12, 2026
65dcde2
docs(store/mongo): correct and relocate the null-vs-absent resource trap
juicycleff Aug 12, 2026
3f8f357
feat(job)!: filter dequeue by resource fit via DequeueOpts
juicycleff Aug 12, 2026
b0f0d0a
refactor(store): settle the dequeue contract on the coordinator's rul…
juicycleff Aug 12, 2026
0c22fb2
fix(retry): fail permanently failed jobs immediately instead of retrying
juicycleff Aug 12, 2026
640b81b
feat(exec): add the execution policy type
juicycleff Aug 12, 2026
15a638d
test(store): close the ordering and locality holes the review found
juicycleff Aug 12, 2026
7be78dc
feat(exec): add execution status, result, and error types
juicycleff Aug 12, 2026
3c29bfc
test(exec): cover the exit-code branch of Error.Error()
juicycleff Aug 12, 2026
9b06e96
fix(redis): stop cjson.encode from corrupting large lease durations
juicycleff Aug 12, 2026
0bb8e0d
feat(exec): add the execution request and registry fingerprint
juicycleff Aug 12, 2026
98d4c52
feat(store/memory): widen DequeueJobs to the resource-aware dequeue c…
juicycleff Aug 12, 2026
8d1278f
feat(exec): add the Executor interface and executor registry
juicycleff Aug 12, 2026
aaaf63d
test(exec): add deterministic tie-break test for executor selection
juicycleff Aug 12, 2026
5d772d8
feat(job): add the Registrable seam and execution policy
juicycleff Aug 12, 2026
860506d
feat(exec): add the in-process executor
juicycleff Aug 12, 2026
564ca6c
feat(exec): add the executor conformance suite
juicycleff Aug 12, 2026
c2f65e1
feat(store/postgres): widen DequeueJobs to the resource-aware dequeue…
juicycleff Aug 12, 2026
fb23299
refactor(worker): rename Executor to Runner and delegate to exec.Exec…
juicycleff Aug 12, 2026
6400d66
fix(worker): treat raw exec.Executor.Run errors as launch failures
juicycleff Aug 12, 2026
a3eab46
feat(engine): wire the executor registry into registration and execution
juicycleff Aug 12, 2026
bb9a8c7
feat(store/mongo): widen DequeueJobs to the resource-aware dequeue co…
juicycleff Aug 12, 2026
8f4f4b2
docs(exec): document the isolation ladder and policy declaration
juicycleff Aug 12, 2026
7da5c08
feat(store/redis): widen DequeueJobs to the resource-aware dequeue co…
juicycleff Aug 12, 2026
99fc0dd
fix(exec): keep the handler's error whole across the executor boundary
juicycleff Aug 12, 2026
6a0bed4
fix(worker): honour permanence from either rung and bound launch retries
juicycleff Aug 12, 2026
661639f
fix(worker,engine): reclaim executors at pool start, close them at en…
juicycleff Aug 12, 2026
e5d9be0
test(exec/exectest): hold claimed capabilities to the advertised level
juicycleff Aug 12, 2026
8e7b2e2
test(engine): drive permanence and executor routing through engine.Build
juicycleff Aug 12, 2026
3e61e80
docs(exec): define SendEmail in the RegisterAll example
juicycleff Aug 12, 2026
41d5432
feat(sqlite): make DequeueJobs resource-aware
juicycleff Aug 12, 2026
ebf9152
test(storetest): pin locality as the tiebreak a tight limit cannot by…
juicycleff Aug 12, 2026
832222d
feat(worker): admit jobs against real capacity
juicycleff Aug 12, 2026
6bfaef4
fix(worker): make the disk budget redeemable and pace refused batches
juicycleff Aug 12, 2026
69d5624
fix(worker): hold cadence on refused batches, share one admission budget
juicycleff Aug 12, 2026
110592b
feat(cache): admit staged bytes against the shared resource manager
juicycleff Aug 13, 2026
9fcff57
fix(resource): make the reclaim window survivable, and eviction O(1)
juicycleff Aug 13, 2026
7cf4ac4
feat(extension,engine): share one admission ledger, end to end
juicycleff Aug 13, 2026
1075e3d
docs(resource): add the mixed-workload example and README section
juicycleff Aug 13, 2026
eb34fa5
fix(extension): close the disk-config gaps, and race the example corr…
juicycleff Aug 13, 2026
af1ce58
fix(redis): bound the dequeue scan when the caller filters nothing
juicycleff Aug 13, 2026
a69b631
fix(engine): stop deriving the fleet ceiling from the local ledger
juicycleff Aug 13, 2026
6d4db8a
fix(postgres,sqlite): make the resource migration survivable on a liv…
juicycleff Aug 13, 2026
66cf785
fix(resource,store): correct the docs that were wrong and the two def…
juicycleff Aug 13, 2026
653c791
docs: fix four doc-comment inaccuracies found in final review
juicycleff Aug 13, 2026
0057b63
fix(engine): copy declared LeaseTTL to job at enqueue
juicycleff Aug 13, 2026
5b01c86
fix(worker): stop Pool.Start blocking on a slow executor Reclaim sweep
juicycleff Aug 13, 2026
1c655ee
feat(exec/wire): add the length-prefixed boundary codec
juicycleff Aug 13, 2026
0d8abe2
fix(redis): re-index a job that UpdateJob returns to a runnable state
juicycleff Aug 13, 2026
44b6cb2
fix(engine): remove out-of-scope executor lifecycle changes
juicycleff Aug 13, 2026
b67124b
Revert "fix(engine): remove out-of-scope executor lifecycle changes"
juicycleff Aug 13, 2026
8d10bba
feat(exec/shim): add a directory-backed artifact backend
juicycleff Aug 13, 2026
8a8df57
fix(exec/shim): reject keys that resolve to the output root itself
juicycleff Aug 13, 2026
5e13c7a
feat(job,store)!: fold the lease grant into DequeueJobs
juicycleff Aug 13, 2026
d210402
feat(exec/shim): add the child entrypoint
juicycleff Aug 13, 2026
441651d
fix(job,store): correct the atomicity rationale and pin the grant in SQL
juicycleff Aug 13, 2026
1776c9a
fix(exec/shim): derive Main's exit code from the Result, not just Run…
juicycleff Aug 13, 2026
c626a99
feat(worker): pool takes job leases and cancels on ErrLeaseLost
juicycleff Aug 13, 2026
eac3809
feat(worker): decouple the reaper's scan cadence from lease reclamation
juicycleff Aug 13, 2026
65d35e4
test(worker): prove the pool cancels on ErrLeaseLost, not just the store
juicycleff Aug 13, 2026
c1d0555
refactor(exec/shim): replace the composite store with a minimal one
juicycleff Aug 13, 2026
245aab6
fix(postgres,sqlite): make the lease migration survivable on a live f…
juicycleff Aug 13, 2026
2908c1c
fix(job,engine): make a definition's WithLeaseTTL actually reach the job
juicycleff Aug 13, 2026
fa2a8cb
feat(exec/subprocess): run handlers in a separate process
juicycleff Aug 13, 2026
8d4b8db
docs(job): correct lease epoch fencing scope in comments
juicycleff Aug 14, 2026
f41bc84
fix(exec/subprocess): bound the post-wait drain and interrupt a block…
juicycleff Aug 14, 2026
a87069b
fix(exec/subprocess): resolve the deadline/reap race in waitLoop, not…
juicycleff Aug 14, 2026
a34b527
fix(exec/subprocess): kill on any probe error, not just liveness, and…
juicycleff Aug 14, 2026
0ca7709
fix(job,store): fence terminal job writes on the lease epoch
juicycleff Aug 14, 2026
cdcba5b
test(storetest): hold every backend to the lease-fenced write
juicycleff Aug 14, 2026
d3b5707
fix(worker): route terminal writes through the lease fence
juicycleff Aug 14, 2026
0b7b8fd
fix(redis): restore queue-index maintenance on the fenced write
juicycleff Aug 14, 2026
3732d58
fix(mongo): drop the non-idempotent retry from the fenced update
juicycleff Aug 14, 2026
bc75acd
feat(exec/subprocess): apply rlimits and a dedicated UID
juicycleff Aug 14, 2026
d78821c
fix(exec/subprocess,exec/shim): fix cross-platform rlimit application…
juicycleff Aug 14, 2026
3123624
fix(exec/subprocess,exec/shim): make WithStrictRlimits catch unverifi…
juicycleff Aug 14, 2026
16436cd
feat(exec/subprocess): add the kill ladder
juicycleff Aug 14, 2026
3f2a9be
fix(exec/subprocess): decide the kill ladder's escalation on the grou…
juicycleff Aug 14, 2026
1237cd2
test(exec/subprocess): pass the conformance suite with enforcement on
juicycleff Aug 14, 2026
19f83b0
feat(worker): give out-of-process rungs a scratch dir and commit thei…
juicycleff Aug 14, 2026
a814739
fix(worker,artifact): close the gaps a review found in output committing
juicycleff Aug 14, 2026
99e34b3
fix(worker,artifact): correct the gate's claim, fix rollback, harden …
juicycleff Aug 14, 2026
50ae4c1
fix(worker): correct two stale comments the round-3 fix left behind
juicycleff Aug 14, 2026
d5fe9c1
feat(engine): configure the subprocess rung from YAML
juicycleff Aug 14, 2026
4ddba3f
docs(job): document the non-positive limit split on ReclaimExpiredLeases
juicycleff Aug 14, 2026
4889430
fix(engine,extension,exec,worker): close four review gaps in the subp…
juicycleff Aug 14, 2026
b036e65
docs(exec): document the subprocess rung and the phase gate
juicycleff Aug 14, 2026
723d140
fix(docs,exec): close review gaps in the subprocess-rung documentation
juicycleff Aug 14, 2026
107dfa1
fix(exec,extension): enforce the dedicated uid by default, not just a…
juicycleff Aug 14, 2026
9f514ea
docs(execution-isolation): make the uid-boundary claims match the enf…
juicycleff Aug 14, 2026
6d49302
test(exec/shim): replace the infrastructure denylist with an allowlist
juicycleff Aug 14, 2026
3e8dcc9
test(exec/subprocess): pin buildEnv against inheriting the parent env…
juicycleff Aug 14, 2026
63da67b
fix(exec/subprocess): stop claiming a killed handler's outputs survive
juicycleff Aug 14, 2026
361cbb7
fix(exec/subprocess): stop crediting a comment for an assertion that …
juicycleff Aug 14, 2026
98bbaaf
fix(extension): merge Execution config so WithConfig survives a YAML …
juicycleff Aug 14, 2026
a2ba0e6
fix(docs): fix second non-compiling engine.Build snippet in execution…
juicycleff Aug 14, 2026
4aa89ee
fix(extension): correct three stale claims in config.go's doc comments
juicycleff Aug 14, 2026
976578d
fix(worker,engine,artifact): widen the scratch-dir sweep and fix six …
juicycleff Aug 14, 2026
4130db7
fix(exec): correct comment drift on env inheritance, hashing, imports…
juicycleff Aug 14, 2026
b944b1d
Unify ReclaimExpiredLeases' non-positive-limit behavior across backends
juicycleff Aug 15, 2026
efe2093
fix(worker): fence the launch-failure requeue against a lost lease
juicycleff Aug 15, 2026
6e687f2
test(engine): stop reading the reclaim count before Pool.Stop joins it
juicycleff Aug 15, 2026
698a7eb
fix(mongo,redis): adopt jobs left running by a pre-lease build
juicycleff Aug 15, 2026
2ac224a
fix(exec,worker): correct two stale comments the review caught
juicycleff Aug 15, 2026
1a473f1
fix(worker): clear the job-assignment fields on every requeue path
juicycleff Aug 15, 2026
aeeb83b
fix(engine): police the reclamation window that actually governs, not…
juicycleff Aug 15, 2026
ec30106
fix(artifact): make FindLinkByName's attempt tie-break deterministic
juicycleff Aug 15, 2026
96fea64
test(storetest): cover Budget and a lease grant in one DequeueOpts
juicycleff Aug 15, 2026
3977603
fix(exec,worker): enforce job.WithResourceLimits at the subprocess rung
juicycleff Aug 15, 2026
4fcdc13
test(mongo,redis): restore a clean lint gate on the lease tests
juicycleff Aug 15, 2026
4bebf56
fix(cache): unlink an evicted file under the lock that publishes one
juicycleff Aug 15, 2026
f256bf8
fix(store): reclaim unleased running jobs on every backend, not by ba…
juicycleff Aug 17, 2026
c157c6a
fix(memory): adopt unleased running jobs here too, matching the others
juicycleff Aug 17, 2026
9a8a124
fix(api): clear lease ownership when retrying a failed job
juicycleff Aug 17, 2026
71fad99
docs(worker,store): correct two reaper comments the lease work invali…
juicycleff Aug 17, 2026
6579491
fix(dlq): carry the fields a replayed job needs to run like the original
juicycleff Aug 17, 2026
93840e8
fix(sqlite,storetest): jitter the busy retry, decouple a suite case f…
juicycleff Aug 17, 2026
338e587
test(storetest): move the non-positive-limit case into the shared lea…
juicycleff Aug 17, 2026
c2a79c8
fix(dispatch): make Dispatcher.Stop idempotent
juicycleff Aug 17, 2026
f633d90
refactor(job)!: remove the unused lease surface
juicycleff Aug 18, 2026
f99dbc4
Merge origin/main into track-c-execution-isolation
juicycleff Aug 18, 2026
bf12931
chore: updated deps
juicycleff Aug 21, 2026
e02d491
chore: updated trove
juicycleff Aug 21, 2026
b6c4926
chore: bumped grove ext
juicycleff Aug 21, 2026
0dd30ee
chore: maked lint fixes
juicycleff Aug 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Superpowers working artifacts — design specs and implementation plans.
# These are scratch for the development process, not part of the library.
docs/superpowers/
.superpowers/

# ─── Local agent config ─────────────────────────────────────
.claude/
59 changes: 59 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ Dispatch is a library — not a service. Import it, configure a store, and regis
- **Extension hooks** — Opt-in lifecycle interfaces for every job, workflow, cron, and shutdown event
- **OpenTelemetry** — Built-in metrics and tracing via the `observability` and `middleware` packages
- **Relay integration** — Emit typed webhook events at every lifecycle point via `relay_hook`
- **Artifact plane** — Track gigabyte-scale job inputs and outputs in object storage, staged to a content-addressed local cache
- **Resource model** — Size jobs by what they actually need and admit them against detected worker capacity, instead of counting identical slots
- **Pluggable storage** — Memory, PostgreSQL (pgx/v5), Grove ORM, SQLite, Redis

## Quick Start
Expand Down Expand Up @@ -77,13 +79,70 @@ func main() {
}
```

## Resource-Aware Scheduling

A worker slot is a promise that one job fits. That works while every job is the same size, and stops working the moment one definition serves both a 2 MB input and a 2 GB one: `concurrency: 4` says the worker may run four jobs, never how big they are, so a slot-counting pool starts four of the large ones on a box sized for the small ones and the kernel decides which of them dies. Adding a dedicated queue per size is the usual workaround, and it trades an OOM for a fleet that is idle in one queue and backed up in another.

The `resource` package replaces the slot with a vector. A job declares what it needs — or computes it at enqueue from its input size — the worker detects what it has (cgroup-first, so a container reports its quota rather than the host's cores), and admission is a comparison. A job that does not fit stays pending instead of being started next to work already using the memory.

Minimal wiring, in one process:

```go
capacity := resource.Detect(resource.CapacityConfig{DiskBytes: 20 << 30})
resources := resource.NewManager(capacity)

// The SAME manager goes to both. The staging cache holds a lease per
// cached entry and reclaims disk on demand; the pool offers disk at
// dequeue as free plus what the cache could evict. Give the cache its
// own manager and that second half is always zero.
staging, _ := cache.New(cacheDir, backend, cache.WithManager(resources))

eng, _ := engine.Build(d,
engine.WithArtifacts(artifacts, staging),
engine.WithResourceManager(resources),
)
```

Sizing one definition from its input:

```go
job.NewDefinition("render", handler,
job.WithArtifactInputs(artifact.Input("scene", artifact.Required)),
job.WithResourceFunc(func(_ context.Context, r resource.Request) (resource.Set, error) {
return resource.MemoryBytes(32<<20 + r.InputBytes*16), nil
}),
)
```

The function runs once, in the enqueuing process, and the result is written to the job row — nothing evaluates user code on the scheduling path, and two workers can never disagree about how big a job is.

Under Forge, the same thing is configuration:

```yaml
extensions:
dispatch:
resources:
enabled: true
cpu_overcommit: 1.0 # CPU is compressible; there is no memory equivalent
memory_fraction: 0.8 # leave the rest for the runtime and the page cache
explicit: # overrides detection, and the only way to declare
gpu: 4000 # a custom resource — nothing detects an FPGA
fpga: 2
```

Leave it out and the runtime behaves as it did before: no ledger is built, the pool dequeues unbounded, every store backend skips its fit predicate, and the staging cache keeps the private disk budget it has always had. The schema is the exception — the resource columns are added by migration whether or not you turn any of this on, because a job row has to be readable by every worker in a mixed-version fleet.

Runnable end to end in [`_examples/resources`](./_examples/resources).

## Package Index

| Package | Description |
|---------|-------------|
| `dispatch` | Root — `Dispatcher`, `Config`, options, errors, `Entity` base type |
| `engine` | Wires all subsystems; `Build`, `Register`, `Enqueue`, `RegisterWorkflow`, `RegisterCron` |
| `job` | `Job` entity, `State` machine, `Definition[T]`, `Registry` |
| `artifact` | Tracked object storage — declared inputs, imperative outputs, staging cache, lifecycle sweeping |
| `resource` | Resource vectors, capacity detection, the shared admission ledger and its reclaimers |
| `workflow` | `Definition[T]`, `Run`, `State`, step checkpointing |
| `cron` | `Entry`, `Scheduler`, distributed leader-elected cron |
| `dlq` | `Entry`, `Service` — list, replay, purge |
Expand Down
Loading
Loading