Skip to content

feat(renderer-service): self-deployable git-driven renderer (RunPod + Docker) - #31

Open
okanasl wants to merge 20 commits into
mainfrom
feat/renderer-self-deploy
Open

feat(renderer-service): self-deployable git-driven renderer (RunPod + Docker)#31
okanasl wants to merge 20 commits into
mainfrom
feat/renderer-self-deploy

Conversation

@okanasl

@okanasl okanasl commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Makes apps/renderer-service self-deployable. You send a job naming a repo, a ref and an exported composition. The worker:

  1. Checks out the commit, including LFS assets.
  2. Installs the project's dependencies, including its own framefields version.
  3. Renders in-process on a warm GPU.
  4. Verifies the output and uploads it.

It runs on RunPod (serverless or pod), with plain Docker, or as a CLI. The full spec is in apps/renderer-service/PLAN.md; the setup guide is in its README.md.

Worker (apps/renderer-service)

  • Job API v2: zod schema, stable error codes with a retryable flag, /v2/jobs, progress over SSE, cancel, bearer auth. Jobs run one at a time.
  • Git:
    • One mirror per worker; each ref is resolved to a commit SHA before checkout.
    • Reusable workspaces with sparse checkout. Submodules are checked against the allowlist.
    • LFS downloads check free disk space first.
    • Tokens and SSH keys work with GitHub, GitLab, Bitbucket, Gitea and generic hosts. SSH host keys are pinned.
  • Install: pnpm, npm or yarn with a frozen lockfile, skipped when the lockfile hasn't changed. Each project brings its own framefields; there's no engine in the image.
  • Engine reuse: a worker loads one framefields version. A job needing a different version fails with retryable ENGINE_SWITCH, and the worker restarts.
  • In-process rendering safeguards:
    • Each job runs with its own cwd, env and argv, and process.exit is trapped.
    • A watchdog catches a blocked event loop.
    • A crash journal reports the in-flight job after a restart.
    • The worker recycles itself after N jobs, high memory, stray errors or leaked handles.
  • Outputs:
    • Checked before upload: SHA-256, size, and an ffprobe frame count.
    • Destinations: S3 (multipart), presigned URL, local directory, or inline.
    • An idempotency key skips re-rendering on retries.
  • Caches: an LRU janitor evicts workspaces, LFS objects, mirrors and package stores as needed.
  • GPU (FF_GPU):
    • auto (default): a machine with a GPU must render on it; a machine without one renders on the CPU (lavapipe).
    • require (RunPod templates): GPU only.
    • cpu: CPU only.
    • The adapter WebGPU picks is checked after the engine loads. A GPU machine that ended up on the CPU fails with GPU_UNAVAILABLE, and RunPod replaces the worker.
  • Deployment:
    • Image: Debian trixie (WebGPU's prebuilt binaries need glibc 2.38+), with tini as PID 1.
    • RunPod: a thin Python shim, templates, and a template-creation script.
    • docker-compose with MinIO, plus a GHCR image workflow.

framefields (minor changeset → 2.2.0)

  • renderVideo accepts signal and onProgress; resetRenderCaches() clears caches between jobs in one process.
  • Video encoders: H.264/HEVC failed on linux-arm64 because FFmpeg's first registered encoder there is h264_rkmpp. They would also fail on GPUs without NVENC, such as A100/H100. Now libx264, libx265, libvpx and SVT-AV1 are the defaults, and a hardware encoder is used only after a one-frame test passes. FRAMEFIELDS_HW_CODECS=off forces software.
  • WebGPU falls back to a software adapter when there's no GPU. The new describeGpu() reports which adapter is in use.

Testing

  • Unit and integration tests pass: renderer-service 151 tests, framefields 459, renderer 29. The 4 GPU tests (FF_GPU_TESTS=1) pass on macOS Metal.
  • In the arm64 Docker image (no GPU):
    • An H.264 job rendered all 48 frames on lavapipe.
    • A project pinned to the published framefields 2.1.3 rendered VP9.
    • FF_GPU=require fails the health check with GPU_UNAVAILABLE.
  • Not yet tested: an amd64 image build (this PR's CI does that) and a RunPod run on a real NVIDIA GPU.

🤖 Generated with Claude Code

okanasl and others added 20 commits October 14, 2026 20:43
@okanasl @claude
feat(renderer-service): self-deploy groundwork — job API v2 and git c…
9a4e717 
…heckout

Phases 0–2 of apps/renderer-service/PLAN.md.

- Remove the inline virtualMedia path (job-processor, render-upload, cli,
  /run and /status), the unused DI container, and BASE_URL/R2_* config.
- Fix the old gatewai-renderer paths; the entrypoint starts the Node server
  once and runpod_worker.py is now a thin localhost client.
- Config: one validated schema reporting every problem at once; secrets are
  scrubbed from process.env before repo code can run.
- Job API v2: zod schema, allowlist/bucket/env checks, stable error codes
  with retryable flags, redaction, single-slot queue, job store with TTL,
  /v2/jobs with bearer auth, SSE progress and cancel.
- Git checkout: per-worker mirror cache with corruption re-clone, ref
  resolution pinned to one SHA, reusable workspace slots, sparse checkout,
  allowlisted submodules, LFS pull with disk preflight and pointer
  postflight, unsupported-filter detection, per-host credential helper and
  GitHub/GitLab/Bitbucket/Gitea error mapping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): dependency install step (phase 3)
605d1da 
- Detect the install root and package manager (packageManager field or
  lockfile; pnpm, npm, yarn classic/berry) and install through corepack with
  a frozen lockfile and shared stores on the cache volume.
- Skip the install when the project only needs framefields plus tooling;
  skip it on warm workspaces whose install stamp still matches.
- Check the declared framefields range against the bundled engine
  (FF_ENGINE_POLICY) and warn on engines.node mismatches.
- Map install failures to INSTALL_FAILED / INSTALL_TIMEOUT / DISK_FULL /
  LOCKFILE_MISSING with hints (stale lockfile, registry auth, native build);
  retry once on registry outages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer): abort, progress and per-job cache reset (phase 4)
588f0b1 
- renderVideo takes `signal` (checked between frames and before the audio
  mix; rejects with signal.reason) and `onProgress` (per frame, then audio).
  framefields' RenderVideoOptions passes both through.
- resetRenderCaches() drops path-keyed caches (media inputs/decoders,
  textures, LUTs, fonts) while keeping the GPU device warm; InputStore gains
  clear(). A long-lived worker needs this because the next job's checkout
  can hold different bytes at the same path.
- GPU test (FF_GPU_TESTS=1) covering progress, abort, and the stale-path
  case before/after the reset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): render repo code in-process (phase 5)
0cc0211 
- Engine alias (module.registerHooks): framefields / @framefields/* imports
  and requires from a workspace resolve to the worker's own ESM build, so
  repo code shares the warm GPU stack. Workspace sources always load as ESM;
  in CJS mode tsx would transpile a second engine copy.
- Per-job loading: tsx register() with the project's tsconfig and a native
  import of the entry tagged ?ff-job=<id>, propagated to workspace modules
  only, so changed code at the same path is never served from cache.
- Export shapes: a Composition, a record of named compositions (with
  target.select), or { outputs: [...] } render plans; video/audio, frames,
  frame grid and LUT outputs.
- Job sandbox: cwd, env, argv and a process.exit trap, restored after each
  job; stray rejections fail the active job instead of the worker.
- Stall detection, abort grace period, event-loop watchdog (helper thread
  kills a blocked process), crash journal reported after restart, and
  recycling by job count / heap / stray errors / leaked handles / GPU loss.
  The RunPod shim sets refresh_worker while the worker drains.
- Tests: real-Node loader harness (alias identity incl. CJS packages, module
  freshness, error classes, exit trap), watchdog kill, and a GPU harness
  rendering six jobs back to back (FF_GPU_TESTS=1).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): verify and deliver outputs (phase 6)
8150448 
- Verify each file: exists, non-empty, within FF_OUTPUT_MAX_BYTES, SHA-256,
  and ffprobe for media with the engine's frame count (±1).
- QA gate: render.qa.failOn matches issue codes or severities; outputs are
  still delivered (unless FF_QA_BLOCK_UPLOAD) and listed in QA_FAILED.
- Sinks: s3 (multipart via lib-storage, ff-* metadata, public URLs),
  presigned PUT (one URL per output), local (atomic rename inside
  FF_OUTPUT_DIR), inline (base64 under FF_INLINE_MAX_BYTES). Retries for
  transient failures; UPLOAD_AUTH_FAILED / UPLOAD_URL_EXPIRED fail fast.
- Idempotency: a manifest per idempotencyKey; a later attempt for the same
  commit with all objects present skips install/render/upload.
- SSE: the terminal event is flushed before the stream closes.
- Tests against an in-memory S3/presigned server, plus ffprobe checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): CLI mode and cache eviction (phase 7)
c6341a0 
- cli.ts: one job in-process from --job-file or stdin; result to
  --result-file or a final {"ffResult": …} line; exit 0/1/2.
- worker/boot.ts: config, engine alias, guards, journal, watchdog, janitor
  and recycling shared by the HTTP server and the CLI.
- CacheJanitor keeps FF_WORK_DIR / FF_CACHE_DIR under their limits between
  jobs: LRU workspaces; then LFS objects, git mirrors (other workers' only
  when idle >1h), package caches; never the next job's repo or the models.
  The LFS disk preflight evicts on demand before failing with DISK_FULL.
- Fix a lost-update race in the directory size walk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): image, RunPod templates, compose and docs (ph…
bae6733 
…ase 8)

- Move Dockerfile.renderer to apps/renderer-service/Dockerfile and the
  entrypoint to docker/entrypoint.sh. Runtime gains git, git-lfs, ssh,
  tini (PID 1) and a HEALTHCHECK; VARIANT=full adds a toolchain; runs as
  user framefields; modes http | runpod-serverless | cli.
- Pinned SSH host keys for github.com, gitlab.com and bitbucket.org,
  verified at build time against the published fingerprints.
- RunPod serverless and pod templates plus a saveTemplate script; a
  docker-compose setup with MinIO; a GHCR release workflow.
- Config: default cache dir prefers /runpod-volume, then
  /var/cache/framefields.
- README rewritten as the self-hosting guide.

Not built yet: no Docker daemon on the dev machine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): render with each project's own framefields
ffd8993 
The image no longer bundles or builds framefields. Each repo installs the
framefields version it pins (package.json + lockfile) and the worker renders
with that install.

- Engine binding: the first job imports its project's framefields and
  initialises the GPU; later jobs on the same version share it (imports from
  any workspace are routed to the bound engine). A different version, or new
  dependencies in a workspace already loaded, fails with retryable
  ENGINE_SWITCH and the worker recycles so the retry gets a fresh process.
- Older engines without worker hooks still render; no stall detection, and
  the worker recycles after each job to avoid stale caches.
- Projects must depend on framefields (ENGINE_NOT_FOUND); the "no install"
  fast path is gone. Only project sources get the per-job module token and
  forced ESM; node_modules keep their format and stay shared.
- Service drops all monorepo runtime deps (own pino logger); framefields
  and @framefields/renderer are test-only devDependencies.
- framefields exports resetRenderCaches().
- Image: worker only (turbo prune → install --ignore-scripts → tsc →
  pnpm deploy --prod), on node:24-trixie-slim because prebuilt Dawn needs
  glibc >= 2.38. Readiness = tools present; the engine is reported in /health.
- Fixes found running it in Docker: EPIPE from writing stdin to commands
  that don't read it; corepack's unpinned npm vs the lockfile's npm; install
  errors now name the real cause instead of npm's usage text.

Verified in the built image (arm64, software Vulkan): a repo pinned to
framefields@2.1.3 from npm installs, binds, renders VP9 and delivers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
fix(renderer): pick working video encoders; software WebGPU fallback
cf65727 
H.264/HEVC renders failed on linux-arm64 with "Encoder 'h264_rkmpp'
cannot be used for quantizer-based encoding". @mediabunny/server uses
Codec.findEncoder(id) as its software encoder, and node-av's linux-arm64
FFmpeg build registers h264_rkmpp/hevc_rkmpp before libx264/libx265.
NVIDIA GPUs without NVENC (A100, H100) would fail the same way, since the
hardware encoder was used without checking it works.

- video-encoders.ts: make libx264/libx265/libvpx/SVT-AV1 the default
  encoders, and pass a hardwareContext resolver that opens the platform's
  hardware encoder and encodes one frame before using it (once per
  codec). node-av is imported natively from @mediabunny/server's
  resolution so patches and HardwareContext instances match its copy.
  FRAMEFIELDS_HW_CODECS=off forces software.
- renderVideo always asks for "prefer-hardware"; the resolver decides
  (AMD keeps software).
- device.ts: fall back to a software WebGPU adapter when no GPU adapter
  exists; describeGpu() reports the adapter and whether it is a CPU one.
- Changeset (minor) also covers the earlier worker hooks: renderVideo
  signal/onProgress and resetRenderCaches().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): GPU selection with CPU fallback (FF_GPU)
0471099 
The image ships Mesa's CPU Vulkan driver, so a GPU machine whose Vulkan
driver isn't mounted silently rendered on the CPU.

- FF_GPU=auto (default): a machine with a GPU renders on it (Vulkan is
  limited to the GPU's ICDs) and fails with GPU_UNAVAILABLE if it can't;
  a machine without one renders on lavapipe. require: GPU only (RunPod
  templates). cpu: lavapipe only.
- After the engine binds, the actual WebGPU adapter is checked (via
  framefields' describeGpu() when available); a CPU adapter on a GPU
  machine fails the job and marks the worker unhealthy, and the RunPod
  shim refreshes the worker on GPU_UNAVAILABLE.
- /health components.gpu and results' engine.gpu report where it ran.
- README: GPU and CPU rendering section, NVENC GPU guidance; PLAN §8.7.

Verified in the arm64 container: H.264 job renders with libx264 on
lavapipe using the fixed framefields; FF_GPU=require fails readiness.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
ci(renderer-image): publish to GHCR on main and release tags
a008162 
- Add the output module (verify, sinks, deliver): the root .gitignore's
  **/output/ rule had kept apps/renderer-service/src/output out of git,
  so CI couldn't build the service.
- Push to main publishes :main and :sha-<short>; renderer-v* tags publish
  :<version>, :<major.minor> and :latest; -full variants alike. Pull
  requests build and smoke-test only; manual runs publish by default.
- Smoke test also boots the worker and requires /health ok on CPU.
- README/PLAN: published tags.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl
images
d48a549
@okanasl @claude
fix(renderer-service): LFS with local repos; disk space in /health
aacc8b1 
LFS end-to-end tests with real git-lfs (file:// origins): cold download
into the shared store, warm hit, objects shared across repos, include
patterns + LFS_POINTERS_LEFT, LFS off, LFS_OBJECT_MISSING. They found two
bugs with local repos:

- `-c lfs.storage` also redirected git-lfs's file:// transfer agent, which
  then looked for the remote's objects in our store ("remote missing
  object"). lfs.storage now goes in the repo config.
- The agent drops GIT_CONFIG_GLOBAL and -c but reads $HOME/.gitconfig, so
  a mounted repo owned by another user failed with "dubious ownership".
  The session's isolated config is now $HOME/.gitconfig, with
  safe.directory for file:// repos.

Verified in the arm64 container: a repo with an LFS-tracked framefields
tarball and logo installed from LFS and rendered the logo.

/health now reports both disks (free, total, eviction threshold,
lowOnSpace), from a snapshot refreshed in the background at most every
10 s. Test timeout raised to 30 s for real git/LFS work under load.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
chore(release): 2.2.0-beta.0 and prerelease renderer images
8248d90 
- Changesets pre mode (beta): all packages 2.2.0-beta.0. framefields is
  published under npm's `beta` dist-tag (release:framefields:beta), so a
  plain `npm i framefields` keeps installing the stable 2.1.x.
- renderer-image: prerelease tags (renderer-v2.2.0-beta.0) publish
  :2.2.0-beta.0 and :beta only, never :latest or :major.minor. A tag push
  runs the workflow from the tagged commit, so this branch can publish
  before it is merged.
- Renderer docs point projects at framefields@beta; RunPod templates and
  compose use the :beta image until the first stable renderer release.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
feat(renderer-service): smoke-test project; ignore implausible disk s…
d81fbc8 
…izes

examples/smoke-project is a ready-made repo for testing a deployment, on
framefields@beta (2.2.0-beta.0). Exports: film (title, media, effects,
charts scenes with push/wipe/iris transitions, score, grain), threeD
(studio environment, shadows, bloom, orbiting camera, primitives,
extruded text, glTF), vision (ONNX matte, pose, boxes), all (three
outputs), plus each scene alone. Covers text animators, mattes, flex,
gradients, shapes with trim paths, image + LUT + curves, SVG, Lottie,
video, video-in-type, ten effects, charts, SRT captions and audio.
Generated assets (glTF, Lottie, captions, a 17³ LUT) are committed; the
repo's media policy keeps fonts and footage out of git, so
scripts/copy-media.sh copies them from a local checkout. package.json
lists allowScripts: npm 11.19 skips unapproved install scripts, and
framefields' native dependencies need theirs.

Verified locally (Metal) and through the worker container (LFS repo,
inline output). The container run showed Docker Desktop host mounts
reporting a 117,870 GB disk with 7.8% free, which made every job evict
everything. The janitor now caps percentage thresholds at 200 GB and
treats disks above 64 TiB (or free > total) as not measurable: no
free-space eviction there, /health reports measurable: false, and the
optional GB caps still apply.

.dockerignore: nested node_modules and the renderer examples are left
out of the build context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
fix(renderer-service): smoke project gem with UVs; inline limit is pe…
e4c8cb9 
…r job

- The generated glTF gem now has TEXCOORD_0. Without it the model drew
  nothing on Linux/Vulkan (it renders on Metal); that engine bug is
  tracked separately, and the smoke test shouldn't trip on it.
- README: FF_INLINE_MAX_BYTES applies to a job's outputs together; `all`
  returns ~7 MB and needs a raised limit or S3 (verified: the default
  failed with OUTPUT_TOO_LARGE after rendering).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
fix(renderer-service): tini subreaper on RunPod; document RunPod results
d683475 
- RunPod starts the container through its own init, so tini isn't PID 1
  and warned that zombie reaping is off; run it with -s.
- README/PLAN: RunPod RTX 3070 results (GPU detected, statfs accurate on
  the container disk, threeD 21 s cold, film 16 s warm, vision 388 s on
  CPU ONNX) and the NVENC finding: node-av 6.1.1's FFmpeg needs NVENC API
  13.1 (driver >= 610); on driver 580 framefields falls back to libx264.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
perf(vision): share sessions across layers, run on WebGPU by default
60c35a0 
- node-vision keyed its one shared runner on every option, so layers that
  differed in classes/confidence closed and recreated their ONNX sessions
  each frame. Runners are now keyed by model options, and decode options
  get a VisionRunner.derive() over the same sessions. This also fixes the
  "thread constructor failed" crash with onnxruntime-web installed.
- Layers showing the same pixels share one RTMDet-Ins pass (content hash);
  runs of one model are queued.
- onnxruntime-web is an optional dependency of framefields. Dawn's
  GPUSupportedFeatures gets the forEach/values/entries ORT-web calls;
  software adapters (lavapipe, SwiftShader) fall back to the CPU EP.
- CPU sessions on Linux size intra-op threads from the CPUs the process
  may use (cgroup quota); FRAMEFIELDS_VISION_THREADS overrides.
- FRAMEFIELDS_VISION_PROVIDER=auto|webgpu|cpu|cuda forces a backend.

Smoke project `vision` export on an Apple Silicon Mac: 84.9 s before;
41-43 s on the CPU and 26-29 s on the GPU after.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
chore(release): 2.2.0-beta.1
a050a6e 
Changesets pre mode (beta): all packages 2.2.0-beta.1, carrying the
vision GPU change (vision-gpu-shared-sessions). Measured in the renderer
worker on a RunPod RTX 3070: vision export 58.7 s on 2.2.0-beta.0,
22-25 s with this build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@okanasl @claude
fix(renderer-service): RunPod result URLs without a query string; cap…
75be57e 
… the network-volume cache

RunPod Serverless returned 500 for every result POST: the platform's job-done
URL had no query string and the SDK appends &isStream=..., so finished jobs
were re-queued. The shim adds the missing ? before importing the SDK and logs
the URL's shape. Network volumes report 2.3 PB, so the serverless template
sets FF_CACHE_MAX_GB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@okanasl