Update adaptors on the fly by stuartc · Pull Request #4801 · OpenFn/lightning

stuartc · 2026-05-27T13:47:55Z

Description

Lightning now keeps its own adaptor catalogue in Postgres and refreshes it while
running, so a new adaptor or a new adaptor version shows up in the editor within
the hour with no rebuild and no redeploy.

Adaptor icons and credential schemas come from that catalogue too, served over
HTTP at request time rather than downloaded into the Docker image at build time.

In a cluster, one node does the refreshing and the others learn about it over
PubSub. Deprecated adaptors and deprecated versions are hidden from the pickers.

The design is strategy based:

`:npm`

Similar to the original implementation, it calls out to NPM to get a list of
adaptors for the @openfn organisation. It filters for language- package
names, and has (currently) a hard-coded list of adaptors to exclude.

It fetches adaptor schemas via jsDelivr and icons from GitHub.

`:local`

Similar to the NPM strategy, but all local - having parity with the original
LOCAL_ADAPTORS env variable. It retrieves the icons and schemas directly from
the local checkout, and supports overlay directories.

Both strategies are used by a centralized store and caching layer, where adaptor
versions, listings, schemas and icons are persisted in the database.

Another new feature is the ability to (more easily) dump and import all adaptor
information, including icons & schemas, into a tarballs and you can import them
onto another (in most cases) offline instance.

This replaces Lightning.AdaptorRegistry, which fetched the npm listing once at
boot and held it in memory, and the four build-time Mix tasks that baked schemas
and icons into the image (install_schemas, install_adaptor_icons,
download_adaptor_registry_cache). All of that is deleted.

These is a new ADAPTORS.md doc that explains how to set things up
for local repos, offline setups and some basic troubleshooting.

Closes #4473
Closes #3114
Closes #2209
Closes #325
Closes #1996
Closes #220

Validation steps

The default path, from nothing.
mix ecto.reset && iex -S mix phx.server. The catalogue starts empty and the
scheduler fills it from npm on boot; until it does, saving a job is refused
with "adaptor catalogue is not ready yet". Open a workflow, confirm the
adaptor picker fills and a job saves.
A new version, no restart. In the running server's iex:
Lightning.Adaptors.refresh(await: true), or click Refresh Adaptor
Registry via the Admin Menu > Maintenance page. An editor left open should
pick up the change without a reload.
Icons. Open a workflow. Every job node in the diagram, the minimap and
the adaptor picker should show its adaptor's icon rather than a text label.
These come from the running instance now, not from the image, so if they
render at all the new path works. To see the caching, open DevTools' network
tab and reload: the icon requests should be served from cache, and each URL
carries a content hash, so a changed icon gets a new URL rather than a stale
one.
Credential schemas. Go to /credentials and click New credential.
The type list is built from the catalogue now, instead of the priv/schemas
files that used to ship in the image, so it should include every
non-deprecated adaptor. Pick one and check the form fields match that
adaptor's configuration schema.
A local adaptors checkout.
ADAPTORS_STRATEGY=local ADAPTORS_LOCAL_REPO=/path/to/adaptors, restart, and
the picker should list the monorepo packages. Comma-separate two roots and
check the log names each shadowed package.
Offline. Set ADAPTORS_ICONS_PATH=/tmp/adaptor-icons first and restart.
By default, the icon cache lands under System.tmp_dir!/0, which can be
random depending on your OS.
```
mix lightning.adaptors.dump --path snapshot.json
tar czf icons.tar.gz -C /tmp/adaptor-icons .
```
Now reset the DB, empty the icon directory, and import with the network off:
```
mix ecto.reset
rm -rf /tmp/adaptor-icons && mkdir -p /tmp/adaptor-icons
tar xzf icons.tar.gz -C /tmp/adaptor-icons
mix lightning.adaptors.import --path snapshot.json --replace
```
Start the server with no network. The picker and the icons should both work,
and the hourly refresh should log a warning and leave the imported rows alone
rather than emptying them. mix lightning.adaptors.snapshot builds the same
file straight from npm if you have no populated instance to dump from, though
it carries no icons.
Refreshing switched off. ADAPTORS_REFRESH_INTERVAL_SECONDS=0 stops the
scheduled hourly refresh, for anyone who wants to control it themselves.
Confirm the two manual routes from step 2 still work with it set:
Lightning.Adaptors.refresh(await: true) in iex, and the Maintenance page's
refresh buttons.

Additional notes for the reviewer

On upgrade. Three migrations run (create_adaptors,
add_adaptor_catalogue_indexes, and widening credentials.schema from 40 to
100 characters). For a connected instance that's all: the first refresh fills
the catalogue from npm. Four things worth knowing:

The network moved from build time to runtime. Image builds no longer hit
npm, so they're faster and can't fail on a registry hiccup, but a fresh
instance now needs npm reachable when it starts. An airgapped instance must be
seeded before use.
SCHEMAS_PATH and ADAPTORS_REGISTRY_JSON_PATH are now ignored, with no
warning. Anyone setting them should drop them.
LOCAL_ADAPTORS=true and OPENFN_ADAPTORS_REPO still work and do warn at
boot, pointing at ADAPTORS_STRATEGY=local and ADAPTORS_LOCAL_REPO.
ADAPTORS_ICONS_PATH is set to /app/priv/adaptor_icons in the release image, and docker-compose.yml mounts a named volume there. Outside the image it still defaults under the system temp dir. Either way, persistent storage has to be mounted at that path or icons refetch after every restart.

New dependency. highlander_pg ~> 1.0 for the advisory-lock singleton,
which needed {:postgrex, override: true}.

Where to look hardest

lib/lightning/adaptors/scheduler.ex and supervisor.ex — the refresher is a
Postgres advisory-lock singleton. When it goes wrong it goes wrong quietly:
either no node refreshes, or all of them do. There's a leader-handover
integration test, but only one.
lib/lightning/adaptors/icon_cache.ex and
lib/lightning_web/controllers/adaptor_icon_controller.ex — filesystem writes
keyed on package names that came from npm, served on an unauthenticated route.
The last three commits are path-traversal hardening, so this is where the
churn was.
lib/lightning/workflows/job.ex — if the adaptors store fails to get
populated, this turns into refused workflow saves. Worth checking "not found"
and "not ready" stay distinguishable.
lib/lightning/credentials/schema_reconciler.ex plus the credentials.schema
widening from 40 to 100 characters, we now store full npm package names
(instead of the short handed http, dhis2 etc) and this process rewrites
existing rows from short adaptor names to full npm names. It would have been
much easier to use a migration but we need an adaptor listing. I imagine we
can remove this in a subseqent release.

Two new routes. /adaptors/icons/:name/:filename is
public and unauthenticated, deliberately, so icons cache. /adaptors/catalogue
needs a session but isn't project-scoped, which is right for global reference
data. /settings/maintenance is superuser-only, checked in mount/3 and again
in each handle_event. No policy module changed.

AI Usage

I have used Claude Code
I have used another model
I have not used AI

You can read more details in our
Responsible AI Policy

Pre-submission checklist

I have performed an AI review of my code (we recommend using /review
with Claude Code)
I have implemented and tested all related authorization policies.
(e.g., :owner, :admin, :editor, :viewer)
I have updated the changelog.
I have ticked a box in "AI usage" in this PR

Lightning.AdaptorRegistry no longer holds an in-memory npm-backed cache (use_cache, ADAPTORS_REGISTRY_JSON_PATH); adaptor metadata now comes from the Postgres-backed adaptors table populated by the scheduler introduced in #4801. Loading a static snapshot instead of npm is now `mix lightning.seed_adaptors_from_file` rather than the old JSON-path cache flag. Closes CON-101

github-actions · 2026-09-10T08:22:17Z

Security Review

⚠️ Review did not complete. See the workflow run.

codecov · 2026-09-10T08:28:10Z

Codecov Report

❌ Patch coverage is 89.35148% with 133 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-2.19.0@15e23fc). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
lib/lightning/adaptors/scheduler.ex	83.0%	46 Missing ⚠️
lib/lightning/adaptors/npm/github.ex	72.7%	18 Missing ⚠️
lib/lightning_web/channels/workflow_channel.ex	80.9%	17 Missing ⚠️
lib/lightning/adaptors/npm/registry.ex	82.0%	11 Missing ⚠️
lib/lightning/adaptors/catalogue_adaptor.ex	72.7%	6 Missing ⚠️
lib/lightning/adaptors.ex	92.5%	5 Missing ⚠️
lib/lightning/adaptors/supervisor.ex	88.4%	5 Missing ⚠️
...ghtning_web/controllers/adaptor_icon_controller.ex	87.5%	4 Missing ⚠️
lib/lightning/adaptor_service.ex	76.9%	3 Missing ⚠️
lib/lightning/adaptors/icon_cache.ex	87.5%	3 Missing ⚠️
... and 12 more

Additional details and impacted files

@@               Coverage Diff                @@
##             release-2.19.0   #4801   +/-   ##
================================================
  Coverage                  ?   91.1%           
================================================
  Files                     ?     452           
  Lines                     ?   22697           
  Branches                  ?       0           
================================================
  Hits                      ?   20677           
  Misses                    ?    2020           
  Partials                  ?       0

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

…efresh scheduler - Postgres-backed data model and store for adaptors/versions - Local-directory and npm source strategies - Supervision tree and periodic refresh scheduler

- Serve adaptor icons over HTTP, add a cacheable catalogue endpoint - Superuser maintenance page for refreshing the registry - Migrate AdaptorRegistry callers onto the Lightning.Adaptors facade - Send adaptors and icon URLs to the editor over the workflow channel, re-fetching the catalogue and broadcasting changed names on every adaptors_updated push - Configurable NPM upstream URLs; scheduler log/test trims and comment cleanup

- Local strategy supports multiple root directories - Lightning.Adaptors gets a public facade, closing Job's direct escape hatch - ADAPTORS_* env vars wired into bootstrap with back-compat, NPM defaults no longer duplicated - Old AdaptorRegistry deleted - Pre-existing lint/dialyzer failures on the branch fixed

- Test-server port randomized so mix test can run across parallel worktrees - bin/adaptor_cache rewritten as a self-contained record-and-replay proxy

- Adaptors subsystem supervised :one_for_one - ADAPTORS_ICONS_PATH documented so the icon cache can outlive a restart - New Credential type list now read from Lightning.Adaptors - Now-unused adaptor icon/schema build steps dropped - WorkflowChannel replies with an error instead of crashing on unrecognised messages

- Lightning.Adaptors waits for the catalogue's first load - Catalogue projection and stamp cached through Store - Adaptor exclusion filter restored, source/strategy fallback fixed, adaptor mix tasks renamed - Local-mode's "local" picker display restored after the registry rewrite

- Comment pass across the adaptors branch: fix stale/narration/inaccurate comments - Icon metadata carried through adaptor dump/import for airgapped mirroring - ADAPTORS.md guide added, adaptor docs elsewhere trimmed - Lightning.Adaptors.refresh/1 no longer crashes on a bare keyword list - Adaptor catalogue seeded in the real-worker integration tests - Two inaccurate warning-timing claims in ADAPTORS.md tightened

- Release form added for dumping the adaptor catalogue, refresh via rpc documented - Real-worker tests fail fast on a non-success run and seed the adaptor they need - Adaptor tests isolated from the shared Lightning.Adaptors instance, with a wait for the scheduler to come up and follow-up cleanup - ADAPTORS.md trimmed to reduce reading time without losing any instruction

- A failed lazy fetch named :fetch_failed, caching an empty version list - fetch_adaptor fails outright when the schema fetch fails - A stored schema is kept when a bumped version reports none - An icon that disagrees with its row is cached instead of refetched - Store.versions/2 and its projection deleted - Catalogue.list_missing_icons/1 deleted

- One adaptor_record fixture shared, package metas primed from the real projection; packages cache primed after inserting adaptors in picker tests - Schema decode+digest step shared between Local and NPM - A per-adaptor fetch failure now warns instead of dropping silently - An adaptor whose stored row has no schema is refetched; the lazy schema fill dropped from Store.schema/2 in favour of that refetch - Touch instead of upsert when a refetched schema is still nil - Single-clause `with` replaced in the schema decode paths - Icon sha put in the on-disk cache filename; unused version_record test helper deleted - Two scheduler tests that passed by accident fixed - Every superseded icon file swept regardless of extension - Stale strategy-error claim dropped from Adaptors.schema/2's doc - A schema-less row is only refetched while its version is still young

… configurable - Boot-time warning added when the adaptor catalogue is empty - Adaptor refresh interval now operator-configurable - Both documented in ADAPTORS.md

…h skills - docs-style rule added for shipped guides, plus a user-invoked docs-review skill; later anchored to root docs so it doesn't fire on any *.md - create-plan, implement-plan and tdd skills replace the old plan commands; create-plan made agent-invocable - reshape-pass skill added

- save_workflow and save_and_sync now push a timeout as a third argument; the four `save_and_sync` expectations pinned the call to two and failed

- Components format numbers and dates with the viewer's own locale, so a test asserting a formatted literal depended on the developer's LANG: ChatInput's counter reads "9,600" under en-US and "9 600" under en-ZA - LC_ALL set in vitest.config.ts rather than the npm scripts, so `npx vitest` behaves the same and the reason can be written down - A note at the one assertion that relies on it, warning against the tempting fix of hardcoding a locale in the component

- PhoenixChannelProvider registers a process 'exit' handler that only destroy() removes, so a test leaving a session initialised leaked one listener plus a Y.Doc, awareness and channel per test; the worst files reached 50 and 34 against a cap of 24 - createTestSessionStore added to sessionStoreHelpers: createSessionStore plus onTestFinished(destroy). The 10 raw call sites and the trigger inspector harness use it - process.setMaxListeners(24) deleted from the test setup, so the next leak surfaces as a warning instead of being absorbed. Node writes that warning to raw stderr, which is why it never reached the junit report

Five modules were mapping a shape to its icon columns: the catalogue schema, the store's projections, the scheduler's record merges, the controller and the URL builder. The schema did it under a different vocabulary again, taking :icon_square where everything else says :square. IconField holds the mapping, so an unknown shape is a FunctionClauseError at the boundary rather than a freshly minted atom.

put_resp_content_type took a value computed from the request, which sobelow flags at medium and which CI fails on. Two clauses, one per extension, make the content type a literal at the call site and give the function a catch-all 404 instead of a FunctionClauseError. The shape string now becomes an atom once in show/2 rather than at each of the three places that needed it, so the controller reads its columns through IconField like everything else and has_icon?/2 and ext_for_shape_param/2 are gone. send_file keeps a sobelow_skip. Sobelow flags any send_file whose path traces back to a param, which is true of every file server; the path is built by Adaptors.icon/2 from a catalogue row, so an unknown adaptor 404s before any of it happens.

Adaptor names become directory segments in IconCache.path/5, and the name format allowed "..": [\w.-]+ matches it, and the format permits one slash, so "../.." resolved two levels above the cache root. write!/6 then ran mkdir_p!, write! and a File.rm sweep wherever it landed. The name check also ran too late to help. Icons are written from the strategy's response in Scheduler.merge_icon/4, well before the row reaches CatalogueAdaptor.changeset/2 and its validate_format — and the changeset is the only place a name is ever validated, since none of the update_all paths touch that column. So the guard goes in path/5, which every File call in the module routes through, and it raises. The scheduler already rescues around write!/6, so a hostile registry entry now degrades to a logged warning. The name format is tightened alongside it: a segment must not start with "." or "_", which is npm's own rule and keeps "." and ".." out.

dump_to_file/2 and seed_from_file/2 take a mix-task or release-command argument, and the Local strategy reads under the paths an operator configured for it. None of them see request input.

Destructuring says which half is the name and which the version, so the TODO asking for that no longer has anything to ask.

An empty catalogue only needs an operator's attention when no timer will fill it: with an interval set the first tick is already due immediately, so that case logs at info and the warning is left for interval=0. The suite's own boot hits exactly that state, so warn_when_empty turns the warning off for the instance application.ex starts. Both it and refresh_interval now reach the scheduler as required opts, read once from config by the supervisor the way strategy already is, which lets the three test helpers that start a scheduler directly pass their own values instead of writing to the application env and restoring it.

- Express the adaptor refresh interval in seconds - Give the adaptor icon cache a fixed path and a volume - Start the adaptors subsystem from the out-of-band setup commands - Start Finch when the adaptors supervisor starts on its own

- Gate every catalogue read on the first load, and give the facade tuples - Count the adaptors a refresh tick failed to write - Settle the first-load gate on a source that lists no adaptors - Stop reporting an unreachable catalogue as a refused adaptor

- Spell out what an upgrade has to act on for the adaptor catalogue - Delete the unused request_adaptors channel handler - Describe install/2's three outcomes in the AdaptorService test moduledoc

…pt-in - Treat an empty npm org listing as an error, not an empty catalogue - Answer catalogue reads immediately and make waiting for the first load opt-in - Drop refresh waiters whose call has already timed out

- Return an error from get_schema/1 instead of raising - Show a retry state in the credential form when the catalogue cannot answer - Log why setup is waiting on the adaptor catalogue - Document what a cold catalogue does while it loads - Resolve adaptor names without waiting for the first load

- Name the errors a catalogue read can return - Write the adaptor docs for operators, not as a change log - Move Lightning.Adaptors namespace into its own docs section

lmac-1 · 2026-09-14T15:40:03Z

Being nosy here as I switched branches and noticed something.. did this PR mean to change Claude commands to skills as part of this PR? @stuartc

stuartc · 2026-09-14T20:15:40Z

Being nosy here as I switched branches and noticed something.. did this PR mean to change Claude commands to skills as part of this PR? @stuartc

Why yes, it did! The branch was in flight for like 3-4 weeks, and I did some housekeeping while working on things in different models/sandbox envs.
I can't remember the exact reason for the commands to skills, it might have been around invocable vs non-invocable via other prompts or subagents; can't recall.

github-project-automation Bot added this to Core May 27, 2026

github-project-automation Bot moved this to New Issues in Core May 27, 2026

stuartc mentioned this pull request May 27, 2026

Allow on-the-fly updates to adaptors, schemas, and icons #4473

Closed

7 tasks

stuartc self-assigned this May 28, 2026

stuartc moved this from New Issues to In progress in Core May 28, 2026

midigofrank mentioned this pull request Jun 19, 2026

Credentials.get_schema/1 does not respect OPENFN_ADAPTORS_REPO when LOCAL_ADAPTORS=true #4885

Open

stuartc force-pushed the adaptors-on-the-fly branch from 6447ee6 to 17876ea Compare August 25, 2026 07:45

stuartc force-pushed the adaptors-on-the-fly branch 2 times, most recently from fb1332f to 0a02fe8 Compare September 2, 2026 09:58

stuartc force-pushed the adaptors-on-the-fly branch from d53b3d3 to 68c2f13 Compare September 3, 2026 13:07

stuartc force-pushed the adaptors-on-the-fly branch 4 times, most recently from f0755f3 to 43ee0c8 Compare September 10, 2026 07:15

stuartc marked this pull request as ready for review September 10, 2026 08:17

midigofrank changed the base branch from main to release-2.19.0 September 10, 2026 11:56

stuartc added 8 commits September 14, 2026 12:37

Add the adaptors data model, Postgres store, source strategies, and r…

0bb4b60

…efresh scheduler - Postgres-backed data model and store for adaptors/versions - Local-directory and npm source strategies - Supervision tree and periodic refresh scheduler

Randomize the test-server port and rewrite bin/adaptor_cache

6750e0d

- Test-server port randomized so mix test can run across parallel worktrees - bin/adaptor_cache rewritten as a self-contained record-and-replay proxy

stuartc added 22 commits September 14, 2026 12:52

Warn at boot on an empty adaptor catalogue, make the refresh interval…

df888e6

… configurable - Boot-time warning added when the adaptor catalogue is empty - Adaptor refresh interval now operator-configurable - Both documented in ADAPTORS.md

Assert the save timeout in GitHubSyncModal's push expectations

205bf0e

- save_workflow and save_and_sync now push a timeout as a third argument; the four `save_and_sync` expectations pinned the call to two and failed

Update sobelow to 0.15.0

c10222f

Annotate the traversal checks on operator-supplied paths

986b8c6

dump_to_file/2 and seed_from_file/2 take a mix-task or release-command argument, and the Local strategy reads under the paths an operator configured for it. None of them see request input.

Read the version group-by without elem/2

e35e499

Destructuring says which half is the name and which the version, so the TODO asking for that no longer has anything to ask.

Run the adaptors subsystem outside the web app

8716e77

- Express the adaptor refresh interval in seconds - Give the adaptor icon cache a fixed path and a volume - Start the adaptors subsystem from the out-of-band setup commands - Start Finch when the adaptors supervisor starts on its own

Document the upgrade and drop the unused adaptors channel handler

321b3ff

- Spell out what an upgrade has to act on for the adaptor catalogue - Delete the unused request_adaptors channel handler - Describe install/2's three outcomes in the AdaptorService test moduledoc

Answer catalogue reads immediately, make waiting for the first load o…

71d56c2

…pt-in - Treat an empty npm org listing as an error, not an empty catalogue - Answer catalogue reads immediately and make waiting for the first load opt-in - Drop refresh waiters whose call has already timed out

Trim and correct comments across the adaptors branch

ea9f963

Validate credential schemas against the widened column

03fe6fa

Write the adaptor docs for operators

6737708

- Name the errors a catalogue read can return - Write the adaptor docs for operators, not as a change log - Move Lightning.Adaptors namespace into its own docs section

stuartc force-pushed the adaptors-on-the-fly branch from b010fbb to 6737708 Compare September 14, 2026 11:04

midigofrank approved these changes Sep 14, 2026

View reviewed changes

update changelog

9e991f2

midigofrank merged commit 75f3f53 into release-2.19.0 Sep 14, 2026
1 check was pending

midigofrank deleted the adaptors-on-the-fly branch September 14, 2026 12:26

github-project-automation Bot moved this from In progress to Done in Core Sep 14, 2026

Navigation Menu

Update adaptors on the fly - #4801

Update adaptors on the fly#4801
midigofrank merged 37 commits into
release-2.19.0from
adaptors-on-the-fly

stuartc commented May 27, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

codecov Bot commented Sep 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

lmac-1 commented Sep 14, 2026

Uh oh!

stuartc commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

stuartc commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

:npm

:local

Validation steps

Additional notes for the reviewer

AI Usage

Pre-submission checklist

Uh oh!

github-actions Bot commented Sep 10, 2026

Security Review

Uh oh!

codecov Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

lmac-1 commented Sep 14, 2026

Uh oh!

stuartc commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stuartc commented May 27, 2026 •

edited

Loading

`:npm`

`:local`

codecov Bot commented Sep 10, 2026 •

edited

Loading