Improvements for credential sync by josephjclark · Pull Request #1548 · OpenFn/kit

josephjclark · 2026-09-14T12:47:21Z

Short Description

With deploy v2 this PR lets you:

omit credentials
force include all credentials
map credentials new names/owners

This also makes a refactor to the core merge code, allowing keys to be removed if not present. I might need to think about this a little more.

To make the feature fit nicely there's a little refactoring of some utils

Examples

Deploy a project spec with only the credentials it actually uses (this is actually the default)

openfn project deploy spec.yaml --creds prune

Deploy a project spec, stripping all credentials

openfn project deploy spec.yaml --creds none

Deploy a project spec, mapping a@gmail.org:cred to service@openfn.org:cred

openfn project deploy spec.yaml --creds cred=service@openfn.org|cred

Deploy a project spec, with a credentials.yaml with aliases

// credentials.yaml
a@gmail.org|cred:
  alias: service@openfn.org|cred

openfn project deploy spec.yaml -c credentials.yaml

Neat!

AI Usage

Please disclose whether you've used AI anywhere in this PR (it's cool, we just
want to know!):

I have used Claude Code
I have used another model
I have not used AI

You can read more details in our
Responsible AI Policy

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

josephjclark · 2026-09-14T16:56:47Z

+  const overrides: any = { ...pickedSource, ...assigns };
+  const merged: any = assign({}, target, overrides);
+
+  // null in an override (from source or assigns) is an explicit removal


This is an importand diff as it allows us to remove keys in the mege (like next, if removing edges, or credentials, which is a bug I've fixed here)

josephjclark · 2026-09-14T17:24:36Z

+  const merged: any = assign({}, target, overrides);
+
+  // null in an override (from source or assigns) is an explicit removal
+  for (const key of Object.keys(overrides)) {


Actually if I remove this code (because no-oneis passing in nulls anyway) everything seems to pass, apart from the bogus test

I think this is the AI leading me around the horn

josephjclark · 2026-09-14T17:47:48Z

Ok I think this is alright. I'll run some manual tests in the morning

* Worker: don't hang when fetch:plan replies with an error joinRunChannel awaited sendEvent(GET_PLAN) inside an unguarded async callback, so a rejection (now possible since Lightning can reply an error to fetch:plan for adaptor resolution failures) became an unhandled rejection instead of settling the outer promise. The worker's local capacity slot for that run leaked until restart. Wrap the await in try/catch and reject, mirroring the existing error/timeout branches in the same file. * Sync removing stuff (rebased) (#1530) * Worker: better reporting on socket errors (#1505) * Worker: attribute sentry reports to the run that produced them Each run already opened a sentry isolation scope, but engine and socket callbacks fire outside its async context, so breadcrumbs and errors were landing on the global scope and interleaving across concurrent runs. The scope is now held on the run context and re-entered where the reporting actually happens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * carry payload size on step-complete events for better reporting * type fix * add payload size to step complete event * report payload size to sentry * changeset --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * Worker: Fix dataclip serialization (#1507) * Worker: attribute sentry reports to the run that produced them Each run already opened a sentry isolation scope, but engine and socket callbacks fire outside its async context, so breadcrumbs and errors were landing on the global scope and interleaving across concurrent runs. The scope is now held on the run context and re-entered where the reporting actually happens. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * carry payload size on step-complete events for better reporting * type fix * report better errors out of sentry * Gate double-encoding fix for output dataclips behind WORKER_NO_STRINGIFY_STATE The worker double-JSON-encodes step output dataclips before sending them to Lightning: once via a manual stringify, then again when phoenix serializes the envelope. That double-encoding is what bloats large dataclips past Lightning's websocket frame limit and kills the connection mid-run. The runtime already sanitizes/clones state before it reaches this point, so the manual stringify is redundant - but skipping it changes the wire format, so it's opt-in via --stringify-state/--no-stringify-state (CLI) or WORKER_NO_STRINGIFY_STATE (env), default false to preserve current behaviour until the matching Lightning-side support (2.19+) is widely deployed. * docs and changelog --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * versions * Workflow: add remove()/isRemoved() to flag removed steps, edges and workflows Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Project: generate delete: true entries from Workflow.removed in to-app-state Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix an issue where removing a workflow with history triggers an error * fix remove workflow * clean up comments * hydrate deleted state * update changeset * mock: support for deletes * test for removing workfow * mock: seed version history * update deploy test * changeset * remove dupe test * remove another dupe * remove another dupe * again --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * CLI: checkout direct from project.yaml (#1533) * cli: start ensuring that v2 spec files can be converted to state Several things wrong * project: fix serialisation issue * fixes * remove stray debug console.log calls Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * little refactor * better logging * test: assert credential linkage and edge structure for new-from-spec deploy Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * add test of scenario 1 * Fix an issue where --name is ignored when loading a project from file * ensure cross deployment works from yaml * fix aliases for new projects * fix the alias fix for new projects from checkout * typing * ensure that projects loaded from state files can be loaded statelessly * changeset * fix test * attempt to fix a deploy to an existing project (#1542) * attempt to fix a deploy to an existing project * fix an issue where the wrong alias gets updated * Checkout project v1 (#1546) * support deploying from a v1 spec yaml A v1 spec (as exported from the app) has no uuids and cross-references everything by key, so fromAppState's uuid matching silently mangled it: credentials were dropped and every edge collapsed onto the first job. Convert it to the v2 spec shape instead and let the v2 parser take it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * create to/from-app-spec serializers * tidy * new tests --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * Improvements for credential sync (#1548) * CLI: add --credentials option to project deploy Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * add new credentials helpers * refactor credential utils * update docs * mock: throw for unmatched credential * add tests on credential mapping * fix to project merging which fixess credential and channels bugs * update changesets * revert a bunch of junk * clarify comments * support credentials.yaml and update credential id and alias formatting * add a little validation --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * ignore credential errors in sentry (#1550) * versions --------- Co-authored-by: Stuart Corbishley <corbish@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

josephjclark and others added 3 commits September 14, 2026 10:10

CLI: add --credentials option to project deploy

57e0722

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

add new credentials helpers

ebce5ca

refactor credential utils

bd002cb

taylordowns2000 added this to Core Sep 14, 2026

github-project-automation Bot moved this to New Issues in Core Sep 14, 2026

josephjclark added 4 commits September 14, 2026 14:00

update docs

ddd86c9

mock: throw for unmatched credential

fc4be95

add tests on credential mapping

e5cb261

fix to project merging which fixess credential and channels bugs

36fe645

josephjclark commented Sep 14, 2026

View reviewed changes

Comment thread packages/project/test/merge/merge-project.test.ts

josephjclark commented Sep 14, 2026

View reviewed changes

update changesets

e7fb724

josephjclark commented Sep 14, 2026

View reviewed changes

josephjclark added 2 commits September 14, 2026 18:33

revert a bunch of junk

910662b

clarify comments

2e2c9b1

This comment was marked as resolved.

Sign in to view

josephjclark added 2 commits September 15, 2026 11:05

support credentials.yaml and update credential id and alias formatting

f184026

add a little validation

9ad802b

josephjclark merged commit 27ecb7b into release-cli-next Sep 15, 2026
10 checks passed

github-project-automation Bot moved this from New Issues to Done in Core Sep 15, 2026

josephjclark deleted the better-credential-sync branch September 15, 2026 10:53

Navigation Menu

Improvements for credential sync - #1548

Improvements for credential sync#1548
josephjclark merged 12 commits into
release-cli-nextfrom
better-credential-sync

josephjclark commented Sep 14, 2026 •

edited

Loading

Uh oh!

Uh oh!

josephjclark Sep 14, 2026

Uh oh!

josephjclark Sep 14, 2026

Uh oh!

josephjclark commented Sep 14, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

josephjclark commented Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Short Description

Examples

AI Usage

Uh oh!

Uh oh!

josephjclark Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

josephjclark Sep 14, 2026

Choose a reason for hiding this comment

Uh oh!

josephjclark commented Sep 14, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

josephjclark commented Sep 14, 2026 •

edited

Loading