Skip to content

Prevent Lightsail SSH certificate expiry during deployments - #139

Open
crohr wants to merge 4 commits into
mainfrom
auto/fix-lightsail-ssh-expiry
Open

Prevent Lightsail SSH certificate expiry during deployments#139
crohr wants to merge 4 commits into
mainfrom
auto/fix-lightsail-ssh-expiry

Conversation

@crohr

@crohr crohr commented Aug 10, 2026

Copy link
Copy Markdown
Member

Summary

  • record the expiration returned with Lightsail SSH access details
  • replace expiring AWS credentials with a run-scoped Ed25519 key after initial SSH readiness
  • enforce a 12-hour server-side expiry and remove the key best-effort when deployment finishes
  • rebuild the bundled Linux amd64 action binary

Validation

  • make test
  • mise exec -- go test ./...
  • mise exec -- go test -race ./internal/pullpreview ./internal/providers/lightsail
  • mise exec -- go vet ./...
  • upx -t dist/pullpreview-linux-amd64
  • clean GPT-5.6-sol xhigh adversarial review

Closes #138

@crohr

crohr commented Aug 10, 2026

Copy link
Copy Markdown
Member Author

@codex review

@phindmarsh

phindmarsh commented Aug 12, 2026

Copy link
Copy Markdown

I tested #139 twice against our longer Lightsail deployment, pinned to the exact PR head commit:

pullpreview/action@1ebf6c794ff8ac84508aade418cd4f95f5c59844

Unfortunately, both runs still failed when a later Docker-context SSH connection was opened.

Run 1:

  • Fresh Lightsail instance created at 21:04:28
  • SSH readiness succeeded at 21:06:25
  • Docker context created at 21:06:49
  • At 21:12:45, Compose opened another connection and received: Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
  • Time from Instance created to authentication failure: about 8m 16s

Run 2:

  • Fresh Lightsail instance created at 20:59:07
  • SSH readiness succeeded at 21:01:34
  • Docker context created at 21:02:04
  • At 21:07:24, Compose opened another connection and received: Permission denied (publickey,gssapi-keyex,gssapi-with-mic)
  • Time from Instance created to authentication failure: about 8m 17s

In both runs:

  • GitHub confirmed it downloaded and ran commit 1ebf6c7
  • deployment progressed normally for several minutes before authentication began failing
  • Established run-scoped SSH access for deployment never appeared
  • the final failure was Docker-over-SSH exit status 255

Given that handoffExpiringSSHAccess() is called after SSH readiness and returns early when Access.ExpiresAt.IsZero(), the missing handoff message appears to indicate that the Lightsail ExpiresAt value is arriving as a zero time. The action consequently continues using the temporary AWS certificate,
which then expires during Compose.

It may be worth logging the received ExpiresAt value and triggering the run-scoped key handoff whenever a Lightsail CertKey is present, even if ExpiresAt is missing or zero.

@HarderBetterFasterStronger

Copy link
Copy Markdown

Tested this branch against a long, fully-cold Lightsail deploy (~13 min image build, region us-west-2). Two findings that might help:

1. The run-scoped handoff never fires for us - ExpiresAt comes back zero. handoffExpiringSSHAccess is reached (Instance ssh access OK logs immediately before it), but i.Access.ExpiresAt.IsZero() is true, so it returns early and the deploy keeps using the temporary cert. It still failed around the 15-min mark with mux_client_request_session: Session open refused by peer / Permission denied (publickey) (all containers healthy, Failed: 0 - only the final reconcile SSH session was refused). It looks like GetInstanceAccessDetails doesn't populate expiresAt for our instances, so the guard disables the whole fix.

2. Gating on the cert's presence instead made it work end to end:

func (i *Instance) handoffExpiringSSHAccess() error {
    if strings.TrimSpace(i.Access.CertKey) == "" {
        return nil
    }
    ...

With that change, the handoff fires (Established run-scoped SSH access for deployment), and a fully cold ~13-min deploy now completes on the first attempt with zero SSH-cert errors. The temporary cert always expires regardless of whether AWS reports expiresAt, so gating on the cert being present seems more robust than gating on ExpiresAt.

One note for anyone else testing: if your workflow sets ControlMaster auto for the host (SSH multiplexing), it can pin every connection to the cert-established master opened before the handoff runs, which defeats the fix - dropping that made the difference alongside the gate change.

FWIW I also had to rebuild the bundled dist/pullpreview-linux-amd64 from source to test (the committed binary is compressed). Happy to open a PR against the branch with the gate change if it's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lightsail SSH certificate expires during longer deployments

3 participants