AWS Quick Create onboarding (Launch in AWS) + CloudTrail coverage label fix by akashmaurya3160 · Pull Request #60 · authsec-ai/authsec

akashmaurya3160 · 2026-09-24T19:47:02Z

AWS Quick Create onboarding ("Launch in AWS") + CloudTrail coverage label fix

Branch: aws-quick-create-onboarding → authsec-staging

Summary

Replaces the manual AWS onboarding flow (download YAML → upload → type two
parameters → copy RoleArn → paste back) with a CloudFormation Quick Create
link and an automatic callback. The customer picks AWS Region(s), clicks
Launch in AWS, ticks the IAM acknowledgement and clicks Create stack.
Nothing is pasted back.

Off by default. Without the new settings, nothing changes: the console
shows the manual flow exactly as today and the worker does not start.
No migrations. Sessions live in Redis.
Onboard() is unchanged and still the only thing that connects an account
(AssumeRole with the ExternalId, GetCallerIdentity, account check, Vault, upsert).
Discovery/scan code is untouched, apart from one separate commit
(6545dcc, below) that fixes a coverage label.

Design and review history: .claude/specs/SPEC-aws-quick-create-onboarding.md.
Runbook: docs/flows/aws-cloud-discovery-onboarding.md → Quick Create.

How it works

Console ──POST /aws/onboarding/sessions──▶ session in Redis (1h, single use) + Quick Create link
Customer's stack (deployment region)
  IAM role ──DependsOn──▶ Custom::AuthSecRegistration ──▶ AuthSec SNS topic (same region, required by AWS)
                                                            └──▶ central SQS queue (+ DLQ)
Callback worker ─▶ session by sha256(ExternalId) ─▶ checks ─▶ Onboard() ─▶ SUCCESS/FAILED to CloudFormation
                                                                         └─▶ per-Region probes
Console polls GET /aws/onboarding/sessions/:id until connected / failed.

The callback is never trusted by itself; the topic accepts publishes from any
account. A connection is made only after the checks below and a real AssumeRole
with the session's ExternalId.

Files changed (26: 21 for Quick Create, 5 for the coverage fix below)

File	Why
`internal/awsdiscovery/authsec-aws-discovery-role.yaml`	Optional `CallbackTopicArn` + conditional `Custom::AuthSecRegistration` (ServiceTimeout 600). `ExternalId` is no longer NoEcho: Quick Create ignores NoEcho params, and the value is already readable in the trust policy. Role permissions unchanged. Version `2026-09-24`.
`internal/awsdiscovery/onboarding.go`	`TemplateVersion` bump.
`internal/awsdiscovery/quickcreate.go`	Builds the Quick Create URL; refuses NoEcho/undeclared params, GovCloud, China.
`internal/awsdiscovery/cfn_callback.go`	CloudFormation custom-resource protocol: SNS envelope, request parsing, ResponseURL exact-host allow-list (from live captures), response PUT with no Content-Type and no redirects. Never logs the presigned query.
`internal/awsdiscovery/callback_config.go`	Parses and validates settings (topic must be in its own region, commercial, not opt-in).
`internal/awsdiscovery/quickcreate_test.go`, `testdata/cfn_response_urls/*.json`	Tests; host fixtures captured live in ap-south-1 and us-east-1.
`services/cloud_aws_quickcreate.go`	Sessions and the trusted callback sequence (details below).
`services/cloud_aws_cfn_callback_worker.go`	SQS worker: bounded concurrency, visibility heartbeat, last-chance answer, panic recovery.
`services/cloud_aws_quickcreate_test.go`, `…worker_test.go`	21 tests, one per rule and per review fix.
`services/cloud_aws_quickcreate_live_test.go`	Opt-in live AWS test (build tag `awslive`; never runs in CI).
`controllers/platform/cloud_aws_controller.go`	`automatic` block in `GET /aws/onboarding`; `POST`/`GET /aws/onboarding/sessions`. The launch link is returned only to the user who started the session.
`routes/routes.go`	Two routes: POST needs `discovery:admin`, GET needs `discovery:read`.
`cmd/main.go`	Starts the callback worker only when configured; built off the boot path.
`go.mod` / `go.sum`	`aws-sdk-go-v2/service/sqs` only. No other dependency changed.
`docs/flows/aws-cloud-discovery-onboarding.md`	Runbook: configuration, infrastructure, operating rules.
`.claude/specs/SPEC-aws-quick-create-onboarding.md`	Design, deviations, review fixes, deferred work.

Callback trust sequence

Every step runs before any AWS call:

Arrived through one of our topics; a well-formed request for AuthSecRegistration.
ResponseURL host exactly on the allow-list for the stack's region; otherwise it is never contacted.
Session found by sha256(ExternalId), and the ExternalId is bound to the session's workspace.
Topic region == stack region == session region; stack account == role account; commercial partition.
Single use; at most 5 attempts.
Onboard(), then SUCCESS. Delete and Update always answer SUCCESS and never change anything.

Failure handling

IAM propagation AccessDenied is retried for 90s.
Throttling and timeouts are retried to the last safe moment.
AuthSec-side errors that don't heal (Vault, DB, own credentials) fail after 60s.
No attempt runs within 40s of the deadline, so the answer always beats CloudFormation's 600s timeout.
On the queue's last delivery, AuthSec answers FAILED instead of letting the message fall into the DLQ unanswered.
A duplicate delivery that finds the session locked never answers.
After a successful Onboard(), a failed save still answers SUCCESS.
The lock has an owner token and a keep-alive.
Visibility is 2 minutes with a heartbeat, so a crash frees messages within minutes.
Connections are audited (onboard cloud_connector CALLBACK aws-quick-create/<region>/<stack>).

Separate fix in this PR: `6545dcc` — page cap is `partial`, not `denied`

Scan coverage showed "CloudTrail events · us-east-1 · ≥ 10000 · Denied" on a
busy account. Nothing was denied:

The CloudTrail reader stops at its deliberate cost bound, the first 10,000
events of 48h.
surfaceResult mapped every non-throttle error, including the page cap, to
denied.

Files: internal/awsdiscovery/{iam.go, cloudtrail_events.go, cloudtrail_events_test.go}, services/{cloud_aws_iam_scan.go, cloud_aws_surface_result_test.go}.

A page cap is now partial ("Partly read" in the console) with a clear
message. Reconciliation is unchanged: only reached is authoritative, so
partial blocks deletion exactly as denied did. This touches scan code, so
please review it on its own.

Production prerequisites (before enabling; not needed to merge)

Item	Notes
`REDIS_URL` on the backend	Redis is already used by GCP OAuth.
SNS topic per supported deployment region	Topic policy allows only `sns:Publish`. The cross-account policy still needs a second-account test.
Central SQS queue + DLQ	Raw delivery off; queue policy limited to our topics. `maxReceiveCount` 25.
Template publishing (CI step)	Upload the embedded YAML to a public, versioned S3 key for each `TemplateVersion`.
Settings	`AUTHSEC_AWS_CFN_CALLBACK_TOPICS`, `AUTHSEC_AWS_CFN_CALLBACK_QUEUE_URL`, `AUTHSEC_AWS_TEMPLATE_BASE_URL`, `AUTHSEC_AWS_OPTIN_SCAN_REGIONS`; optional `AUTHSEC_AWS_CFN_CALLBACK_CONCURRENCY` (default 16, max 128).
AuthSec's AWS identity	`sqs:ReceiveMessage`, `DeleteMessage`, `ChangeMessageVisibility`, `GetQueueAttributes` on the queue.
Alarms	Oldest-message age over 60s, DLQ depth over 0, `[aws-onb] ALERT` log lines.
Operating rule	Topics and the worker must stay up while any customer stack exists: they answer the stack's Delete.

Scale

About 1000 concurrent onboardings:

Redis: under 10 MB.
UI polling: about 333 reads/s at peak, each a cheap Redis GET.
Worker: about 100–150 callbacks/min per replica at 16 slots, so 2–3
replicas clear a simultaneous 1000-customer burst in minutes, inside the 540s
deadline.
Scaling further: add replicas, or raise the concurrency after measuring
STS throttling.
Deferred optimisation: move the per-Region probes off the worker slot.
They already run after SUCCESS. Build it when the oldest-message alarm fires.

Testing

go build ./..., go vet clean. Quick Create + worker: 21 tests pass.
Full suite: only controllers/admin, controllers/shared and
internal/migration fail, identically on the base commit, because they need
a local Postgres on :5432.
Merge with origin/authsec-staging: no conflicts; the merged tree builds and passes.
Live, lab account 429418377036:
- Quick Create in ap-south-1 and us-east-1.
- Tampered ExternalId: FAILED plus rollback that deletes the role.
- Stack delete: prompt SUCCESS.
- No answer: 600s timeout, then clean rollback.
- Manual template, no callback: role only; the right ExternalId assumes it, a wrong one gets AccessDenied.
- Browser end to end on the dev stack, including re-onboarding and the audit record.

Not in this PR

Organizations/StackSets; region editing (Phase 2 T2.1); GovCloud/China; metrics
and the daily canary; async per-Region probes (deferred, see the spec).

The design for replacing the manual AWS onboarding flow (download YAML, upload, type two parameters, copy RoleArn, paste it back) with a CloudFormation Quick Create link and a Custom::AuthSecRegistration callback over regional SNS topics into one central SQS queue. Committed before any code so the build can be reverted step by step. Backend only in this branch; the wizard changes come later. The AWS spike (Part C) still has to finish before any of it merges.

The adapter half of automatic AWS onboarding. Nothing calls it yet. Template (version 2026-09-24, role permissions unchanged): - ExternalId is no longer NoEcho. Quick Create ignores NoEcho params, so the Launch link could never have pre-filled it; the value is already readable in the role's trust policy and is not a secret. - Optional CallbackTopicArn and a conditional Custom::AuthSecRegistration that reports RoleArn/ExternalId/AccountId to AuthSec's regional SNS topic, ServiceTimeout 600. Empty topic = the manual flow, unchanged. awsdiscovery: - QuickCreateURL in AWS's documented #/stacks/create/review format; refuses non-commercial partitions and any NoEcho or undeclared param. - CFN callback protocol: SNS envelope, custom-resource request, StackId and topic ARN parsing, response encoding under the 4096-byte limit, PUT with no Content-Type and no redirects. - ValidateResponseURL: exact host match against fixture-backed forms (region without dashes in the bucket name), https/443, no userinfo, presigned query required. Errors never carry the signed query. - CallbackConfig: region->topic map (topic must be in its region, commercial, not opt-in), queue URL, versioned template URL, and the opt-in regions AuthSec's own account can scan.

AWSQuickCreateService wraps AWSOnboardingService (unchanged) and connects accounts only through the existing Onboard(). Sessions (Redis, 1h, single use): validated AWS Region(s) -- commercial only, opt-in regions only if AuthSec's own account enabled them -- a deployment region with a callback topic (moved first, since Onboard probes regions[0]), a fresh ExternalId, and stack/role names sharing an 8-char suffix. Refuses to hand out a link whose template is not published (cached HEAD check). Callback, every check before any AWS call: our topic; well-formed request for AuthSecRegistration; ResponseURL on the allow-list (else never contacted, left for the DLQ); session by sha256(ExternalId) and its workspace binding; topic region == stack region == session region; stack account == role account. Then Onboard() -- AssumeRole with the session's ExternalId and GetCallerIdentity -- and only then SUCCESS. Failure handling: IAM-propagation AccessDenied retried for 90s; AuthSec-side errors retried until SNS timestamp + 540s, then an honest FAILED before CloudFormation's own 600s timeout. Per-session lock, per-request stored result replayed on redelivery, attempt cap. Delete and Update always SUCCESS and never change anything. A 4xx from the presigned URL flags the session. Regional probes run after the answer through the same path a scan uses. Logs carry no signed query and no raw ExternalId.

Drains the central queue every regional callback topic delivers to and hands each message to HandleCallbackMessage. Done deletes the message; Retry makes it visible again in 30s; Reject releases it immediately and never answers it, so after maxReceiveCount it lands in the DLQ and its alarm. Messages in a batch are handled concurrently so one slow Onboard cannot push the others past their callback deadline. Receive visibility is 15 minutes, longer than the 540s callback deadline plus the regional probes; the plan's 5 minutes would have let a slow message reappear to a second replica mid-flight. Started from main only when automatic onboarding is configured and Vault is reachable; AUTHSEC_DISABLE_AWS_CFN_CALLBACK_WORKER=true turns it off. Adds github.com/aws/aws-sdk-go-v2/service/sqs v1.52.0 (no core SDK bump).

POST /authsec/discovery/aws/onboarding/sessions (discovery:admin) starts a session and returns its Quick Create link; GET .../sessions/:id (discovery:read) returns its status and per-region results. A session of another workspace reads as 404. Errors carry their aws_onb_* code: invalid regions are 422, deployment-side problems 503. GET /aws/onboarding gains automatic: {enabled, supported_deployment_regions, default_deployment_region, optin_scan_regions}; enabled:false means the console keeps the manual flow. Its note no longer calls the ExternalId a secret.

Flow doc: ExternalId is no longer NoEcho, the two session endpoints, and a Quick Create section with configuration, per-environment infrastructure and operating rules (topics and worker are permanent; alerts; support ref; fixture-backed ResponseURL hosts). Spec: the deliberate deviations from the plan.

The first live run (lab account, stack in ap-south-1) was rejected at the ResponseURL check: CloudFormation sends cloudformation-custom-resource-response-apsouth1.s3.ap-south-1.amazonaws.com, the .s3.<region>. form, while the only form on the list was the legacy .s3-<region>. one from AWS's documentation example. Adds the captured fixture (host and path verbatim, query values redacted) and the form it proves. The legacy form stays, documented as documented-not-observed. Also adds the opt-in live harness (build tag awslive) that drove the run: real template, CloudFormation, SNS, SQS, worker and AssumeRole; Redis and Onboard's Postgres/Vault write substituted.

The second live run panicked on a nil existing-connector lookup (the harness built the service without an onboarding service). Two real problems behind it: a panic in one callback would take the whole backend down, and the message it was holding stays hidden for the 15-minute visibility timeout -- longer than the stack's 600s ServiceTimeout, so the customer's stack fails. The worker now recovers per message and hands it back for a prompt retry. The optional seams are nil-checked; a service with no Onboard answers FAILED instead of dereferencing nil.

Live run in us-east-1 was rejected at the ResponseURL check: CloudFormation sends cloudformation-custom-resource-response-useast1.s3.amazonaws.com there -- S3's legacy home region, global endpoint, no region in the host. Adds the captured fixture and the form, restricted to us-east-1: the same global form for any other region is still rejected.

Fixes from the production-readiness review. Behaviour on the happy path is unchanged; every change is about slowness, failure or duplicates. Worker - B1 No batch barrier: each message starts as it arrives, bounded by 16 in flight per replica. One slow customer no longer holds up everyone else's callbacks until their stacks time out. - B7 Visibility 2 min, extended every minute while a message is handled. A crashed or redeployed worker releases its messages within minutes, inside the stack's 600s ServiceTimeout, instead of stranding them 15. - B2 Reads the queue's maxReceiveCount and passes a last-chance flag: on the delivery before the DLQ a transient failure is answered FAILED, so the stack fails with a reason instead of timing out in silence. Service - B3 No Onboard attempt may still run once a 40s answer reserve before the deadline begins; each attempt's context ends there too. The answer can no longer land after CloudFormation's own timeout. - B4 A delivery that finds the session locked never answers, even past the deadline: its FAILED could overtake the holder's SUCCESS. - B5 After a successful Onboard, a failed session save still answers SUCCESS: FAILED would roll back and delete the role just connected. - B6 InvalidClientTokenId / ExpiredToken / SignatureDoesNotMatch mean AuthSec's own credentials, not the customer's trust policy. - B15 AuthSec-side failures that do not heal (Vault, DB, credentials) fail after 60s instead of making the customer wait the full 9 minutes. Throttling and timeouts still retry to the last safe moment. - B8 Session lock has an owner token, compare-and-delete release, and a keep-alive; TTL 3 min. - B9 Quick Create connections are audited like the manual flow's. - B10 An opt-in region is never the one Onboard probes when a default region is selected. - B11 A panic in a region probe is recovered. - B13 Past the attempt cap nothing is written, and a finished session's lifetime is fixed once: leaked-ExternalId messages cannot grow a session or keep it alive. Each fix has a test.

…ator B12 The Quick Create service is built once per controller, so the template check's cache works; before, every launch paid an S3 HEAD and every GET /aws/onboarding re-parsed the environment. B14 GET /aws/onboarding/sessions/:id is discovery:read, but its launch link and ExternalId let whoever opens them connect an account -- an admin action. Only the admin who started the session gets them back; anyone else sees status and result. Runbook: maxReceiveCount 25 and the worker's own visibility/heartbeat. Spec: records the review fixes, superseding the 15-minute note.

- The launch link and ExternalId from GET /sessions/:id go only to the user who started the session, compared on the per-user id recorded at start. The actor used before can be a workspace-level client id that every user shares, which made the check match everyone. The audit record uses the same per-user id. - The callback worker is built inside its goroutine and reads the queue settings with a 10s bound, so a slow or unreachable SQS can no longer delay backend boot. - The heartbeat is stopped before a message is settled, so a late tick cannot undo the 30s retry / immediate reject visibility. - Past the attempt cap nothing is written, including the dead-URL flag. - Worker concurrency is configurable (AUTHSEC_AWS_CFN_CALLBACK_CONCURRENCY, default 16, max 128) for large onboarding bursts. - The misconfiguration ALERT is logged once, not on every GET /aws/onboarding. - go.mod: service/sqs moved to the direct requirements (it is imported). - Test harness clock guarded by a mutex (the worker test runs handlers concurrently).

Scan coverage showed "CloudTrail events · us-east-1 · ≥ 10000 · Denied" on a busy account. Nothing was denied: - RecentEvents reused IAM's maxPages (200), sized for 1,000-item pages and a runaway marker. LookupEvents returns 50 per page, so it stops at the first 10,000 events of the 48h window -- a cost bound a busy account reaches in normal operation. It now has its own named cap (unchanged at 10,000) and says so in the message. - surfaceResult mapped every non-throttle error to denied, including ErrTooManyPages. A page cap is now partial ("Partly read" in the console): the rows are real, the set is not known to be whole. Applies to every reader using the cap (Bedrock, EKS, Lambda too, though their larger pages make it unlikely). Reconciliation is unchanged: only reached is authoritative, so partial blocks deletion exactly as denied did. CloudTrail events never gate reconciliation anyway (bonus evidence).

…create-onboarding

Four conflicts, resolved by keeping every line of the graph work and re-applying Quick Create on top: - internal/awsdiscovery/onboarding.go: kept T2.4's comment and the new TemplateOutdated(); TemplateVersion moves from 2026-09-23 to 2026-09-24 so Quick Create's template change stays the newest version. - authsec-aws-discovery-role.yaml: kept T2.4's GetGateway and DescribeRegions grants; version 2026-09-24 in metadata, output and the AuthSecRegistration property. - services/cloud_aws_iam_scan.go: kept the new collectionCoverage() / out-based surfaceResult; the page-cap case is added in the same style. - controllers/platform/cloud_aws_controller.go: kept svc and cursors; the Quick Create fields are added after them. routes.go, cmd/main.go, iam.go and the runbook merged cleanly.

…une iteration

akashmaurya3160 added 16 commits September 24, 2026 07:40

spec(aws): record why region checks stay on the worker slot for now

bb9065b

spec(aws): neutral review wording

666cda9

Merge remote-tracking branch 'origin/authsec-staging' into aws-quick-…

f98f19d

…create-onboarding

akashmaurya3160 mentioned this pull request Sep 24, 2026

AWS: Launch in AWS onboarding + inventory coverage labels authsec-ai/Authsec-ui#6

Open

akashmaurya3160 added 2 commits September 25, 2026 13:09

fix(awsdiscovery): reject empty-string Principal; fix wildcardMatch r…

d7841e8

…une iteration

Navigation Menu

AWS Quick Create onboarding (Launch in AWS) + CloudTrail coverage label fix - #60

AWS Quick Create onboarding (Launch in AWS) + CloudTrail coverage label fix#60
akashmaurya3160 wants to merge 18 commits into
authsec-stagingfrom
aws-quick-create-onboarding

akashmaurya3160 commented Sep 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

akashmaurya3160 commented Sep 24, 2026

AWS Quick Create onboarding ("Launch in AWS") + CloudTrail coverage label fix

Summary

How it works

Files changed (26: 21 for Quick Create, 5 for the coverage fix below)

Callback trust sequence

Failure handling

Separate fix in this PR: 6545dcc — page cap is partial, not denied

Production prerequisites (before enabling; not needed to merge)

Scale

Testing

Not in this PR

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Separate fix in this PR: `6545dcc` — page cap is `partial`, not `denied`