From c7230c3da5d07fbb0946e93c465e624b1182dc7f Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 07:13:23 +0000 Subject: [PATCH 01/36] docs(design): add gitops-api's ask to wake the reconciler only on foreign commits Wishlist ask #29 from the gitops-api team: every commit the reverser pushes wakes Flux, which can then apply that commit over a cluster that has already moved past it, so the branch ping-pongs. The ask is to have the reverser forward a reconcile trigger only for commits it did not author, and only after any pending writes have been replayed on top. Imported as written, with the filename typo fixed, markdownlint's emphasis style applied, and the links to gitops-api's own docs made absolute. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/bi-directional-right.md | 338 ++++++++++++++++++++++++++++ 1 file changed, 338 insertions(+) create mode 100644 docs/design/bi-directional-right.md diff --git a/docs/design/bi-directional-right.md b/docs/design/bi-directional-right.md new file mode 100644 index 00000000..57be7a5a --- /dev/null +++ b/docs/design/bi-directional-right.md @@ -0,0 +1,338 @@ +# Feedback: the reconciler should only be woken by commits the reverser did not make + +_From the [gitops-api](https://github.com/ConfigButler/gitops-api) team to gitops-reverser, +2026-10-15. Wishlist ask **#29** ([upstream-wishlist.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/upstream-wishlist.md)). Written to be +readable from the gitops-reverser repo; no gitops-api knowledge assumed. **We run `v0.52.0`.** Every +gitops-reverser link is pinned to +[`400edf68`](https://github.com/ConfigButler/gitops-reverser/tree/400edf68242086dde5f650ac9835072f2bdfb5e1) +(`main` today)._ + +**The ask, in one sentence.** Make gitops-reverser the component that tells the GitOps reconciler to +apply, and have it do so **only when the branch gained a commit the reverser did not author**, +never for its own pushes. + +**Why.** In the bi-directional setup your own guide recommends, every commit the reverser pushes +wakes Flux, and Flux then applies that commit to a cluster that may already have moved past it. +Two ordinary API edits a few seconds apart are enough. The older commit is applied over the newer +edit, the reverser faithfully mirrors that revert into Git as the reconciler's commit, and the +branch ping-pongs until a fetch happens to land on the tip. Which value survives is luck. There is +no conflict being decided here: **both edits came from the API side.** + +## 1. What we saw + +Our product is an editor over a kcp workspace. A user's save is a Server-Side Apply into the +workspace, gitops-reverser (one instance, `v0.52.0`) mirrors it to the tenant's repository, and a +Flux `GitRepository` + `Kustomization` (Flux 2.9.6) applies the repository back. Our Git-host webhook +handler wakes the **`GitRepository` only**, never the `Kustomization`, as +[`bi-directional.md` → Configure Flux](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#configure-flux) +recommends (we learned that one the hard way too). + +Our browser check fires two saves at once. We serialise them ourselves, so the second save writes +only after the first save's `CommitRequest` is `Ready=True/Committed`. One run, 2026-10-15, tenant +repository history: + +| Time (UTC) | Commit | Author | Change to `data` | +|---|---|---|---| +| 15:53:00 | `3b08d12a` | Simon (save A) | `+ a` | +| 15:53:05 | `08b9bf84` | Simon (save B) | `- a`, `+ b` | +| 15:53:11 | `ad719e84` | `flux-tenant-configbutler` | `+ a` | +| 15:53:19 | `f22d63e1` | `flux-tenant-configbutler` | `- a` | +| 15:53:26 | `f4fbc60b` | `flux-tenant-configbutler` | `+ a` | +| 15:53:32 | `d88fc968` | `flux-tenant-configbutler` | `- a` | + +Save B's `- a` is our harness, not the race: both saves applied under one field manager, so B's +apply released A's key. Everything after it is the race. + +The same pattern appeared in the 2026-10-13 run. To check the mechanism, we patched our +serialisation back to "release on `Committed`" and reran with both saves writing **the same +field**. The result was two reconciler-authored commits after the saves, `a03446a1` and `49b4baea`. +The `Kustomization` never settled on the later save's commit. + +```mermaid +sequenceDiagram + autonumber + actor U as User (two saves) + participant API as Kubernetes API + participant Rev as GitOps Reverser + participant Git as Git branch + participant Flux as Flux (GitRepository + Kustomization) + + U->>API: save A (x = 1) + API-->>Rev: watch event + Rev->>Git: push A + Git-->>Flux: push webhook → fetch A + U->>API: save B (x = 2) + Note over API: live is now AHEAD of A + API-->>Rev: watch event + Flux->>API: apply A (x = 1) + Note over API: B is reverted in the cluster + Rev->>Git: push B (x = 2) + API-->>Rev: watch event: x = 1, by Flux + Rev->>Git: push "x = 1" authored by Flux + Git-->>Flux: webhook → fetch B → apply (x = 2) + API-->>Rev: watch event: x = 2, by Flux + Rev->>Git: push "x = 2" authored by Flux + Note over Git,Flux: lags one commit behind, ping-pongs
until a fetch lands on the tip +``` + +Steps 5 and 7 are the whole bug: the cluster moves, then Flux applies the commit before it. +Everything after them is the reverser doing exactly what it should, which is why nothing errors: +mirroring stays faithful and every commit is attributed. + +## 2. Why it happens, in your own terms + +[`bi-directional.md` → Why the loop stops](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#why-the-loop-stops) +gives the reason the loop settles: _"When the reconciler applies a commit Reverser wrote, the live +object already holds those values, because they were read from that object in the first place."_ +That carries a precondition the page states once (_"With no intervening edits"_) but nothing +enforces: **the live object has not moved since.** The page is also already explicit that nothing +reserves the time: + +- [The guide's short version](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md): + _"There is no merge, no lock, and no timing guarantee."_ +- [How a refused edit can undo an allowed one](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#how-a-refused-edit-can-undo-an-allowed-one): + _"The same race exists whenever a new Git revision arrives … Shortening the commit window reduces + exposure but does not reserve time against an apply."_ +- [`facts/gitops-apply-and-field-ignore.md`](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/facts/gitops-apply-and-field-ignore.md): + _"a reverse merge alone cannot stop the forward applier from enforcing an older Git value before + publication completes."_ + +We agree with all of that. Our point is narrower. **For a commit the reverser authored, the apply +can never do anything useful.** That commit _is_ the cluster's state, read from it. Applying it back +has two possible outcomes: a no-op, if the cluster has not moved, or a revert, if it has. No case +exists where waking the reconciler for the reverser's own push brings in new intent. The whole +exposure in §1 comes from triggers that cannot pay off. + +[§1.2 of the push-notification design](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#12-which-side-wins-before-anything-else) +already draws this line: _"A push we did not author is new truth … getting it applied quickly is +the point."_ The converse is the ask: **a push we did author is not news to the cluster, and +nothing should be told to apply it.** + +## 3. The ask: the reverser decides when the reconciler applies + +```mermaid +flowchart LR + api(["API edit"]) --> cluster[("Cluster")] + cluster -- "watch event" --> rev["GitOps Reverser"] + rev -- "commit + CAS push" --> git[("Git branch")] + human(["Someone pushes to Git"]) --> git + git -- "push webhook (one, to the reverser)" --> rev + rev -- "after = a tip WE pushed?" --> drop["ignore:
the cluster already has it"] + rev == "branch gained a commit we did NOT author" ==> rec["Flux Receiver / Argo CD webhook"] + rec -- "apply" --> cluster + + style rev fill:#e8f4fd,stroke:#2196f3 + style rec fill:#fff3e0,stroke:#ff9800 + style drop fill:#eeeeee,stroke:#9e9e9e +``` + +This is your [Option 4](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#option-4-take-over-the-webhook-and-forward-it) +(the Git host talks only to the reverser) combined with +[Option 3](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#option-3-receive-a-push-notification)'s +dedupe table. One change: **the dedupe decides whether to forward, not only whether to fetch.** + +| `after` on the delivery | Meaning | Option 3 today | **Asked for** | +|---|---|---|---| +| a tip the reverser pushed | our own push coming back | ignore | **ignore, and do not forward** | +| anything else | somebody else moved the branch | invalidate `baseTrusted` | invalidate, **then forward** (§4 says when, §5 how) | + +As written, Option 4 forwards every delivery, so it would keep the race exactly as it is today. +[Option 2](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#option-2-tell-the-reconciler-after-we-push) +("tell the reconciler after we push") is the race made into a feature, and we would ask you **not** +to build it in that form. Its own selling point ("the loop closes without the user configuring a Git +host webhook") is a loop that has nothing left to close. + +The bookkeeping this needs is the bookkeeping Option 3 already names: `lastCommitSHA` must advance +on a successful push, or every own push looks foreign. Option 3's argument against a `before` field +still holds. A set of our own SHAs cannot tell our own delayed delivery from a rollback through our +commits. Misclassifying our own push as foreign costs one useless apply, which is today's +behaviour. Misclassifying the other way would lose a human's push. So **when in doubt, forward**. + +The reverser also has two other ways to learn the branch moved, both shipped. Each should feed the +same rule: + +- a **rejected compare-and-swap push**, whose advertisement names a hash none of its publications + produced; +- **`--git-refresh-interval`**, which reports `status.remote.verifiedBy: Fetch` for exactly that + case + ([Telling Reverser that somebody else pushed](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#telling-reverser-that-somebody-else-pushed)). + +So even an installation with **no** inbound webhook could trigger the reconciler correctly: late, +bounded by the refresh interval, but never for its own commits. + +## 4. Ordering: forward after the replay, not before + +A foreign commit `F` can land while the reverser still holds API edits it has not pushed yet: an +open commit window, or writes retained for replay. Call those edits `W`. They are live in the +cluster, absent from Git, and so absent from `F`. + +If the reconciler is woken at `F`, it applies `F` and reverts `W` in the cluster before `W` ever +reaches Git. Replay still puts `W` on top of `F` in Git, but the reverser then mirrors the revert +over it, and the API side loses. Your §1.2 table says the API wins this case (the top-left cell); +the early apply makes it lose. + +```mermaid +sequenceDiagram + participant Human as Git-side author + participant Git + participant Rev as GitOps Reverser + participant Rec as Flux / Argo CD + participant API as Cluster + + Note over Rev,API: open window W: an API edit Git does not have yet + Human->>Git: push F + Git-->>Rev: webhook (after = F, foreign) + Rev->>Rev: invalidate, fetch F, replay W onto F + Rev->>Git: CAS push F + W' + Rev-->>Rec: reconcile, at the tip it just pushed + Rec->>API: apply F + W' + Note over API: W' equals live for W's objects: a no-op there.
Only F's changes land. +``` + +**Asked for:** when a foreign commit arrives with retained or open writes, request the reconcile +after the replayed push lands, so the applied revision already carries the API side's answer. With +no pending writes, forward at once. This narrows the remaining window, and does not claim to close +it. An edit that lands between that push and the apply is still exposed, as your guide says. But +that exposure now exists **only when somebody else pushed to Git**, which is a genuine two-sided +event. Today it follows every commit the reverser makes. + +Option 5's caveats ("it cannot carry correctness") apply in full, and we are not asking for a hold +on the watch or on the push. This is only _when to send one HTTP request_. + +## 5. How: the request that wakes the reconciler + +The reverser learns of a foreign commit in one of two ways: a host delivery reaches it, or it +detects the move itself. Each case needs a different call. + +| Reconciler | With a host delivery (5.1) | Without one (5.2) | +|---|---|---| +| Flux | forward it to a `Receiver` of the host's type (`github`, `gitlab`, …) | POST to a `generic-hmac` `Receiver` | +| Argo CD | forward it to `/api/webhook` | a synthesized GitHub-shaped push to `/api/webhook` | + +In both cases **the reverser names no `GitRepository` and no `Application`.** The downstream already +holds that mapping: the `Receiver` in `spec.resources`, and Argo CD in every `Application` whose +source matches the repository and branch. This is your Option 4's load-bearing reason, and it holds +here too. + +### 5.1 With a host delivery: forward the bytes, later if need be + +When the Git host's webhook points at the reverser, forward the delivery exactly as Option 4 +describes: the same body and headers, with nothing re-signed or re-serialised. #29 changes only +_whether_ the delivery is forwarded (an own push is dropped) and _when_ (a foreign push waits for +the replay in §4). Holding a delivery back is safe for two reasons: + +- **The signature has no expiry.** GitHub's `X-Hub-Signature-256` is an HMAC over the body alone, + and GitLab's `X-Gitlab-Token` is a static token. A delivery forwarded a few seconds late still + verifies. +- **Neither downstream applies the delivery's `after`.** Each treats the delivery as "look at this + branch now" and resolves the tip itself. A delivery that names `F` but is forwarded after + `F + W'` lands therefore applies `F + W'`, which is the state §4 asks for. + +### 5.2 Without one: the reverser's own request + +Three of the triggers in §3 bring no delivery to forward: a rejected compare-and-swap push, a +`--git-refresh-interval` fetch, and any installation whose host webhook cannot point at the +reverser. **Ours is in the last group.** A GitHub App has one webhook URL for all of its +installations, and ours belongs to gitops-api. We would relay `{repository, branch, after}` to your +[Option 3 endpoint](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#41-the-wire-contract-for-option-3), +so the reverser would never hold a GitHub-signed delivery. + +In these cases the reverser makes its own request. That is your Option 2's outbound leg, with the +dedupe from §3 in front of it: + +- **Flux: a `generic-hmac` `Receiver`.** Send a POST to the path the Receiver publishes in + `status.webhookPath` (served by the `webhook-receiver` Service), with + `X-Signature: sha256=`. This is the call + `flux trigger receiver` makes. The body is free-form, so §4.1's three fields will do. The + Receiver's `spec.resources` should list the `GitRepository` only. The source is enough, and a + poked `Kustomization` applies the artifact it already holds, which is the commit _before_ the one + that triggered it ([field-ownership.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/field-ownership.md)). +- **Argo CD: a synthesized push event.** Argo CD has no generic receiver type, so the reverser POSTs + a minimal GitHub-shaped push to `/api/webhook`. It carries `X-GitHub-Event: push`, `ref`, `after` + and `repository.html_url`, and is signed with `webhook.github.secret` from `argocd-secret` when one + is set. Leave the `commits` file lists empty. As your Option 4 notes, a payload that lists no + changed files makes `AppFilesHaveChanged` return true, so Argo CD refreshes every matching + `Application` instead of guessing paths. This is the inverse of Option 4's parser, and it can come + after Flux. + +**Annotating the object directly** (`reconcile.fluxcd.io/requestedAt` on the `GitRepository`, +`argocd.argoproj.io/refresh` on the `Application`) also works. It is what our `triggerFluxForPush` +does today. We would still not recommend it for the reverser, because the reverser would then have +to know which object syncs the folder and hold patch rights on it. That is the ownership interpreter +your Option 4 avoids. A `Receiver` keeps the mapping where the user wrote it. + +Option 2's four rules apply unchanged. The POST requests a reconcile and never reports one as +applied. It runs off the publication path, retries on its own, and never fails a publication that +succeeded. + +## 6. What we would stop doing + +```mermaid +flowchart TB + subgraph TODAY["Today (gitops-api)"] + direction LR + gh1[("GitHub")] -- "every push" --> api1["gitops-api webhook"] + api1 -- "poke GitRepository" --> f1["Flux"] + rev1["Reverser"] -- push --> gh1 + end + subgraph ASKED["With #29"] + direction LR + gh2[("GitHub")] -- "every push" --> rev2["Reverser"] + rev2 -- "foreign pushes only" --> f2["Flux Receiver"] + rev2 -- push --> gh2 + end +``` + +- **Our webhook stops waking Flux.** We wake the `GitRepository` on every push today + (`triggerFluxForPush`), the reverser's own included. That is the trigger in §1. With #29 the + webhook relays the push to the reverser instead (§5.2), and the reverser becomes the only + component that decides a revision should be applied. +- **Flux keeps its own timers as the backstop**, and they stay long (`24h` on both objects). Your + Option 4's warning, _"'long intervals' must never be read as 'no intervals'"_, holds. One + addition for your Flux guidance: a **short** `GitRepository` interval (the `1m` that + [Configure Flux](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#configure-flux) + recommends) picks up the reverser's own commits too, because a new artifact revision wakes the + `Kustomization`. That brings §1 back on a timer. Under #29 the short poll would undo what the + filter buys, so the recommendation should say so. +- **Our stopgap goes.** Until then we serialise our own saves behind Flux's echo. A save holds a + per-repository gate until the `Kustomization` reports `lastAppliedRevision` = that save's commit, + so Flux's apply of it is a no-op before the next save may write + ([forward-leg-loop.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/forward-leg-loop.md#the-third-lap-a-save-that-overtakes-the-echo-2026-10-15)). + This is your deferred + [Option 4, "applier watermark"](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/future/git-api-three-way-comparison.md#option-4-use-the-appliers-applied-revision-watermark), + done crudely, outside the engine, and only for writes that pass through our server. A `kubectl + apply` straight into the workspace is not covered. It costs a queued save 10–15 s (30 s when + another push lands in between), and it is the kind of guard in one component that we would rather the engine made unnecessary. + +## 7. A test you could add + +[What the tests prove](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#what-the-tests-prove) +says _"no spec tests simultaneous writes to one field from both sides, because the outcome depends +on arrival order."_ This case is different: **both writes come from the API side**, so the expected +outcome does not depend on a policy. It is simply the later write. Under the recommended Flux +configuration (Receiver on the `GitRepository`, webhook on every push): + +1. Patch one field of one object to `1`. Wait for its commit to push. +2. Within ~2 s of that push, patch the same field to `2`. +3. Wait for both commits, plus two reconciler intervals. +4. Assert that the live value is `2`, that Git's file holds `2`, and that **no commit after step 2 is + authored by the reconciler's identity**. + +On our cluster, step 4's last clause fails on today's configuration and passes with our stopgap. It +should pass with #29 and no stopgap. We would rather it lived in your bi-directional corner than in +our browser harness, because it is a property of the engine plus the reconciler, not of our editor. + +## 8. What we are not asking for + +- **Not a merge, and not a conflict policy.** The + [three-way comparison record](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/future/git-api-three-way-comparison.md) + stays deferred as far as we are concerned. #29 is only about not causing a conflict where none + exists. +- **Not a filter on the audit stream or on events.** The reverser must keep mirroring every write, + the reconciler's included. A reconciler-authored commit is exactly how we saw this, and it should + stay visible + ([forward-leg-loop.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/forward-leg-loop.md) + is why we feel strongly about that). +- **Not a guarantee of being first.** As your Option 4 says, a timer, a retry, or `flux reconcile` + can always reach Git without the reverser. #29 removes the trigger that fires on every own + commit. It does not claim to remove every trigger. From b10c9cd6742084cb3fdae8e91387de5d681b3bc3 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 07:28:37 +0000 Subject: [PATCH 02/36] docs(design): answer #29: GitOps Reverser takes over the reconciliation triggers The ask filters the reconciler's webhook so our own pushes stop waking it. That is not enough: Argo CD's timed refresh resolves the tip, and with selfHeal off a new revision still auto-syncs, so every push of ours is applied two to three minutes later over whatever is still pending. Flux applies on every reconcile, whatever woke it. So the reconcilers keep fetching, rendering and applying, but apply only when we ask: Argo CD without automated sync and a sync pinned to the checked revision, Flux through its Receiver with long intervals. The response also records where the ask's rule needs correcting (empty refusal commits, the replayed push, per-cluster suppression), the branch-worker state that replaces per-delivery classification, and what changes for gitops-api and in this repository. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/bi-directional-right.md | 147 ++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) diff --git a/docs/design/bi-directional-right.md b/docs/design/bi-directional-right.md index 57be7a5a..5f05cf53 100644 --- a/docs/design/bi-directional-right.md +++ b/docs/design/bi-directional-right.md @@ -7,6 +7,11 @@ gitops-reverser link is pinned to [`400edf68`](https://github.com/ConfigButler/gitops-reverser/tree/400edf68242086dde5f650ac9835072f2bdfb5e1) (`main` today)._ +> **Response (2026-10-16): GitOps Reverser takes over the reconciliation triggers.** Flux and +> Argo CD keep doing what they are good at, but they apply when GitOps Reverser asks, not on their +> own schedule. The ask is right, and filtering webhooks turns out not to be enough: see +> [§9](#9-response-gitops-reverser-takes-over-the-triggers). + **The ask, in one sentence.** Make gitops-reverser the component that tells the GitOps reconciler to apply, and have it do so **only when the branch gained a commit the reverser did not author**, never for its own pushes. @@ -336,3 +341,145 @@ our browser harness, because it is a property of the engine plus the reconciler, - **Not a guarantee of being first.** As your Option 4 says, a timer, a retry, or `flux reconcile` can always reach Git without the reverser. #29 removes the trigger that fires on every own commit. It does not claim to remove every trigger. + +## 9. Response: GitOps Reverser takes over the triggers + +_From gitops-reverser, 2026-10-16._ + +**The short version: GitOps Reverser takes over the reconciliation triggers. It has to.** Flux and +Argo CD keep doing what they are good at: fetching, rendering, applying, pruning, and reporting +health. They stop doing it at their own will. An apply happens because GitOps Reverser decided the +revision is safe to apply, because a human asked for one, or because of a backstop timer that was +chosen knowing what it costs. + +We accept the ask's rule and its ordering. We go further than §8's last bullet: the webhook is not +the only trigger to take away, because every timer the reconciler runs is a trigger too. + +### 9.1 Why it has to + +An apply is safe only when nothing live is missing from the revision being applied. GitOps +Reverser is the only component that can know that, because it holds the open commit windows and +the writes waiting to be pushed. The applier cannot see an edit that has not reached Git yet. An +apply GitOps Reverser did not ask for is one nobody checked, and it reverts exactly the edits we +captured and have not published. + +That makes the trigger part of the engine's correctness rather than an integration detail, and a +component that cannot see the pending writes cannot own it. + +### 9.2 Timers are triggers too + +§6 notes this for Flux's short `GitRepository` poll. On Argo CD it is worse, and it is not a matter +of tuning. + +Argo CD's timed refresh (`timeout.reconciliation`, 120s plus up to 60s of jitter) resolves the +branch tip. With `selfHeal: false`, automated sync is skipped only when that tip equals the last +synced revision +([`alreadyAttemptedSync`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2648-L2670)). +Every push GitOps Reverser makes is a new revision. So two to three minutes after each of our +pushes, Argo CD syncs the whole Application at the tip, and reverts whatever is live but not yet +pushed at that moment. An idle branch is safe; during an editing session something is usually +pending, so in practice it resets an arbitrary edit. + +Filtering the webhook removes the sync that lands about four seconds after a push. It does nothing +about this one. On Flux the same holds for every reconcile: kustomize-controller applies whatever +woke it. + +### 9.3 What it means per reconciler + +| | Argo CD | Flux | +| --- | --- | --- | +| Configure | `syncPolicy` **without** `automated` | `Receiver` on the `GitRepository` only, long intervals on both objects | +| GitOps Reverser's trigger | a sync of a declared `Application`, pinned to the revision we judged safe | a signed POST to the `Receiver` | +| Their own timers | keep them: without auto-sync a refresh only reports `OutOfSync` | the residual: one apply per `Kustomization` interval that nobody checked | +| What stays exposed | an edit landing between our push and the sync's apply | the same, plus that interval apply | + +**Argo CD.** Without `automated`, a refresh from the timer, a webhook, or the UI only compares and +reports. Nothing applies until a sync, and a sync names its revision, so it applies exactly the +commit we checked rather than whatever the tip has become. This needs the `Application`'s name and +Argo CD's `applications, sync` permission. That is naming an orchestrator object, which this +repository has avoided until now. What +[push-notification-and-reconcile-trigger.md](push-notification-and-reconcile-trigger.md) objected +to was **inferring** which object owns a folder. A reference the user declares is a different claim, +and we accept it. + +**Flux** has no refresh-only mode, and `suspend: true` stops a `Receiver` as well. The closest +setup is long intervals with the `Receiver` as the only fast path. The `Kustomization` interval +then remains the one apply we do not control. A very long interval removes it, and removes the +backstop with it. That is the installer's choice, and the guide has to say so. + +### 9.4 What we take from the ask, and what we change + +We take the rule (trigger only when the branch gained commits we did not author), the ordering in +§4 (trigger after the replayed push lands), and the test in §7. Four things change. + +1. **Not every commit we author is a no-op.** There are three exceptions: + - `GitTarget.spec.onRefusal: PushEmptyCommit` exists only to make the reconciler apply, so it + must trigger. + - The replayed `F + W'` is our commit, but it carries `F`, which nothing has applied. + - A folder applied to a cluster other than the one it was mirrored from: a shared base, a fleet, + or a `GitTarget` that mirrors a remote cluster through `spec.kubeConfig`. Our commit is a no-op + only for the reconciler that applies to the cluster it was read from, so suppression is per + trigger target, not global. +2. **The decision is state on the branch worker, not a verdict on each delivery.** One flag, "the + reconciler is behind". It is set when the worker sees a tip it did not push (a rejected + compare-and-swap push, a refresh fetch, a foreign delivery) or pushes a refusal empty commit. It + is cleared by sending the trigger once the worker is quiet: no pending writes, base trusted. + Deliveries become hints. This settles the cases a per-delivery table gets wrong: + - the delivery for `F + W'` names our own SHA, yet `F` was never applied; + - a late delivery for an older push of ours; + - a rollback onto one of our own older commits, settled by one ref advertisement rather than by + guessing order; + - a delivery that arrives before `PushAtomic` returns, handled by recording the SHA while the + push is in flight. +3. **The hold cannot wait for commit windows to close.** A window closes on silence, so a user who + keeps editing would hold back a foreign push indefinitely. On a foreign commit, close the open + windows, replay onto `F`, push, then trigger, with a bound on the whole sequence. +4. **Forwarding host deliveries is no longer the mechanism.** Under the takeover an Argo CD webhook + only refreshes, and a Flux `Receiver` is a trigger target we call. Forwarding stays useful for a + Git host that can point at only one place, and to keep the reconciler's status fresh. + +### 9.5 The seam + +The decision is core and names no orchestrator. Delivering it is a short list of trigger targets, +named by what they send: + +| Trigger target | What GitOps Reverser sends | For | +| --- | --- | --- | +| `argocd-sync` | a sync of a declared `Application` at the checked revision | Argo CD | +| `webhook` | a signed POST, `generic-hmac` or a synthesized `github-push` | a Flux `Receiver`, or any third-party receiver | +| _(always)_ | a Kubernetes Event naming the branch, the revision, and why | whatever the installer wires up | + +A third-party GitOps tool plugs in through `webhook` or the Event. There is no in-process plugin, +for the reason Option 8a was declined: it would run with every credential the operator holds. + +### 9.6 What changes for gitops-api + +- Your webhook stops waking Flux and relays `{repository, branch, after}` to the Option 3 endpoint. +- The `lastAppliedRevision` gate goes, and has to: Flux no longer applies your own saves, so that + watermark never arrives for them. +- The `GitRepository` loses its `1m` poll, and the `Receiver` lists the `GitRepository` only. + +### 9.7 What changes in this repository + +- [`bi-directional.md`](../bi-directional.md): Argo CD without `automated`, Flux without the short + poll, and "Why the loop stops" restated around the trigger. +- [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md): a + direction is chosen. Option 2 as written is rejected, and Options 3 and 4 become inputs to the + trigger decision. +- The bi-directional e2e corner: the Argo CD specs that rely on automated sync change. §7's test + lands there, with an Argo CD variant run on a short `timeout.reconciliation`, asserting that no + reconciler-authored commit appears. + +### 9.8 What we accept, and what is open + +- **GitOps Reverser becomes the only fast path to a deployment.** While it is down, a push by a + human reaches the cluster only through a manual sync or Flux's backstop interval. Bi-directional + editing is already broken in that state; Git-side deployments were not, and now they are. +- **Uncertainty triggers.** When the branch worker is absent, suspended, or on another replica, the + trigger fires. Only a commit known to be ours is held back. +- **Open: where trigger targets are configured.** `GitProvider` (the repository) or `GitTarget` + (the folder and the cluster). The per-cluster suppression in §9.4 argues for the `GitTarget`. +- **Open: high availability.** The decision needs the branch worker's state, which lives on the + replica that owns the worker. +- **To verify:** that Flux's `github` `Receiver` type accepts a synthesized push without matching + the repository it names. If it does, `generic-hmac` is not needed. From 6e700ec82a3bb3328a1cae3a01853060c72aee70 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 08:08:56 +0000 Subject: [PATCH 03/36] docs: describe GitOps Reverser owning the apply trigger, ahead of the code Docs-first, for review: the guides now read as they will once GitOps Reverser decides when Flux or Argo CD applies a folder. - A GitTarget names its applier (spec.applier: kustomizationRef, applicationRef or webhook). Requests go out only on a quiet branch, for a foreign commit, an eligible refusal, or, for Flux, a CatchUp that keeps its artifact current after our own pushes. - Argo CD runs with automated.enabled: false, and our sync copies the operation its automated sync would build, pinned to the checked commit. - GitProvider.spec.receiver takes the Git host's push webhook on its own listener; a delivery is a hint judged against the ref advertisement. - onRefusal: PushEmptyCommit becomes onRefusal: Reapply. - definitions.md gains applier, apply request, quiet branch and receiver. The push-notification design drops its options menu for the decided design, and the reply to gitops-api's ask is aligned with it. The generated settings table is untouched until the CRDs change. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 126 +- docs/bi-directional.md | 681 +++++---- docs/configuration.md | 295 +++- docs/definitions.md | 28 +- docs/design/bi-directional-right.md | 197 +-- ...push-notification-and-reconcile-trigger.md | 1330 +++++------------ 6 files changed, 1186 insertions(+), 1471 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 40eb92ca..394bd51e 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -109,10 +109,23 @@ than a side effect, and `reconcile.configbutler.ai/requestedAt` still forces a f human who wants one now. A refusal a write found is left alone: a structural read cannot see a write-boundary precondition, so only the resync that writes the affected type clears that one. -What none of this closes is the latency: seconds-fresh needs the inbound receiver, -designed but not built. -[§8.3](design/push-notification-and-reconcile-trigger.md#83-the-wire-contract-for-whoever-calls-it) is the -request shape it will accept. +Seconds-fresh is the push receiver's job: a `GitProvider` with `spec.receiver` accepts the Git +host's push webhook. A delivery is only a hint: it makes the branch worker read the ref +advertisement, and what the remote says decides whether the branch moved. It never triggers a +cluster snapshot: the cluster still holds the values from before the push, so a snapshot would +publish them over the incoming change. + +**GitOps Reverser decides when Git is applied to the cluster.** An apply is safe only when nothing +live is missing from the commit being applied, and only the branch worker can see the edits that +are live and not yet pushed. So a `GitTarget` names its **applier** (a Flux `Kustomization`, an +Argo CD `Application`, or a webhook), the applier's own triggers are switched off, and the branch +worker sends **apply requests** instead. A request is sent only on a **quiet branch**, with no open +commit window and no pending writes: for a commit somebody else pushed (after closing open windows +and replaying the pending writes onto it), for a refused edit the target opted to revert, and, for +Flux, to keep its artifact current after GitOps Reverser's own pushes. Argo CD is never asked to +apply GitOps Reverser's own commits, because the cluster already holds those values. A request asks +and never reports; it runs off the publication path and cannot fail a publication. See +[the apply trigger design](design/push-notification-and-reconcile-trigger.md). **Redis/Valkey is optional but advised.** The default configured-author mode runs without it: a plain `helm install` comes up healthy and watches initialize from current state on startup. When an endpoint @@ -251,11 +264,16 @@ graph LR subgraph SOURCE["Source cluster selected by ClusterProvider"] API["Kubernetes API\ndiscovery · Namespace policy · watches"] + APPLIER["Applier\nFlux Kustomization or Argo CD Application"] end + HOST[("Git host")] + CP -. "optional kubeconfig reference" .-> KCFG CP -->|"source identity + config"| WM WM -->|"source client"| API + GT -. "spec.applier" .-> APPLIER + HOST -. "push webhook to spec.receiver" .-> GP style GP fill:#e8f4fd,stroke:#2196f3 style CP fill:#e8f4fd,stroke:#2196f3 @@ -272,6 +290,9 @@ source context: client, discovery catalog, followability registry, source-namesp reachability state. Its `GitTarget`s open their source watches through that context. The contexts are keyed by provider name and are released when no `GitTarget` references them. +The applier is not one of the six kinds. It belongs to Flux or Argo CD, lives in the source cluster +it applies to, and is reached through the same source context and identity the watches use. + | CRD | Scope | One line role | |---|---|---| | `WatchRule` | namespaced | which NAMESPACED resources, in which source namespaces, route to a GitTarget | @@ -370,6 +391,11 @@ Key fields: - `spec.commit.window`: when this target's commit windows close: `idleTimeout` (default `5s`) of silence, or `maxDuration` (default `1m`) after the window opened. - `spec.commit.message`: `liveTemplate` / `reconcileTemplate` Go templates. +- `spec.applier`: optional; exactly one of `kustomizationRef`, `applicationRef` or `webhook`, plus + `quietPeriod` (default `30s`). The object that applies this folder, and that GitOps Reverser asks + to apply. +- `spec.onRefusal`: `Ignore` (default) or `Reapply`, which asks the applier to reapply the tip when + an edit to an existing document is refused. `Reapply` requires `spec.applier`. `spec.commit` describes the folder, not the connection, so two `GitTarget`s sharing one `GitProvider` may batch and phrase their commits differently. A branch worker serves a `(provider, branch)` pair and @@ -393,7 +419,10 @@ Status has a kstatus-compatible summary layer plus domain conditions: source API the data plane can currently reach. - `GitPathAccepted` explains the target side: the selected Git path is safe for the operator to materialize. +- `ApplierConfigured`, present only with `spec.applier`, says the applier exists in the source + cluster, applies this repository and branch, and can be patched. It is a gate. - `status.streams` is a bounded count summary, not a per-type list. +- `status.applier` is an observation: the last apply request's commit, time and reason. WatchRule and ClusterWatchRule add `ResourcesResolved` and `GitTargetReady`. `ResourcesResolved` explains the source selector. `GitTargetReady` mirrors the referenced GitTarget's write readiness. This keeps @@ -412,7 +441,10 @@ Represents a Git repository and the credentials/configuration used to write it. - `spec.allowedBranches`: glob patterns that gate writable branches. - `spec.commit.committer`: committer identity (defaults to `GitOps Reverser` / `noreply@configbutler.ai`). - `spec.commit.signing`: SSH signing key reference and optional key generation. +- `spec.receiver`: optional push receiver, `type` (`github`, `gitlab`, `gitea`, `generic-hmac`) and + the `secretRef` holding its token, named after Flux's `Receiver`. - `status.signingPublicKey`: populated when signing is configured and key material is available. +- `status.webhookPath`: the receiver's path, `/git-push//`, when `spec.receiver` is set. The controller verifies repository reachability and manages the signing key lifecycle. It generates an ed25519 keypair when `signing.generateWhenMissing` is set. The portable artifact across GitOps @@ -598,6 +630,43 @@ For a `CommitRequest`, success is reported only after the push reaches the remot replay on top of someone else's commit, `status.commit` is the refreshed post-replay commit, not the stale local commit from before the retry. +### Somebody else pushed, and the cluster has to follow + +A rejected push is one of three ways the branch worker learns the branch gained a commit it did +not push. The other two are a push-receiver delivery and the periodic refresh. All three set the +same piece of worker state, **the applier is behind**, and the worker clears it by sending an apply +request once the branch is quiet. + +```mermaid +flowchart TD + HINT["Receiver delivery"] --> ADV["Read the ref advertisement"] + REJECT["Rejected compare-and-swap push"] --> SEEN + REFRESH["Periodic refresh fetch"] --> SEEN + ADV --> OURS{"Tip is a commit
we pushed?"} + OURS -->|yes| DROP["Nothing to do"] + OURS -->|"no, or cannot tell"| SEEN["Applier is behind"] + SEEN --> PENDING{"Open windows or
pending writes?"} + PENDING -->|yes| FLUSH["Close the windows, fetch,
replay onto the foreign commit, push"] + FLUSH --> QUIET + PENDING -->|no| QUIET["Branch is quiet"] + QUIET --> REQ["Apply request to every applier on the branch,
naming the tip, reason ForeignCommit"] + + style DROP fill:#eeeeee,stroke:#9e9e9e + style REQ fill:#fff3e0,stroke:#ff9800 +``` + +The worker records the commit of a push while the push is in flight, so a delivery that arrives +before `PushAtomic` returns is still recognized as its own. A rollback onto one of its own older +commits is foreign, because the advertisement names a tip that is not the last one it pushed. + +The same quiet-branch rule sends two other requests: `Refusal`, to the refusing target's applier +when `spec.onRefusal: Reapply` is set, and `CatchUp`, to Flux and webhook appliers once the branch +has been quiet for `spec.applier.quietPeriod` after the worker's own pushes. Delivering them is +the job of the applier client in `internal/applier`, off the worker's queue: a Kubernetes patch +for Flux and Argo CD, a signed POST for a webhook, retried with backoff, recorded as an +`ApplyRequested` Event and on `status.applier`. A failed request never reaches back into the publication that +caused it. + *** ## What it writes to Git @@ -1134,11 +1203,11 @@ write side is shared with live writes): - nothing is committed if the apply cannot complete safely. A **write-boundary** refusal is a different thing and is scoped accordingly: the folder is -accepted, and one edit had nowhere to land in it. Only those can earn the empty commit a target +accepted, and one edit had nowhere to land in it. Only those can earn the apply request a target opts into with -[`spec.onRefusal: PushEmptyCommit`](configuration.md#reverting-a-refused-edit-specconrefusal), and -only when the edit is to a document the folder already holds. A folder-level refusal above is -excluded deliberately: an empty commit cannot repair a folder, and only a human can. +[`spec.onRefusal: Reapply`](configuration.md#reverting-a-refused-edit-speconrefusal), and only +when the edit is to a document the folder already holds. A folder-level refusal above is excluded +deliberately: reapplying Git cannot repair a folder, and only a human can. The acceptance gate is **structure-only on purpose**: it never refuses on a discovery-derived followability fact (unwatched / out-of-scope), which can blink on a discovery wobble; only facts that are @@ -1167,6 +1236,8 @@ stops a fact being published by something that cannot actually prove it. | Where is branch B on the remote, and when was that proved? | the branch worker, on its queue | `GitTarget.status.remote`, sampled by each `GitTarget` on the branch | | What is this repository being used for? | the `GitTarget`s that reference it | `GitProvider.status.branches` | | Is this folder accepted, streaming and matching live? | the watch plane, per `GitTarget` | the `GitTarget`'s conditions and `status.placement` | +| Does the applier need to apply, and is now safe? | the branch worker, on its queue | `GitTarget.status.applier` and an `ApplyRequested` Event | +| Can the applier be asked at all? | the `GitTarget` controller, reading the applier in the source cluster | `ApplierConfigured` | A branch worker sits at the intersection and belongs wholly to neither object: `GitTarget`s decide which branches are needed, `GitProvider` decides which repository those branches are in. So the @@ -1196,8 +1267,10 @@ the newest revision beside a stale clock. ### Everything about a branch happens on that branch's queue Reading or writing a branch's repository state is the branch worker's job, and the worker does it on -one FIFO queue, on one goroutine. Live events, resyncs, `CommitRequest` attaches and periodic -refreshes all enter the same queue and are served in order. Nothing else opens a connection on a +one FIFO queue, on one goroutine. Live events, resyncs, `CommitRequest` attaches, periodic +refreshes and push-receiver deliveries all enter the same queue and are served in order. Whether +the branch is quiet is therefore a fact one goroutine holds, which is what lets the worker decide +when an apply request may go out. Nothing else opens a connection on a branch's behalf, and nothing else touches its clone. That is what makes "the base is trusted" a statement one goroutine can hold. @@ -1461,7 +1534,9 @@ Controllers watch their dependencies so dependents reconcile quickly after spec Namespace label changes re-check provider-reference access. Rule changes re-resolve selected collections. It deliberately does **not** watch encryption Secrets, so their recovery is picked up by periodic reconciliation without retaining - every Secret value in the control-plane cache. + every Secret value in the control-plane cache. Nor does it watch appliers: it reads the + `Kustomization` or `Application` on its tick to set `ApplierConfigured`, so a cluster without the + Flux or Argo CD CRDs needs no informer for them. - `WatchRuleReconciler` / `ClusterWatchRuleReconciler` watch `GitTarget` and `GitProvider`, populate the RuleStore, and trigger the rule-change reconcile. - `GitProviderReconciler` validates reachability and manages the signing key lifecycle. @@ -1500,7 +1575,12 @@ flowchart TD Hq -->|no| J[Configured-author: no fact streams or index; audit webhook skipped] I --> K[Setup + register Watch Manager] J --> K - K --> L[Register GitProvider + ClusterProvider + GitTarget + CommitRequest controllers] + K --> R{receiver-bind-address set?} + R -->|yes| Ri[Start push receiver listener; route deliveries to branch workers] + R -->|no| Rj[No receiver; foreign pushes found by rejection and refresh] + Ri --> L + Rj --> L + L[Register GitProvider + ClusterProvider + GitTarget + CommitRequest controllers] L --> M[Add cert watchers + health checks] M --> N[mgr.Start] ``` @@ -1559,6 +1639,15 @@ as one funnel and its loss paths as one selector: target `reason="publication"` is flat at zero. See [Interpreting metrics](interpreting-metrics.md#reading-git_fetches_total). +**Apply requests.** `apply_requests_total{applier_kind,reason,outcome}` counts every +request the worker decided to send (`applier_kind` is `kustomization`, `application` or +`webhook`; `reason` is `ForeignCommit`, `Refusal` or `CatchUp`), with `outcome` separating +`sent` from `failed` and `superseded` (a newer request for the same applier replaced it before it +went out). `receiver_deliveries_total{type,outcome}` counts receiver deliveries: +`foreign`, `own`, `ignored` (another branch, or a tag), and `rejected` (bad signature). A healthy +bi-directional branch shows `CatchUp` and `ForeignCommit` and never a request on a branch with +pending writes, which is what the decision rule promises. + **Authorship**, when `--author-attribution` is on: - **Ingress.** `gitopsreverser_audit_eventlist_duration_seconds{outcome}` times every request at @@ -1626,6 +1715,15 @@ Current limitations: is refused with more than one replica, and it loses every unjoined fact on restart by design; a multi-replica install must use the Redis transport. - **No pull request creation;** the operator writes directly to branches. +- **GitOps Reverser is the fast path to a deployment.** With `spec.applier` set, the applier's own + triggers are off, so while the operator is down a Git-side push reaches the cluster only through a + manual sync, `flux reconcile`, or Flux's long backstop interval. +- **The applier must live in the cluster it applies to.** It is looked up in the source cluster + with the `ClusterProvider`'s identity. A hub Argo CD whose `Application` lives in a management + cluster that is not the source is not supported. +- **An apply request cannot reserve time.** An edit made while a requested apply runs is reverted + and mirrored as the applier's commit. The rule confines that to the seconds after a foreign push, + a refusal with `Reapply`, or a quiet period on a Flux branch. - **Audit ingress uses a shared-CA trust boundary.** Named `/audit-webhook/` routes and the annotation-routed shared endpoint both require a client certificate signed by the audit CA, but that certificate is not bound to one provider. This is an accepted privileged-control-plane assumption, not @@ -1643,6 +1741,7 @@ Current limitations: | Package | Role | |---|---| | [api/v1alpha3/](../api/v1alpha3/) | CRD types | +| `internal/applier/` | apply requests: the Flux, Argo CD and webhook clients, and the `ApplierConfigured` check | | [cmd/](../cmd/) | operator entry point and server setup | | [internal/auditutil/](../internal/auditutil/) | audit identity, objectRef, and subresource helpers feeding attribution facts | | [internal/controller/](../internal/controller/) | Kubernetes reconcilers | @@ -1661,7 +1760,7 @@ Current limitations: | [internal/types/](../internal/types/) | shared resource identity/reference and sensitivity policy | | [internal/typeset/](../internal/typeset/) | type followability registry, lookup model, and the relevance filter (controller-owned → don't mirror) | | [internal/watch/](../internal/watch/) | discovery catalog, watch manager, per-`(GVR, scope)` raw watches with `sendInitialEvents` replay + mark-and-sweep, the author resolver, watched type tables, event router | -| [internal/webhook/](../internal/webhook/) | `/audit-webhook` ingress and attribution fact extraction (no body joiner) | +| [internal/webhook/](../internal/webhook/) | `/audit-webhook` ingress and attribution fact extraction (no body joiner), and the `/git-push` receiver | *** @@ -1688,6 +1787,7 @@ Deeper dives live under [docs/design/](design/): - [Kubernetes API resource catalog](facts/kubernetes-api-resource-catalog.md) - [GitTarget status design](spec/status-conditions-guide.md) - [The support contract: what we edit, what we refuse, and why](design/support-boundary/support-contract.md) +- [Push notification and the apply trigger](design/push-notification-and-reconcile-trigger.md) - [Git credentials interop](finished/git-credentials-interop.md) - [SOPS/age key management](finished/sops-repo-bootstrap-and-key-management-architecture.md) - [Commit signing](commit-signing.md) diff --git a/docs/bi-directional.md b/docs/bi-directional.md index ed4f23d3..88a2969d 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -1,93 +1,310 @@ # Bi-directional usage guide GitOps Reverser publishes live cluster state to Git. Flux or Argo CD applies Git back to the -cluster. Run both and a resource can be changed from either side. This guide covers what that -costs you, how to configure the two reconcilers for it, and which parts are proven by tests. - -**The short version.** Point your Git host's push webhook at the reconciler, turn Argo CD's -`selfHeal` off, and treat publishable live state as the input Reverser writes to Git. There is no -merge, no lock, and no timing guarantee. Choose what should happen when an edit cannot be -published: [leave it for intervention or request Git re-application](#choosing-speconrefusal). - -**What you are trading.** Round trips to the Git host are the seconds you wait through, so a -publication talks to the remote **twice**: it opens the push, reads where the branch is, and sends -the commits, never reading the branch beforehand. A publication with nothing to change costs one -request; a foreign push costs a handful of extra requests and a recomputation on a checkout -already on disk, and the branch still ends up correct. If your workload is mostly Git-side edits -with occasional live changes, you are using the tool against its grain. The per-operation numbers -are measured, in -[`git-roundtrip-ledger.golden`](../internal/git/testdata/git-roundtrip-ledger.golden). +cluster. Run both and a resource can be changed from either side. This guide covers who decides +what, how to configure each actor, and which parts are proven by tests. + +**The short version.** GitOps Reverser decides when the cluster is applied from Git. Flux and +Argo CD keep doing what they are good at (fetching, rendering, applying, pruning, reporting +health), but they apply when GitOps Reverser asks, not on their own schedule. You configure that +in three places: + +1. Name each folder's **applier** on its `GitTarget`: the Flux `Kustomization` or the Argo CD + `Application` that applies it. See [declaring the applier](#declaring-the-applier). +2. Switch off the applier's own triggers: Argo CD's automated sync, Flux's short intervals and any + Git-host webhook into Flux. See [configure Argo CD](#configure-argo-cd) and + [configure Flux](#configure-flux). +3. Point your Git host's push webhook at GitOps Reverser, not at the applier. See + [telling GitOps Reverser that somebody else pushed](#telling-gitops-reverser-that-somebody-else-pushed). + +There is no merge and no lock. What remains exposed is small and named in +[what is still exposed](#what-is-still-exposed). ```mermaid flowchart LR subgraph CLUSTER["Kubernetes cluster"] API["Kubernetes API"] REV["GitOps Reverser"] - REC["Flux / Argo CD"] + APP["Applier
Flux Kustomization or
Argo CD Application"] end GIT[("Git branch")] + HUMAN(["Somebody pushes to Git"]) API -->|"watch event"| REV REV -->|"commit, then compare-and-swap push"| GIT - GIT -->|"push webhook"| REC - REC -->|"apply"| API - GIT -. "push webhook: not built yet" .-> REV + HUMAN --> GIT + GIT -->|"push webhook"| REV + REV ==>|"apply request"| APP + APP -->|"fetch and apply"| API style REV fill:#e8f4fd,stroke:#2196f3 - style REC fill:#fff3e0,stroke:#ff9800 + style APP fill:#fff3e0,stroke:#ff9800 +``` + +## Who does what + +| Actor | Does | Does not | +| --- | --- | --- | +| Kubernetes API | Takes edits from people and controllers | | +| GitOps Reverser | Mirrors every edit into Git, and decides when the applier may apply | Apply anything to the cluster itself | +| Git host | Stores the branch, and tells GitOps Reverser when somebody else pushed | Notify the applier | +| Applier (Flux or Argo CD) | Fetches, renders, applies and prunes, when asked | Apply on a timer or a webhook of its own | +| A human | Can still run `flux reconcile` or press Sync | | + +## Why GitOps Reverser owns the trigger + +An apply is safe only when nothing live is missing from the commit being applied. Between an edit +and its push, the edit is live in the cluster and absent from Git: it sits in an open commit window +or in the pending writes. An apply at that moment writes Git's older value over it. GitOps Reverser +then mirrors that revert faithfully, as a commit authored by the applier, and the edit is gone. + +Only GitOps Reverser can see those pending edits. The applier cannot. So an apply nobody asked +GitOps Reverser about is an apply nobody checked, and every trigger the applier runs by itself is +one of those: + +- **A Git-host webhook into the applier** fires on every push, including GitOps Reverser's own. Two + API edits a few seconds apart are enough to have the first commit applied over the second edit. +- **Argo CD's timed refresh** (`timeout.reconciliation`, 120s plus up to 60s of jitter) resolves + the branch tip, and automated sync runs whenever that tip is a commit it has not synced. With + `selfHeal: false` it skips only a commit it already synced. Every push GitOps Reverser makes is a + new commit, so two to three minutes after each push the whole `Application` is synced over + whatever is pending at that moment. +- **Flux's intervals** apply on every reconcile, whatever woke it: kustomize-controller corrects + drift each time. A short `GitRepository` interval turns each of GitOps Reverser's pushes into an + apply a minute later. + +None of these can be filtered from outside. They have to be switched off, and GitOps Reverser has +to make the requests instead. + +## When GitOps Reverser requests an apply + +Every request waits for a **quiet branch**: no open commit window and no pending writes on the +branch worker. That is the moment nothing live is missing from the tip. + +| Reason | When | Which applier | +| --- | --- | --- | +| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push | Every applier declared on the branch | +| `Refusal` | An edit was refused and the target has `onRefusal: Reapply` | The refusing target's applier | +| `CatchUp` | The branch has been quiet for `spec.applier.quietPeriod` after GitOps Reverser's own pushes | Flux and webhook appliers only | + +GitOps Reverser never requests an apply of its own commits from an Argo CD `Application`: the +cluster already holds those values, because they were read from it. Flux is the exception, and +[configure Flux](#configure-flux) says why. + +A foreign commit that arrives while edits are pending does not wait for their windows to close. +GitOps Reverser closes them, replays the pending writes onto the foreign commit, pushes, and asks +the applier to apply the commit it just pushed: + +```mermaid +sequenceDiagram + actor Bob as Git-side author + participant Git + participant Rev as GitOps Reverser + participant App as Applier + participant API as Kubernetes API + + Note over Rev,API: Alice's edit W is live, in an open commit window + Bob->>Git: push F + Git-->>Rev: push webhook: the branch is at F + Rev->>Rev: close the window, fetch F, replay W onto F + Rev->>Git: compare-and-swap push F + W + Rev->>App: apply request: commit F + W, reason ForeignCommit + App->>API: apply F + W + Note over API: W's objects already hold W's values,
so only Bob's change lands +``` + +The applied commit carries the API side's answer, so applying it reverts nothing Alice did. That +is the [contested-cell rule](#what-happens-when-somebody-else-pushed-first) seen from the cluster: +for an object both sides changed, the API value wins. + +A request asks; it never reports that the apply happened. Requests are sent off the publication +path, retried on their own, and a failed request never fails a publication that succeeded. Read the +applier's own status to see what it applied. + +## Declaring the applier + +`spec.applier` names exactly one applier per `GitTarget`: + +```yaml +apiVersion: configbutler.ai/v1alpha3 +kind: GitTarget +metadata: + name: checkout + namespace: team-a +spec: + gitProviderRef: + name: platform-repo + branch: main + path: apps/checkout + applier: + kustomizationRef: # or applicationRef, or webhook + namespace: flux-system + name: checkout + onRefusal: Reapply +``` + +GitOps Reverser looks the applier up in the **source cluster**, the cluster it applies to, with +the identity the `ClusterProvider` already uses. That identity needs permission to read and patch +the applier; the chart grants it for the operator's own cluster. The applier must apply this +folder's repository and branch, or the target reports `ApplierConfigured=False`. The +[configuration reference](configuration.md#who-applies-this-folder-specapplier) has every field +and reason. + +Several `GitTarget`s may name the same applier, for example an `Application` whose path covers +several folders. Requests to one applier object are sent once. + +`webhook` is for any other tool: GitOps Reverser sends a signed POST naming the repository, the +branch, the commit and the reason, and the receiver decides what to do with it. + +## Configure Argo CD + +Switch automated sync off and keep its settings. Argo CD 3.1 and later has a field for exactly +that: + +```yaml +spec: + syncPolicy: + automated: + enabled: false # GitOps Reverser makes the sync requests + prune: true # still read: GitOps Reverser's syncs prune when this is true + syncOptions: + - ServerSideApply=true +``` + +GitOps Reverser's request is the sync operation Argo CD's own automated sync would have made, +pinned to one commit: `prune` from `automated.prune`, `syncOptions` and `retry` from the +`Application`, and `initiatedBy` naming GitOps Reverser. It waits for a running operation to +finish rather than replacing it. Because the commit is pinned, the sync applies exactly the commit +GitOps Reverser checked, not whatever the tip has become by the time it runs. + +With automated sync off, a refresh only compares: the timed refresh, a Git-host webhook into +Argo CD, and the UI's Refresh button all report `Synced` or `OutOfSync` and apply nothing. So +`timeout.reconciliation` can stay at its default, and a Git-host webhook into Argo CD is optional; +it only makes the comparison status fresher. `selfHeal` has nothing left to act on. + +On an Argo CD older than 3.1, omit `automated` altogether. GitOps Reverser's syncs then do not +prune. + +Every sync runs the `Application`'s resource hooks, which is the other reason GitOps Reverser never +syncs an `Application` for its own commits. + +### The split-ownership alternative + +`ignoreDifferences` on a field, with `RespectIgnoreDifferences=true` in `syncOptions`, hands that +one field to the API. Its differences alone do not make the `Application` `OutOfSync`, and the +option copies the live value into the apply target, so GitOps Reverser's syncs preserve it. + +The cost is that the field stops being Git-driven on an existing object: Git seeds it at creation, +but later Git edits to it do not reach the object while the ignore is active. It protects no other +field from GitOps Reverser's whole-object replay. + +## Configure Flux + +Flux has no refresh-only mode: kustomize-controller applies whenever it reconciles, and +`suspend: true` also blocks the requests GitOps Reverser makes. So the configuration keeps Flux's +timers as a long backstop and removes every short path: + +```yaml +# GitRepository: long. GitOps Reverser requests the fetches. +spec: + interval: 24h +--- +# Kustomization: long, and unsuspended. +spec: + interval: 24h + retryInterval: 1m + suspend: false ``` -Both solid legs are configuration you control. The dashed leg does not exist: nothing tells -Reverser that somebody else moved the branch. [Telling Reverser that somebody else -pushed](#telling-reverser-that-somebody-else-pushed) covers what that costs and how to work -around it. +Remove any Git-host webhook into a Flux `Receiver` for this repository. GitOps Reverser's request is +the one `flux reconcile kustomization --with-source` makes: it stamps +`reconcile.fluxcd.io/requestedAt` on the `GitRepository`, and stamps the `Kustomization` too when +the artifact already holds the requested commit. + +Flux needs one request Argo CD does not. Its timers and a controller restart apply whatever artifact +the `GitRepository` holds, and with a long interval that artifact falls behind GitOps Reverser's own +pushes. Applying a stale artifact would revert published edits, not only pending ones. So after its +own pushes, once the branch has been quiet for `spec.applier.quietPeriod` (`30s` by default), +GitOps Reverser asks the `GitRepository` to fetch, with reason `CatchUp`. The new artifact wakes the +`Kustomization`, which applies values the cluster already holds. + +`quietPeriod` is a trade. A short one keeps the artifact current; a long one makes it less likely +that an edit arrives while that catch-up apply is running. Every new edit restarts it. + +Flux's equivalent of split ownership is `Kustomization.spec.ignore`, added in kustomize-controller +`v1.9.0`, with the same trade as Argo CD's. The +[source review](facts/gitops-apply-and-field-ignore.md) has the code paths. + +When upgrading a chart whose pin is declared in Git, reconcile the owning `Kustomization` first +so it applies the new `HelmRelease` and source configuration, then the `HelmRelease` if needed. + +## Telling GitOps Reverser that somebody else pushed + +GitOps Reverser learns that the branch moved in three ways, and each feeds the same decision: + +| Signal | Latency | Configure | +| --- | --- | --- | +| A push webhook from the Git host | Seconds | `GitProvider.spec.receiver` | +| A rejected compare-and-swap push | At the next publication | Nothing: it is how every push works | +| The periodic refresh | Within `--git-refresh-interval` (10m) plus the 5-minute reconcile tick | Nothing: on by default | + +Point the Git host's push webhook at the URL in `GitProvider.status.webhookPath`, served on the +receiver listener. The receiver speaks GitHub's, GitLab's and Gitea's native push payloads, plus a +`generic-hmac` body for a relay you run yourself. See +[receiving push notifications](configuration.md#receiving-push-notifications-specreceiver). + +A delivery is only a hint. GitOps Reverser does not trust the commit it names: it reads the +branch's ref advertisement (one request) and decides from what the remote says. So a late delivery +for one of GitOps Reverser's own pushes, a duplicate, or a rollback onto one of its older commits +all come out right. When it cannot tell whether a commit is its own, it treats it as foreign. + +Without a receiver everything still works, later: a foreign push is applied once the refresh finds +it. + +**A notification that the branch moved never triggers a cluster snapshot.** The cluster still holds +the values from before the push, so a snapshot would publish them over the incoming change. A push +tells GitOps Reverser about Git; it says nothing about the cluster. + +Read where the branch is, what proved it, and what was last requested: + +```bash +kubectl get gittarget checkout -n team-a \ + -o jsonpath='{.status.remote}{"\n"}{.status.applier}{"\n"}' +# {"commit":"4f2c1ab9...","lastVerifiedAt":"2026-10-20T10:14:02Z","verifiedBy":"Fetch"} +# {"requestedCommit":"4f2c1ab9...","requestedAt":"2026-10-20T10:14:03Z","reason":"ForeignCommit"} +``` ## One queue per branch Every watched event for a given `GitProvider` and branch lands on one queue owned by one branch worker. Events are applied in arrival order and never overtake each other, and two `GitTarget` -objects on the same provider and branch share that worker and its ordering. +objects on the same provider and branch share that worker, its ordering, and its quiet moments. 1. **Commit.** Events collapse into one local commit per author after [`spec.commit.window.idleTimeout`](configuration.md#the-commit-window-speccommitwindow) of silence, `5s` by default, or at most `maxDuration` (`1m`) after the first. `0s` gives one commit per event. 2. **Push.** Local commits go to the remote at most once every five seconds, as a - compare-and-swap. The push declares the SHA it expects the branch to be at, and the Git host + compare-and-swap. The push declares the commit it expects the branch to be at, and the Git host refuses the update if the branch has moved off the commits' base. -That compare-and-swap is what detects a foreign push, and it does so from the remote's ref -advertisement before uploading a single object, on a connection the cycle was making anyway. - -**A target that is publishing reads the remote only through the push it was making anyway.** That -is measured rather than approximate, and no interval setting changes it: the push session reads the -advertisement and the server names the hash it accepted, so the knowledge is free. - -An IDLE target is the one exception, and it is a setting: -[`--git-refresh-interval`](configuration.md#keeping-an-idle-target-fresh---git-refresh-interval) -has it re-prove where its branch is, by default every 10 minutes. That costs one ref advertisement -per idle branch per interval, and a fetch only when the branch has actually moved. `0` turns it off -and returns an idle target to no Git traffic at all. - -Planning does not read the branch either: the worker plans on the checkout it already has and lets -the compare-and-swap catch a move. The exceptions are a worker that has not yet seen the remote, a -previous write that failed and left the checkout in an unknown state, and a resync, which keeps -reading because it can finish without ever opening a push. +A publishing target reads the remote only through the push it was making anyway. An idle target +re-proves where its branch is every +[`--git-refresh-interval`](configuration.md#keeping-an-idle-target-fresh---git-refresh-interval), +at the cost of one ref advertisement, and fetches only when the branch actually moved. ## What happens when somebody else pushed first -Your push is rejected and the worker **replays**: it fetches the new tip, re-plans every retained -write against it, and pushes again. Three attempts, then the writes stay retained for a later try. - -Replaying is the cheap half: the re-planning is local and the branch ends up correct either way. -What a rejection costs is the extra conversations with the Git host around it, which is what -[a webhook pointed back at Reverser](#telling-reverser-that-somebody-else-pushed) would remove. +Your push is rejected and the worker **replays**: it fetches the new tip, re-plans every pending +write against it, and pushes again. Three attempts, then the writes stay pending for a later try. +Whatever the worker learned along the way, it now knows the branch gained a foreign commit, and the +appliers are asked to apply once the replayed push lands. ```mermaid flowchart TD - PUSH["Push, declaring the SHA the commits were based on"] --> MOVED{"Is the branch still at that SHA?"} + PUSH["Push, declaring the commit the writes were based on"] --> MOVED{"Is the branch still there?"} MOVED -->|yes| DONE["Accepted"] MOVED -->|"no: somebody else pushed"| FETCH["Fetch the new tip"] - FETCH --> REPLAN["Re-plan every retained write against the new tree"] + FETCH --> REPLAN["Re-plan every pending write against the new tree"] REPLAN --> DIFF{"Does the captured object still differ from what Git now holds?"} DIFF -->|no| NOOP["No commit. The other writer's value stands."] DIFF -->|yes| COMMIT["Commit on top, then push again"] @@ -97,20 +314,16 @@ flowchart TD style NOOP fill:#e8f5e9,stroke:#43a047 ``` -Replay re-plans, it does not rebase a diff. Each retained write still carries the object as the +Replay re-plans; it does not rebase a diff. Each pending write still carries the object as the watch captured it, and writing that object onto the new tree has one of two outcomes: - **No change, so no commit.** The file in Git already says what the captured object says, - because whoever pushed made the same change. Git had it first and that is the end of it. + because whoever pushed made the same change. - **A change, committed on top.** The captured object is written over the other writer's value, and the branch ends at the API side's value. -There is no third outcome for a write that has a legal destination at all — one that does not is -refused, and the table below says what that leaves on disk — and there is no merge: a watch event -carries the object's whole state, with no retained baseline that would say which fields changed on -each side. The -[deferred merge investigation](future/git-api-three-way-comparison.md#what-three-way-merge-means-here) -defines the comparison that would take. +A write with no legal destination is refused instead, and there is no merge: a watch event carries +the object's whole state, with no baseline that would say which fields changed on each side. ### What a write does to the file @@ -138,9 +351,8 @@ objects are independent of each other. Separate fields of one object are not. ## There are no timing guarantees Committing and pushing takes on the order of a second at best, and the Kubernetes API accepts a -great many changes in a second. Nothing paces the two against each other and nothing reserves a -window, so treat live editing as a different way of making changes rather than a faster -`git push`: +great many changes in a second. Treat live editing as a different way of making changes rather +than a faster `git push`: - You make a change and then observe that it arrived. You do not save and receive a confirmation. - Intermediate values may never reach Git. Two edits inside one commit window produce one commit @@ -151,310 +363,127 @@ When a specific change has to be confirmed in Git, use a [`CommitRequest`](configuration.md#commitrequest). `Pushed=True` with `status.commit` and `status.branch` names the commit it produced. -## Why the loop stops +### What is still exposed -The obvious fear is a feedback loop: Reverser commits, the reconciler applies, the apply produces -a watch event, Reverser commits again. With no intervening edits or admission changes, it settles -instead. +Owning the trigger removes every apply that could revert an edit for no reason. It cannot reserve +the seconds an apply takes: -```mermaid -sequenceDiagram - actor Alice - participant API as Kubernetes API - participant Rev as GitOps Reverser - participant Git as Git branch - participant Rec as Flux / Argo CD - - Alice->>API: set replicas to 5 - API-->>Rev: watch event, replicas 5 - Rev->>Git: commit and push - Git-->>Rec: push webhook - Rec->>Git: fetch the new revision - Rec->>API: apply, replicas 5 - Note over API: live value is already 5,
so the apply changes nothing - API-->>Rev: watch event, if one is produced at all - Note over Rev: sanitized content equals
the file in Git, so no commit -``` +- **An edit made while a requested apply runs** is reverted, then mirrored as the applier's + commit. That happens only after somebody else pushed, after a refusal with `Reapply`, or after a + `quietPeriod` of silence on a Flux branch. It no longer follows every commit GitOps Reverser + makes. +- **Flux's backstop timers** still apply on their own, at the tip as the last catch-up left it. +- **A human** running `flux reconcile` or pressing Sync applies whenever they do. + +## Why the loop stops + +The obvious fear is a feedback loop: GitOps Reverser commits, the applier applies, the apply +produces a watch event, GitOps Reverser commits again. It settles instead, for two reasons. -When the reconciler applies a commit Reverser wrote, the live object already holds those values, -because they were read from that object in the first place. The apply changes nothing, the event -it produces carries content identical to the file already in Git, and no commit is made. In -general: **applying a resource that already matches does not produce a change.** What can break -that reasoning is controller bookkeeping. +GitOps Reverser does not ask for its own commits to be applied, except Flux's catch-up. And when a +commit it wrote is applied, the live object already holds those values, because they were read +from that object in the first place. The apply changes nothing, the event it produces carries +content identical to the file already in Git, and no commit is made. In general: **applying a +resource that already matches does not produce a change.** What can break that reasoning is +controller bookkeeping. ## Keep controller bookkeeping out of Git Argo CD stamps `argocd.argoproj.io/tracking-id` on every non-CRD object it applies, Flux stamps `kustomize.toolkit.fluxcd.io/` labels and annotations, and a client-side apply writes `kubectl.kubernetes.io/last-applied-configuration`. None of it is user intent. If it reached Git -the sanitized live object would stop matching the committed file, and every reconciler sync would -produce a commit. +the sanitized live object would stop matching the committed file, and every apply would produce a +commit. -Reverser strips all of it before writing, along with `argocd.argoproj.io/installation-id`, +GitOps Reverser strips all of it before writing, along with `argocd.argoproj.io/installation-id`, `kro.run/`, `applyset.kubernetes.io/`, and `kcp.io/cluster`. The rules and the reasoning for each are in [`internal/sanitize/types.go`](../internal/sanitize/types.go). Under `argocd.argoproj.io/` only the exact tracking keys go: siblings such as `sync-wave`, `sync-options`, and `hook` are user intent that belongs in Git. -**Leave Argo CD on its default `annotation` resource tracking for Reverser-managed paths.** The -`label` and `annotation+label` methods stamp `app.kubernetes.io/instance`, which is -indistinguishable from the standard recommended label that Helm and Kustomize set for application -reasons. Reverser therefore does not strip it, and label tracking would put Argo's bookkeeping -into your commits. - -## Configure Argo CD - -`selfHeal` is the setting that decides what happens to a live edit. - -| `selfHeal` | What happens to a live edit | Use when | -| --- | --- | --- | -| `true` | Argo reverts it, in about a second in our e2e corner, usually before Reverser can publish. The edit is lost and Git records a two-commit flap. | The path is Git-owned and drift is an error | -| `false` | It survives, is captured to Git, and is applied onward from there. Both directions work on the same field. | Anything edited through the API | - -Both rows are measured outcomes. Neither is an ordering guarantee: self-heal is watch-driven and the first -revert of fresh drift has zero backoff, while Argo notices a new Git revision only on its timed -refresh (`timeout.reconciliation`, 120s plus up to 60s of jitter) or when a webhook tells it. And -`selfHeal: false` only suppresses repeating a sync of the same revision and parameters, so a new -revision or a manual sync still applies Git values over an unpublished API edit. - -There is no knob that delays that first revert. -[Why `selfHeal` cannot be made selective](design/support-boundary/argocd-bi-directional.md) walks -the code paths, including the options that look like they would work, and the -[apply and field-ignore source review](facts/gitops-apply-and-field-ignore.md) separates Argo's -scheduling, comparison, and write paths. - -```yaml -spec: - syncPolicy: - automated: - selfHeal: false - prune: true -``` - -Use `prune: true` only when a deletion in Git should delete the live object. Then configure the -[Argo CD webhook](https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/) on your Git -host, pointing at `https:///api/webhook` with a shared secret. In the e2e corner -that path drives a sync about four seconds after the push. Without it you wait for the timed -refresh. A webhook only requests a refresh: an apply follows only where a sync is permitted, so it -does not substitute for automated sync. - -### The split-ownership alternative - -`ignoreDifferences` on a field, with `RespectIgnoreDifferences=true`, keeps `selfHeal: true` -everywhere else and hands that one field to the API. Its differences alone do not make the -Application `OutOfSync`, and the respect option copies the live value into the apply target, so an -ordinary sync preserves it. Without that option, a later sync can still write the field from Git. - -The cost is that the field stops being Git-driven on an existing object: Git seeds it at creation, -but later Git edits to it do not reach the object while the ignore is active. It buys authority for -one named field, and protects no other field from Reverser's whole-object replay. The e2e corner -checks that an API edit survives and is published back to Git; it does not test a later Git edit to -the ignored field. - -## Configure Flux - -Flux has no `selfHeal` toggle: kustomize-controller corrects drift on every reconcile, so a live -edit survives until the next apply of a revision that still carries the old value. The two -settings that matter are therefore the apply interval and how quickly a push reaches Flux. - -```yaml -# GitRepository: short, so a missed webhook becomes a delay rather than a stall. -spec: - interval: 1m ---- -# Kustomization on the shared path: long, so the timer is not what reverts live edits. -spec: - interval: 24h - retryInterval: 1m - suspend: false -``` - -Configure a [Flux `Receiver`](https://fluxcd.io/flux/guides/webhook-receivers/) against the -**`GitRepository`**, not the Kustomization. Source-controller fetches Git and publishes a new -artifact revision, which wakes the dependent Kustomizations. A Receiver aimed at the Kustomization -can re-apply the artifact Flux already has cached, which is the old content. - -A long interval delays periodic correction. It reserves nothing: a new source revision, a -configuration change, a retry, or a controller restart all trigger an apply sooner. Keep the -Kustomization **unsuspended**, because suspension blocks source-triggered applies as well and a -manual reconcile request does not bypass it. To trigger the source and the apply together: - -```bash -flux reconcile kustomization editing -n flux-system --with-source -``` - -When upgrading a chart whose pin is declared in Git, reconcile the owning `Kustomization` first -so it applies the new `HelmRelease` and source configuration. Then reconcile the `HelmRelease` if -needed. `flux reconcile helmrelease --with-source` refreshes its configured chart source; it does -not first apply the Git manifests that change that configuration. A successful reconcile can -therefore still use the old pin. - -Flux's equivalent of split ownership is `Kustomization.spec.ignore`, added in kustomize-controller -`v1.9.0`, with the same trade as Argo's: selected live fields survive later applies, Git seeds them -at creation, later Git edits to them do not land, and Reverser still captures them back into Git. -Our e2e installs Flux `2.9.6` but no spec exercises `spec.ignore`, so it is upstream behavior we -have read in the source and have no test of our own behind; the -[source review](facts/gitops-apply-and-field-ignore.md) has the code paths. - -## Telling Reverser that somebody else pushed - -Everything above points a webhook from the Git host at the reconciler. The other direction does -not exist yet, so what a moved branch costs depends on what the target is doing: - -- **Actively writing:** nothing breaks. The compare-and-swap catches the move and replay handles - it. It is only slower than it needs to be, because the move is discovered by a push that was - always going to fail. -- **Refused:** it recovers on its own. While `GitPathAccepted` is `False` the target is not - converged, so it requeues every 10 seconds and each pass forces a re-read. Fix an unsupported - folder in Git and the condition clears within about ten seconds. -- **Healthy and idle:** it finds out on its own, within `--git-refresh-interval` (10m by default) - plus the 5-minute reconcile tick that carries the request. The target spends one ref - advertisement, fetches only if the branch actually moved, and republishes `status.placement`; - `status.remote` follows on a later tick, since it is sampled rather than pushed. If that push - left content the writer cannot handle, the same read sets - `GitPathAccepted=False` — which moves the target into the refused row above, where it re-checks - every ten seconds until somebody fixes the folder. The refresh writes no content of its own. - -Read where a branch is, and when that was last proved: - -```bash -kubectl get gittarget editing -n gitops-reverser \ - -o jsonpath='{.status.remote}{"\n"}' -# {"revision":"4f2c1ab9...","lastVerifiedAt":"2026-09-23T10:14:02Z","verifiedBy":"Fetch"} -``` - -`verifiedBy: Fetch` next to a revision none of your publications produced is how a foreign push is -read off `kubectl`; `Push` means the revision is Reverser's own work and the server took it. - -Ask for a re-read **now**, rather than waiting out the interval. This is the one that also forces a -full re-check, so it is the tool for a refusal you have just fixed: - -```bash -kubectl annotate gittarget editing -n gitops-reverser \ - reconcile.configbutler.ai/requestedAt="$(date +%s)" --overwrite -``` - -A receiver that does this automatically is designed in -[inbound push notification](design/push-notification-and-reconcile-trigger.md); the other half of -that design, removing the head-of-cycle fetch, has shipped. Its request shape is -[§8.3, the wire contract](design/push-notification-and-reconcile-trigger.md#83-the-wire-contract-for-whoever-calls-it): -one signed `POST /git-push/` carrying the repository, the branch, and the SHA the branch now -points at. It deliberately does not take a Git host's native payload, so the mapping from your -host's webhook belongs in the host's own configuration or in a relay. - -One rule matters if you build anything similar yourself: **a notification that a branch moved must -not trigger a fresh cluster snapshot.** Our handler and the reconciler's both fire on the same -push, and ours has less work to do, so it would reliably publish the pre-push cluster state over -the incoming change before the reconciler ever applied it. That is also why the design does not -wire a webhook to the annotation above, which asks for exactly such a resync. A push tells you -about Git; it says nothing about the cluster. +**Leave Argo CD on its default `annotation` resource tracking for mirrored paths.** The `label` +and `annotation+label` methods stamp `app.kubernetes.io/instance`, which is indistinguishable from +the standard recommended label that Helm and Kustomize set for application reasons. GitOps Reverser +therefore does not strip it, and label tracking would put Argo CD's bookkeeping into your commits. ## When a write is refused or a push fails A live edit can succeed while its publication does not. The cluster holds the change and Git does -not, and **a refused write writes no file, so nothing reaches the reconciler through the ordinary -path**. Watch push failures and the target's write-acceptance conditions: a successful Kubernetes +not. Watch push failures and the target's write-acceptance conditions: a successful Kubernetes edit confirms the API write and nothing beyond it. -Retained writes are held in worker memory. They do not survive a restart, and an edit that failed -before becoming a pending write is gone from Reverser's side. Recovery is a resync, which derives -state from the cluster as it exists at that moment, so it cannot reproduce an intermediate value -that has since been overwritten. +Pending writes are held in worker memory. They do not survive a restart, and an edit that failed +before becoming a pending write is gone from GitOps Reverser's side. Recovery is a resync, which +derives state from the cluster as it exists at that moment, so it cannot reproduce an intermediate +value that has since been overwritten. ### Choosing `spec.onRefusal` -**Enable `PushEmptyCommit` when restoring Git state takes priority over keeping unpublished live -edits.** It is useful on a dedicated demo or editing branch where users accept that trade and an -active GitOps reconciler applies the branch. Keep the default `Ignore` when live edits need human -review before being discarded, or when unrelated workloads share the branch and their owners have -not agreed to an extra apply. +A refused edit has no legal destination in Git, so nothing is committed for it and it stays live +until something applies Git over it. | Policy | Response to a refused edit | Operator's trade | | --- | --- | --- | -| `Ignore` (default) | Report the refusal without moving the branch | Leave recovery to a human or a later apply | -| `PushEmptyCommit` | For eligible edits, publish an empty commit requesting Git re-application | Accept an extra apply across the branch, including over unpublished edits | - -[`spec.onRefusal: PushEmptyCommit`](configuration.md#reverting-a-refused-edit-speconrefusal) -changes no file. The new revision gives Flux or Argo CD a reason to apply Git again. Restoration -depends on the reconciler observing the revision and successfully applying it. GitOps Reverser -does not write the old value back to the Kubernetes API itself. - -Eligibility is narrow: the folder must be accepted, the refused edit must concern an existing -document, and the write must remove no document. Invalid or unsupported folders, new objects, and -removals do not trigger it. A standing refusal is deduplicated; rechecks do not keep producing empty -commits for the same observation. A failed push is a separate failure and this policy does not -repair Git connectivity. - -The apply still follows the reconciler's configuration. Flux must be unsuspended; Argo CD needs -automated sync and an `OutOfSync` Application. An Argo CD Application using -`argocd.argoproj.io/manifest-generate-paths` can skip the empty revision because no relevant file -changed. Fields protected by the split-ownership settings above remain protected. A webhook -reduces discovery delay but gives no completion guarantee. - -### How a refused edit can undo an allowed one - -The setting belongs to a `GitTarget`, but the new revision can wake every consumer of its branch. -Separate folders or targets on that branch do not isolate this effect. One possible ordering is: - -1. Alice makes an allowed live edit. It is still in an open commit window, so Git has the old value. -2. Bob makes a refused edit whose window closes first. Reverser pushes an empty commit for it. -3. The reconciler applies that revision, restoring Git's values over both live edits. -4. Reverser observes the restoration. Alice's earlier value may never reach Git. - -Alice and Bob can be the same person, and the edits can affect different objects. The empty commit -does not erase accepted commits already included in the published revision; the risk is live -state missing from the revision being applied. The same race exists whenever a new Git revision -arrives. `PushEmptyCommit` adds a trigger at the moment a publication was refused. Shortening the -commit window reduces exposure but does not reserve time against an apply. An extra apply can also -advance unrelated work, including pruning already requested in Git. +| `Ignore` (default) | Report the refusal | Leave recovery to a human or a later apply | +| `Reapply` | Ask the target's applier to apply the current tip, once the branch is quiet | The refused edit is reverted; nothing else on the branch is | + +`Reapply` needs a declared applier, and the request goes to that applier alone: other folders on +the branch are not woken. Waiting for a quiet branch means an allowed edit still in a commit window +is published first, so the apply reverts only what has no home in Git. + +Eligibility is narrow: the folder must be accepted, the refused edit must concern a document the +folder already holds, and the write must remove no document, because re-applying would put back an +object somebody deliberately deleted. Invalid or unsupported folders, new objects and removals do +not trigger it. A standing refusal is requested once, not on every recheck, and a suspended target +requests nothing. See +[reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal). ### A restored value does not prove the target recovered -`Ignore` still reports `GitPathAccepted=False`, `Stalled=True`, and `Ready=False`. The target keeps -rechecking, roughly every ten seconds; `Stalled` does not mean the controller has stopped running. -Normal publication can remain blocked, and changing `onRefusal` does not bypass acceptance or -render-fidelity checks. +A refusal still reports `GitPathAccepted=False`, `Stalled=True`, and `Ready=False`, under either +policy. The target keeps rechecking, roughly every ten seconds; `Stalled` does not mean the +controller has stopped running. -An empty commit neither clears those conditions nor makes an unrepresentable edit writable. A -successful resync covering the refused scope must establish acceptance again. Re-applying Git can -remove the offending live difference, but other differences can remain: API defaults on a -Deployment inherited from a read-only base can still have no writable destination in the overlay. +Re-applying neither clears those conditions nor makes an unrepresentable edit writable. A +successful resync covering the refused scope must establish acceptance again. Treat recovery as two +observations: the live resource has the intended value, and the `GitTarget` returns to +`Ready=True`. -Treat recovery as two observations: the live resource has the intended value, and the `GitTarget` -returns to `Ready=True`. If it remains refused, use the condition's reason to correct the folder, -the live state, or what the target watches. Leaving the target stalled indefinitely is an -unresolved publication failure under either policy. +## What it costs + +**GitOps Reverser becomes the only fast path to a deployment.** While it is down, a push by a +human reaches the cluster only through a manual sync, `flux reconcile`, or Flux's backstop +interval. Bi-directional editing is already stopped in that state; Git-side deployments now slow +down with it. Monitor the operator the way you monitor the applier. + +**Round trips to the Git host are the seconds you wait through.** An uncontended publication talks +to the remote twice, and a foreign push costs a fetch, a replay on a checkout already on disk, and +one push. If your workload is mostly Git-side edits with occasional live changes, you are using the +tool against its grain. The per-operation numbers are measured, in +[`git-roundtrip-ledger.golden`](../internal/git/testdata/git-roundtrip-ledger.golden). ## What the tests prove -The bi-directional corner runs against deployed Flux and Argo CD. Its timing and event-count -assertions describe those scenarios; they do not guarantee the ordering of concurrent writers. +The bi-directional corner runs against deployed Flux and Argo CD, each configured as this guide +says. Its timing assertions describe those scenarios; they do not guarantee the ordering of +concurrent writers. | Covered | Where | | --- | --- | -| Argo bookkeeping never reaches Git, asserted as zero commits during a sync | [Argo phase 1](../test/e2e/argocd_bi_directional_e2e_test.go) | -| `selfHeal: true` loses the API edit, and the flap is exactly two commits | Argo phase 2 | -| `ignoreDifferences` keeps the edit and holds the Application `Synced` | Argo phase 3 | -| `selfHeal: false` keeps the API edit and still applies a Git-side change to the same field | Argo phase 4 | -| Git to cluster to Git round trip settles without a commit loop | [Flux](../test/e2e/flux_bi_directional_e2e_test.go) | +| Argo CD bookkeeping never reaches Git, asserted as zero commits during a sync | [Argo CD](../test/e2e/argocd_bi_directional_e2e_test.go) | +| Two API edits to one field seconds apart end at the later value, with no commit authored by the applier | Argo CD and [Flux](../test/e2e/flux_bi_directional_e2e_test.go) | +| With a short `timeout.reconciliation`, Argo CD applies nothing GitOps Reverser did not request | Argo CD | +| A Git-side push while an API edit is pending lands both, and the applier applies the replayed commit | Argo CD and Flux | +| After a quiet period the `GitRepository` artifact holds the tip, and a forced `Kustomization` reconcile changes nothing | Flux | +| A refused edit with `onRefusal: Reapply` is reverted, and only its own applier is asked | Argo CD and Flux | +| With no receiver configured, a foreign push is still applied after the refresh finds it | Flux | | A SOPS-encrypted Secret round-trips without a reverse commit | Flux | | A Git revert removes the live object and the committed file | Flux | -| Waking only the `GitRepository` makes the Kustomization apply a new revision by itself, reverting live drift, with an **empty** commit as that revision | Flux | -| An eligible refusal produces an empty commit; unchanged refusals settle, and a new refused value produces another commit | [Refusal action](../test/e2e/refusal_empty_commit_e2e_test.go) | - -The refusal-action spec and the Flux spec prove the two halves separately. They do not exercise an -actual refusal through reconciler restoration to `GitTarget` recovery in one scenario, or an -allowed edit racing the empty commit's apply. -Two other gaps worth knowing. The **Flux spec drives reconciliation manually** with 30m intervals, so -only the Argo webhook is exercised; the mechanism a `Receiver` depends on is covered by the Flux -test that wakes the `GitRepository` alone and lets the Kustomization apply on its own. And **no -spec tests simultaneous writes to one field from both sides**, because the outcome depends on +**No spec tests simultaneous writes to one field from both sides**, because the outcome depends on arrival order the corner does not control. The replay mechanism is covered; a conflict policy is not, because there is not one. -Run `task test-e2e-bi-directional` for the corner, which is also a CI job. `task test-e2e` -excludes the corner and includes the refusal-action spec. `task argocd-ui` opens the installed -Argo CD UI. The -[corner specification](spec/e2e-bi-directional-corner.md) describes the setup and the coverage -still missing. +Run `task test-e2e-bi-directional` for the corner, which is also a CI job. `task argocd-ui` opens +the installed Argo CD UI. The [corner specification](spec/e2e-bi-directional-corner.md) describes +the setup. diff --git a/docs/configuration.md b/docs/configuration.md index 9b091d1d..8b5b15ca 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -32,6 +32,10 @@ The usual flow is: 3. Create one or more `WatchRule` or `ClusterWatchRule` objects that point at that target. 4. Create a `CommitRequest` only when you want a save: your writes in one commit, on the request's timers rather than the target's, and optionally with your message. +5. When the folder is also applied back to the cluster by Flux or Argo CD, name that applier on the + `GitTarget` and give the `GitProvider` a push receiver, so GitOps Reverser decides when Git is + applied. See [who applies this folder](#who-applies-this-folder-specapplier) and the + [bi-directional guide](bi-directional.md). That means one repository connection can back multiple targets, and one target can be fed by multiple watch rules. @@ -389,6 +393,53 @@ want custom `spec.commit` behavior because the starter values do not currently e For the platform-facing behavior behind "valid signature" versus "verified badge", see [commit-signing.md](commit-signing.md). +### Receiving push notifications (`spec.receiver`) + +`spec.receiver` lets the Git host tell GitOps Reverser that somebody else pushed. The names follow +Flux's `Receiver`, so a Flux user already knows them: + +```yaml +spec: + url: https://github.com/acme/platform.git + receiver: + type: github # github | gitlab | gitea | generic-hmac + secretRef: + name: platform-receiver-token # key: token +status: + webhookPath: /git-push/team-a/platform-repo +``` + +Point the host's push webhook at the receiver listener plus `status.webhookPath`, with the same +token as the webhook secret. Each type verifies the request the way its host signs it: + +| `type` | Verifies | Body | +| --- | --- | --- | +| `github` | `X-Hub-Signature-256`, an HMAC of the body | GitHub's push event | +| `gitlab` | `X-Gitlab-Token`, compared with the token | GitLab's push event | +| `gitea` | `X-Gitea-Signature`, an HMAC of the body | Gitea's push event | +| `generic-hmac` | `X-Signature: sha256=`, an HMAC of the body | `{"repository": "...", "branch": "main", "after": ""}` | + +`generic-hmac` is for a relay, such as one GitHub App webhook fanned out to many repositories. Its +header is the one Flux's `generic-hmac` `Receiver` checks. + +The signature is checked before anything in the body is read, so an unsigned request cannot probe +which repositories or branches are tracked. A verified delivery answers `202 Accepted` even when it +names a branch no `GitTarget` writes, because that is not the host's fault and an error teaches a +host to disable the hook. Pushes to other branches and tag pushes are accepted and ignored. + +A delivery is a hint, never a fact. It makes the branch worker read the branch's ref advertisement +(one request), and what the remote says decides whether the branch gained a foreign commit. A late +delivery for one of GitOps Reverser's own pushes, a duplicate, or a rollback onto one of its older +commits are all judged by the remote's answer, not by the commit the delivery names. Nothing fetches +or plans until there is something to apply, and a delivery never triggers a cluster snapshot. + +The receiver needs its own listener, because the Git host cannot present the client certificate +the audit listener requires. See [push receiver settings](#push-receiver-settings). + +Without a receiver, GitOps Reverser still learns of a foreign push, from its next rejected push or +from the [periodic refresh](#keeping-an-idle-target-fresh---git-refresh-interval). The receiver +makes it seconds instead of minutes. + ## `ClusterProvider` `ClusterProvider` names the Kubernetes cluster a `GitTarget` mirrors **from**. It is the read-side @@ -570,6 +621,12 @@ The important fields are: omit it and each document's namespace is inferred from the folder - `spec.prune`: which deletion paths may remove documents from this target's folder (see [Deletion policy](#deletion-policy-specprunemode)); omit it for the safe default +- `spec.applier`: the Flux `Kustomization` or Argo CD `Application` that applies this folder, so + GitOps Reverser decides when it applies (see + [Who applies this folder](#who-applies-this-folder-specapplier)); omit it when nothing applies the + folder back +- `spec.onRefusal`: whether a refused edit asks the applier to reapply Git (see + [Reverting a refused edit](#reverting-a-refused-edit-speconrefusal)) Example: @@ -624,6 +681,9 @@ The most useful status fields are: - `status.retention`: how many documents `spec.prune.mode` is keeping, and under which mode. - `LayoutResolved`: what the last scan resolved about the folder's shape, with `status.placement` carrying the detail. See below. +- `ApplierConfigured`: true when the declared applier exists, applies this folder's repository and + branch, and can be asked to apply. Present only when `spec.applier` is set. +- `status.applier`: the last apply request GitOps Reverser made, and why. Use conditions for automation. @@ -648,6 +708,7 @@ summary reason, while the domain conditions name the axis to fix: | `ClusterProviderReady=False` or `SourceClusterReachable=False` | Source-cluster credentials and access. | | `GitPathAccepted=False` | Repository files under `spec.path`; the message includes the issue kind, path, and actor when known. | | `RenderMatchesLive=False` | Fields that no longer match the rendered source, often transformer-owned values. | +| `ApplierConfigured=False` | The applier named in `spec.applier`, its source, and the permission to patch it. Commits still flow; apply requests may not. | A green `GitTarget` with a red `WatchRule` or `ClusterWatchRule` means the destination may be fine, but the selected source objects are not being watched or authorized. Inspect the rule's `Ready`, @@ -840,6 +901,11 @@ Clearing `suspend` resumes from the cluster's current state on the next resync. suppressed while it was set are not replayed, so what lands is what the cluster holds then, not a backlog of the values it passed through. +**Suspend stops writes, not deployments.** A suspended target's applier is still asked to apply a +commit somebody else pushed, because with [`spec.applier`](#who-applies-this-folder-specapplier) +set nothing else asks it, and a paused mirror must not stall Git-side changes. It requests nothing +for a refusal, since that request exists to act on a live edit the target is no longer publishing. + To re-read the folder now rather than on the periodic cadence, stamp the reconcile-request annotation with any value that changes: @@ -999,10 +1065,9 @@ a human fixing a folder in Git produces no event either, so a periodic re-read i that can notice. What a refresh still never does is write anything. It plans nothing, commits nothing and pushes nothing. -That includes [`spec.onRefusal: PushEmptyCommit`](#reverting-a-refused-edit-speconrefusal), which a -folder refusal never triggers. That setting reverts a refused **edit**, and an empty commit cannot -fix a folder: only a human can, and the reconciler may be part-way through its own corrections -there. +That includes [`spec.onRefusal: Reapply`](#reverting-a-refused-edit-speconrefusal), which a folder +refusal never triggers. That setting reverts a refused **edit**, and reapplying Git cannot fix a +folder: only a human can, and the applier may be part-way through its own corrections there. A refusal a **write** found is not cleared by a refresh. A structural read sees the files; it cannot see that a particular write may not touch one of them, so only the resync that writes the @@ -1014,84 +1079,173 @@ effective granularity is the 5-minute control-plane reconcile, so a value below tick". [`gitopsreverser_git_fetches_total{reason="refresh"}`](interpreting-metrics.md#reading-git_fetches_total) is what it costs. -Reverser still does not **act** on what a refresh reads: applying a Git-side change is the GitOps -reconciler's job. And it is not a latency mechanism: it buys bounded staleness with no inbound -surface and no Git-host configuration. Seconds-fresh is -[the inbound receiver's job](design/push-notification-and-reconcile-trigger.md). +A refresh that finds a commit GitOps Reverser did not push is a foreign commit like any other: the +target's [applier](#who-applies-this-folder-specapplier) is asked to apply it once the branch is +quiet. That makes the refresh the backstop for deployments when no push webhook reaches the +operator. It is not a latency mechanism: it buys bounded staleness with no inbound surface and no +Git-host configuration. Seconds-fresh is the job of the +[push receiver](#receiving-push-notifications-specreceiver). -### Reverting a refused edit (`spec.onRefusal`) +### Who applies this folder (`spec.applier`) + +When Flux or Argo CD applies this folder back to the cluster, `spec.applier` names the object that +does it, and GitOps Reverser becomes the one that decides when it applies. The applier keeps doing +the fetching, rendering, applying and pruning. The reason it has to be this way, and how to +configure each tool, is the [bi-directional guide](bi-directional.md). + +Exactly one of three: + +```yaml +spec: + applier: + kustomizationRef: # a Flux Kustomization + namespace: flux-system + name: checkout +``` -`spec.onRefusal` chooses whether an eligible refused edit should request Git re-application. -`Ignore`, the default, records the refusal without moving the branch. Choose `PushEmptyCommit` -when restoring Git state takes priority over preserving unpublished live edits. The -[bi-directional guide](bi-directional.md#choosing-speconrefusal) explains when to enable it and how -one refused edit can cause an allowed edit to be lost. +```yaml +spec: + applier: + applicationRef: # an Argo CD Application + namespace: argocd + name: checkout +``` + +```yaml +spec: + applier: + webhook: # anything else + url: https://deployer.example.com/apply + secretRef: + name: deployer-hmac # key: token + quietPeriod: 30s # kustomizationRef and webhook only +``` + +| Field | Default | What it does | +| --- | --- | --- | +| `kustomizationRef` | | A Flux `Kustomization` whose `sourceRef` is a `GitRepository` on this repository and branch | +| `applicationRef` | | An Argo CD `Application` whose source is this repository and branch | +| `webhook.url` | | Where to POST each apply request | +| `webhook.secretRef` | | Secret whose `token` key signs the request | +| `quietPeriod` | `30s` | How long the branch must stay quiet after GitOps Reverser's own pushes before a `CatchUp` request. Ignored for `applicationRef` | + +**Where the reference is looked up.** In the source cluster, the cluster the applier applies to, +with the identity the [`ClusterProvider`](#clusterprovider) already uses. For the operator's own +cluster the chart grants `get` and `patch` on `kustomizations` and `gitrepositories` +(`kustomize.toolkit.fluxcd.io`, `source.toolkit.fluxcd.io`) and on `applications` +(`argoproj.io`). For a remote source cluster, grant the same to the kubeconfig's identity. An +applier that lives in a cluster other than the one it applies to, such as a hub Argo CD managing a +remote cluster, is not supported. + +**When a request is made.** Only on a quiet branch: no open commit window and no pending writes on +the branch worker, which is the moment nothing live is missing from the tip. + +| `reason` | When | Sent to | +| --- | --- | --- | +| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Open windows are closed and pending writes replayed and pushed first, so the request names that pushed commit | Every applier declared on the branch, once per applier object | +| `Refusal` | An eligible refused edit, with `spec.onRefusal: Reapply` | This target's applier | +| `CatchUp` | The branch has been quiet for `quietPeriod` since GitOps Reverser's own pushes | `kustomizationRef` and `webhook` appliers on the branch | + +An `applicationRef` applier is never asked to apply GitOps Reverser's own commits: with automated +sync off, Argo CD applies nothing on its own, so there is no stale state to catch up, and every sync +runs the `Application`'s hooks. Flux applies on its timers and on a controller restart, so its +`GitRepository` artifact is kept current instead. + +**What a request does.** + +| Applier | Request | +| --- | --- | +| `kustomizationRef` | What `flux reconcile kustomization --with-source` does: stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository`, then on the `Kustomization` when the artifact already held the commit. A `CatchUp` stamps only the `GitRepository` | +| `applicationRef` | Set `operation.sync` exactly as Argo CD's automated sync would: `revision` pinned to the commit, `prune` from `syncPolicy.automated.prune`, `syncOptions` and `retry` from the `Application`, `initiatedBy.username: gitops-reverser`. A running operation is waited for, never replaced | +| `webhook` | POST `{"repository", "branch", "commit", "reason"}` with `X-Signature: sha256=`, the header Flux's `generic-hmac` `Receiver` checks | + +A request asks; it never claims the apply happened. It runs off the publication path, is retried +with backoff on its own, and a failed request never fails a publication that succeeded. Read the +applier's own status (`lastAppliedRevision` on a `Kustomization`, `status.sync` on an +`Application`) for what it applied. + +**What it reports.** + +```yaml +status: + conditions: + - type: ApplierConfigured + status: "False" + reason: AutomatedSyncEnabled + message: 'Application argocd/checkout has automated sync enabled, so it also applies on its own' + applier: + requestedCommit: 4f2c1ab9... + requestedAt: "2026-10-20T10:14:03Z" + reason: ForeignCommit # ForeignCommit | Refusal | CatchUp +``` + +`ApplierConfigured` is a gate, so `Ready` aggregates it. Its `False` reasons: + +| Reason | Means | +| --- | --- | +| `ApplierNotFound` | No such object in the source cluster, or its kind is not installed there | +| `SourceMismatch` | The applier applies a different repository or branch | +| `AutomatedSyncEnabled` | The `Application` still syncs on its own. Requests are still sent; the guarantee is not there | +| `Forbidden` | The source-cluster identity may not read or patch the applier | + +It is checked on the target's reconcile tick rather than watched, so a cluster without the Flux or +Argo CD CRDs installed costs nothing. Each request also records an `ApplyRequested` Event on the +`GitTarget`, and a request that keeps failing records `ApplyRequestFailed`. + +### Reverting a refused edit (`spec.onRefusal`) A refusal means the acceptance gate or a write-boundary check found no legal destination for the -write. Nothing is committed for that write, and `GitPathAccepted` goes `False`. The live edit stays -until something corrects it. With Argo CD `selfHeal` off, refreshing an already-synced revision -does not restore it; an explicit sync or a new revision can. Flux corrects drift on its next apply, -which can follow its interval, a new source revision, or another reconcile trigger. +write. Nothing is committed for it, `GitPathAccepted` goes `False`, and the live edit stays until +something applies Git over it. `spec.onRefusal` decides whether GitOps Reverser asks for that: ```yaml spec: path: apps/checkout - onRefusal: PushEmptyCommit # default: Ignore + applier: + kustomizationRef: {namespace: flux-system, name: checkout} + onRefusal: Reapply # default: Ignore ``` -`PushEmptyCommit` pushes a commit that changes no file. Flux can observe a new artifact revision -and re-apply; Argo CD can sync an `OutOfSync` Application against a revision it has not synced, even -with `selfHeal: false`. The commit message says what was refused and why its diff is empty. - -This is a request for restoration. It requires a successful push and a reconciler configured to -observe and apply the new revision. Suspension, sync restrictions, or field-ignore settings can -prevent restoration. The empty commit does not clear the refusal condition: recovery still -requires a successful resync covering the refused scope. Re-applying Git can remove the offending -difference, but it cannot repair an unsupported folder or give a field a writable destination. +| Value | What happens | +| --- | --- | +| `Ignore` (default) | The refusal is reported and nothing else happens | +| `Reapply` | Once the branch is quiet, this target's [applier](#who-applies-this-folder-specapplier) is asked to apply the current tip, with reason `Refusal`. The refused edit is reverted | -Reverser deliberately does not name the `Kustomization` or `Application` that renders the folder. -Neither tool derives that mapping itself, so deriving it here to write into somebody else's object -would be guessing on their behalf. Moving the branch asks them instead. +`Reapply` requires `spec.applier`; a target without one is rejected at admission. The request goes +to this target's applier alone, so other folders on the branch are not woken, and it waits for a +quiet branch, so an allowed edit still in a commit window is published before the apply and is not +reverted with the refused one. The [bi-directional guide](bi-directional.md#choosing-speconrefusal) +covers when to choose it. **It only fires for an edit to a document the folder already holds, and never for a write that -removes one.** That is the pruning fence, and it is why the action is narrower than it first looks. -Re-applying corrects drift on an object Git manages, which is the point. For an object Git does not -manage, re-applying either does nothing ( -[Flux prunes from its inventory](https://fluxcd.io/flux/components/kustomize/kustomizations/#prune) -and Argo CD from the resources it tracks, and a live-created object is in neither) or, if the object -was managed and has since been removed from Git, prunes it. **Hurrying along somebody else's delete -is not this operator's call**, and the two cases cannot be told apart from here, so both are -excluded. +removes one.** That is the pruning fence. Re-applying corrects drift on an object Git manages, +which is the point. For an object Git does not manage, re-applying either does nothing (Flux prunes +from its inventory and Argo CD from the resources it tracks, and a live-created object is in +neither) or, if the object was managed and has since been removed from Git, prunes it. Hurrying +along somebody else's delete is not this operator's call, and the two cases cannot be told apart +from here, so both are excluded. Establishing that takes more than the refusal's kind, and more than the file. A render refusal fires for a brand-new resource an `images:` entry would override; removing one document from a file that holds two leaves the file in place, which looks exactly like an edit. Eligibility is therefore decided per document, and a write that removes one anywhere in the same flush is excluded whole. +Folder-level refusals, such as invalid YAML or unsupported content, never trigger it: they need a +correction to the folder. -**Read this before enabling it.** +Three more guards. A suspended target requests nothing. A `GitTarget` that cannot be read is +treated as `Ignore`, since missing evidence is not consent. And requests are coalesced per target: +a controller rewriting a base-owned field refuses on every one of its own reconciles, so refusals +that arrive while a request is outstanding share the next one rather than each making their own. -| | What to expect | -| --- | --- | -| Blast radius | The commit wakes **everything** watching the branch, not the refused object alone, so it can hurry unrelated work including another target's pending prune | -| Unpublished edits | An allowed edit still waiting in the commit window can be reverted along with the refused one | -| Argo CD exception | An `Application` with `argocd.argoproj.io/manifest-generate-paths` ignores the commit, because no file under its refresh paths changed | - -Folder-level refusals, such as invalid YAML or unsupported content, do not trigger this action. -They still require a correction to the folder. - -Three more guards. A suspended target never commits, because an empty commit is a write. A -`GitTarget` that cannot be read is treated as `Ignore`, since missing evidence is not consent. And -the commit is rate-limited per target, because a controller rewriting a base-owned field refuses on -every one of its own reconciles; a refusal arriving inside that window is **coalesced into one -trailing commit rather than dropped**, since the reconcile an earlier commit triggered may already -have finished. - -**A refusal nobody corrects is committed for once, not once per interval.** The target keeps -re-reading its folder and keeps refusing, and an observation of the same objects with the same -issues is already covered by the commit made for it. Anything different (a new value, another -object, a different refusal) is a new trigger and earns its own commit. So is the same edit made a -second time after the reconciler reverted it, because a reconcile of that resource type was accepted -in between. +**A refusal nobody corrects is requested once, not once per recheck.** The target keeps re-reading +its folder and keeps refusing, and an observation of the same objects with the same issues is +already covered by the request made for it. Anything different (a new value, another object, a +different refusal) is a new trigger. So is the same edit made a second time after the applier +reverted it, because a write of that resource type was accepted in between. + +Reapplying does not clear the refusal condition: recovery still requires a successful resync +covering the refused scope. It removes the offending live difference, but it cannot repair an +unsupported folder or give a field a writable destination. ### Deletion policy (`spec.prune.mode`) @@ -2320,6 +2474,23 @@ attribution: collectionUIDCap: 10000 ``` +## Push receiver settings + +The [push receiver](#receiving-push-notifications-specreceiver) is its own HTTP listener, separate +from the audit listener. A Git host authenticates with an HMAC or a token over the body and cannot +present the client certificate the audit listener requires, so the two cannot share one. + +| Flag | Helm value | Default | What it does | +| --- | --- | --- | --- | +| `--receiver-bind-address` | `receiver.bindAddress` | `0` (off) | `host:port` the receiver listens on. `0` serves no receiver | + +The chart creates a `gitops-reverser-receiver` Service when the listener is on. Expose it to your +Git host with an Ingress or a Gateway route of your own that terminates TLS, the way Flux's +`webhook-receiver` Service is exposed. Every request still has to carry a valid signature for the +route it names; a route whose `GitProvider` has no `spec.receiver` does not serve. + +Like everything that drives a branch worker, a delivery is handled on the elected replica. + ## Quickstart vs hand-managed resources Keep using the [quickstart](quickstart.md) when you want the fastest first commit. diff --git a/docs/definitions.md b/docs/definitions.md index 9860e4c7..a9a87fd8 100644 --- a/docs/definitions.md +++ b/docs/definitions.md @@ -153,10 +153,10 @@ one. One word for the act, in the API (`window.attach`, `attachTimeout`), the co "claim", which names a rule's demand on the read side. **Pending write.** A write a branch worker has decided to make and not yet published: a closed -commit window, an atomic batch, a resync, a save's empty record, or a refusal's empty commit. The -branch worker keeps its pending writes in order, commits them, and pushes them on one retry -schedule. A pending write can be waiting to become a commit, committed locally and waiting for a -push, or replayed onto a remote that moved. `PendingWrite` and `pendingWrites` in Go. Not "log", +commit window, an atomic batch, a resync, or a save's empty record. The branch worker keeps its +pending writes in order, commits them, and pushes them on one retry schedule. A pending write can +be waiting to become a commit, committed locally and waiting for a push, or replayed onto a remote +that moved. `PendingWrite` and `pendingWrites` in Go. Not "log", which also names diagnostic logs and suggests a durable journal, which this is not. **Publication.** One commit-and-push cycle by a branch worker. The thing a `GitTarget` reports in @@ -188,6 +188,22 @@ called pruning. **Refresher.** The periodic re-proof of an idle branch's remote state. Freshness is not base trust, and the two are not interchangeable. +## Applying back + +**Applier.** The object that applies a folder to its source cluster: a Flux `Kustomization`, an +Argo CD `Application`, or a webhook standing in for another tool. `GitTarget.spec.applier` names +it. Not "reconciler", which also names this operator's own controllers, and not "orchestrator", +which says nothing about what it does to the cluster. + +**Apply request.** What GitOps Reverser sends an applier: apply this commit, for this reason +(`ForeignCommit`, `Refusal`, `CatchUp`). It asks; it never reports that the apply happened. + +**Quiet branch.** A branch whose worker holds no open commit window and no pending writes. Apply +requests go out only on a quiet branch, because only then is nothing live missing from the tip. + +**Receiver.** `GitProvider.spec.receiver`, the endpoint a Git host notifies on a push. The word and +its `type` values are Flux's. A delivery to it is a hint to look, never a fact about the branch. + ## Attribution The word alone is ambiguous, so it is never used alone. There are two: @@ -296,7 +312,8 @@ type already carries, and never says "yet": the `Unknown` status already says it ### 7. An enum value we invent is PascalCase; a value that names something else keeps its spelling `Never`, `OnEvent`, `Always`, `Plain`, `KustomizeRoot`, `KustomizeOverlay`, `Push`, `Fetch`, -`Ignore`, `PushEmptyCommit`, `ConfigMap`, `Secret`. An acronym stays uppercase (`HTTP`, not `Http`). +`Ignore`, `Reapply`, `ForeignCommit`, `CatchUp`, `ConfigMap`, `Secret`. An acronym stays uppercase +(`HTTP`, not `Http`). The exception is a value that names a thing outside this API: a tool, an operating system, a protocol, a Kubernetes resource name. It keeps that thing's own spelling. @@ -392,3 +409,4 @@ outage. | failure, error (for a declined write) | refusal | the operator worked correctly | | sha, revision (for a commit hash) | commit | `revision` is Flux's word for a polymorphic identifier, and is reserved for one | | namespace access (bare) | access, or source namespace | two directions, two names | +| reconciler, orchestrator (for Flux or Argo CD) | applier | "reconciler" also names our own controllers | diff --git a/docs/design/bi-directional-right.md b/docs/design/bi-directional-right.md index 5f05cf53..83634d45 100644 --- a/docs/design/bi-directional-right.md +++ b/docs/design/bi-directional-right.md @@ -344,142 +344,83 @@ our browser harness, because it is a property of the engine plus the reconciler, ## 9. Response: GitOps Reverser takes over the triggers -_From gitops-reverser, 2026-10-16._ +_From gitops-reverser, 2026-10-16. The design is +[push-notification-and-reconcile-trigger.md](push-notification-and-reconcile-trigger.md); the +guide as it will read is [bi-directional.md](../bi-directional.md)._ **The short version: GitOps Reverser takes over the reconciliation triggers. It has to.** Flux and Argo CD keep doing what they are good at: fetching, rendering, applying, pruning, and reporting -health. They stop doing it at their own will. An apply happens because GitOps Reverser decided the -revision is safe to apply, because a human asked for one, or because of a backstop timer that was -chosen knowing what it costs. +health. They stop doing it at their own will. An apply happens because GitOps Reverser asked for +it on a branch with nothing pending, because a human asked, or because of a long backstop timer. -We accept the ask's rule and its ordering. We go further than §8's last bullet: the webhook is not -the only trigger to take away, because every timer the reconciler runs is a trigger too. +We accept the rule and the ordering in §4, and we go further than §8's last bullet: the webhook is +not the only trigger to take away, because every timer the applier runs is a trigger too. ### 9.1 Why it has to -An apply is safe only when nothing live is missing from the revision being applied. GitOps -Reverser is the only component that can know that, because it holds the open commit windows and -the writes waiting to be pushed. The applier cannot see an edit that has not reached Git yet. An -apply GitOps Reverser did not ask for is one nobody checked, and it reverts exactly the edits we -captured and have not published. - -That makes the trigger part of the engine's correctness rather than an integration detail, and a -component that cannot see the pending writes cannot own it. +An apply is safe only when nothing live is missing from the commit being applied. GitOps Reverser +is the only component that can know that, because it holds the open commit windows and the writes +waiting to be pushed. An apply it did not ask for is one nobody checked. ### 9.2 Timers are triggers too -§6 notes this for Flux's short `GitRepository` poll. On Argo CD it is worse, and it is not a matter -of tuning. - -Argo CD's timed refresh (`timeout.reconciliation`, 120s plus up to 60s of jitter) resolves the -branch tip. With `selfHeal: false`, automated sync is skipped only when that tip equals the last -synced revision +§6 notes this for Flux's short `GitRepository` poll. On Argo CD it is worse. The timed refresh +(`timeout.reconciliation`, 120s plus up to 60s of jitter) resolves the branch tip, and with +`selfHeal: false` automated sync skips only a commit it already synced ([`alreadyAttemptedSync`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2648-L2670)). -Every push GitOps Reverser makes is a new revision. So two to three minutes after each of our -pushes, Argo CD syncs the whole Application at the tip, and reverts whatever is live but not yet -pushed at that moment. An idle branch is safe; during an editing session something is usually -pending, so in practice it resets an arbitrary edit. - -Filtering the webhook removes the sync that lands about four seconds after a push. It does nothing -about this one. On Flux the same holds for every reconcile: kustomize-controller applies whatever -woke it. - -### 9.3 What it means per reconciler - -| | Argo CD | Flux | -| --- | --- | --- | -| Configure | `syncPolicy` **without** `automated` | `Receiver` on the `GitRepository` only, long intervals on both objects | -| GitOps Reverser's trigger | a sync of a declared `Application`, pinned to the revision we judged safe | a signed POST to the `Receiver` | -| Their own timers | keep them: without auto-sync a refresh only reports `OutOfSync` | the residual: one apply per `Kustomization` interval that nobody checked | -| What stays exposed | an edit landing between our push and the sync's apply | the same, plus that interval apply | - -**Argo CD.** Without `automated`, a refresh from the timer, a webhook, or the UI only compares and -reports. Nothing applies until a sync, and a sync names its revision, so it applies exactly the -commit we checked rather than whatever the tip has become. This needs the `Application`'s name and -Argo CD's `applications, sync` permission. That is naming an orchestrator object, which this -repository has avoided until now. What -[push-notification-and-reconcile-trigger.md](push-notification-and-reconcile-trigger.md) objected -to was **inferring** which object owns a folder. A reference the user declares is a different claim, -and we accept it. - -**Flux** has no refresh-only mode, and `suspend: true` stops a `Receiver` as well. The closest -setup is long intervals with the `Receiver` as the only fast path. The `Kustomization` interval -then remains the one apply we do not control. A very long interval removes it, and removes the -backstop with it. That is the installer's choice, and the guide has to say so. - -### 9.4 What we take from the ask, and what we change - -We take the rule (trigger only when the branch gained commits we did not author), the ordering in -§4 (trigger after the replayed push lands), and the test in §7. Four things change. - -1. **Not every commit we author is a no-op.** There are three exceptions: - - `GitTarget.spec.onRefusal: PushEmptyCommit` exists only to make the reconciler apply, so it - must trigger. - - The replayed `F + W'` is our commit, but it carries `F`, which nothing has applied. - - A folder applied to a cluster other than the one it was mirrored from: a shared base, a fleet, - or a `GitTarget` that mirrors a remote cluster through `spec.kubeConfig`. Our commit is a no-op - only for the reconciler that applies to the cluster it was read from, so suppression is per - trigger target, not global. -2. **The decision is state on the branch worker, not a verdict on each delivery.** One flag, "the - reconciler is behind". It is set when the worker sees a tip it did not push (a rejected - compare-and-swap push, a refresh fetch, a foreign delivery) or pushes a refusal empty commit. It - is cleared by sending the trigger once the worker is quiet: no pending writes, base trusted. - Deliveries become hints. This settles the cases a per-delivery table gets wrong: - - the delivery for `F + W'` names our own SHA, yet `F` was never applied; - - a late delivery for an older push of ours; - - a rollback onto one of our own older commits, settled by one ref advertisement rather than by - guessing order; - - a delivery that arrives before `PushAtomic` returns, handled by recording the SHA while the - push is in flight. -3. **The hold cannot wait for commit windows to close.** A window closes on silence, so a user who - keeps editing would hold back a foreign push indefinitely. On a foreign commit, close the open - windows, replay onto `F`, push, then trigger, with a bound on the whole sequence. -4. **Forwarding host deliveries is no longer the mechanism.** Under the takeover an Argo CD webhook - only refreshes, and a Flux `Receiver` is a trigger target we call. Forwarding stays useful for a - Git host that can point at only one place, and to keep the reconciler's status fresh. - -### 9.5 The seam - -The decision is core and names no orchestrator. Delivering it is a short list of trigger targets, -named by what they send: - -| Trigger target | What GitOps Reverser sends | For | -| --- | --- | --- | -| `argocd-sync` | a sync of a declared `Application` at the checked revision | Argo CD | -| `webhook` | a signed POST, `generic-hmac` or a synthesized `github-push` | a Flux `Receiver`, or any third-party receiver | -| _(always)_ | a Kubernetes Event naming the branch, the revision, and why | whatever the installer wires up | - -A third-party GitOps tool plugs in through `webhook` or the Event. There is no in-process plugin, -for the reason Option 8a was declined: it would run with every credential the operator holds. - -### 9.6 What changes for gitops-api - -- Your webhook stops waking Flux and relays `{repository, branch, after}` to the Option 3 endpoint. -- The `lastAppliedRevision` gate goes, and has to: Flux no longer applies your own saves, so that - watermark never arrives for them. -- The `GitRepository` loses its `1m` poll, and the `Receiver` lists the `GitRepository` only. - -### 9.7 What changes in this repository - -- [`bi-directional.md`](../bi-directional.md): Argo CD without `automated`, Flux without the short - poll, and "Why the loop stops" restated around the trigger. -- [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md): a - direction is chosen. Option 2 as written is rejected, and Options 3 and 4 become inputs to the - trigger decision. -- The bi-directional e2e corner: the Argo CD specs that rely on automated sync change. §7's test - lands there, with an Argo CD variant run on a short `timeout.reconciliation`, asserting that no - reconciler-authored commit appears. - -### 9.8 What we accept, and what is open - -- **GitOps Reverser becomes the only fast path to a deployment.** While it is down, a push by a - human reaches the cluster only through a manual sync or Flux's backstop interval. Bi-directional - editing is already broken in that state; Git-side deployments were not, and now they are. -- **Uncertainty triggers.** When the branch worker is absent, suspended, or on another replica, the - trigger fires. Only a commit known to be ours is held back. -- **Open: where trigger targets are configured.** `GitProvider` (the repository) or `GitTarget` - (the folder and the cluster). The per-cluster suppression in §9.4 argues for the `GitTarget`. -- **Open: high availability.** The decision needs the branch worker's state, which lives on the - replica that owns the worker. -- **To verify:** that Flux's `github` `Receiver` type accepts a synthesized push without matching - the repository it names. If it does, `generic-hmac` is not needed. +Every push GitOps Reverser makes is a new commit, so two to three minutes after each push the whole +`Application` is synced over whatever is pending. Filtering webhooks does nothing about it. + +### 9.3 What we take from the ask, and what we change + +We take the rule (apply when the branch gained commits we did not author), the ordering in §4, and +the test in §7. What changes: + +1. **The applier is named, on the `GitTarget`.** `spec.applier` holds a `kustomizationRef`, an + `applicationRef` or a `webhook`. The objection to naming orchestrator objects was to inferring + which one applies a folder; a declared reference infers nothing. That also replaces §5's + `Receiver` and synthesized-event plumbing: GitOps Reverser patches the applier through the + Kubernetes API, as `flux reconcile --with-source` does, or sets the `Application`'s sync + operation pinned to the checked commit. +2. **Argo CD runs with automated sync off.** `automated.enabled: false` (Argo CD 3.1 and later) + keeps `prune` configured; GitOps Reverser's sync copies it. A refresh then only compares, so + Argo CD's timer can stay. +3. **Flux needs one request for our own commits after all.** Flux has no refresh-only mode: its + timers and a controller restart apply whatever artifact it holds, and with long intervals that + artifact falls behind our pushes. Applying it would revert published edits. So once the branch + has been quiet for `quietPeriod` (`30s`), GitOps Reverser asks Flux to fetch, reason `CatchUp`. + The wait keeps it out of an editing session like the one in §1. +4. **Not every commit we author is a no-op**, and the design does not rely on that. Our replayed + `F + W'` carries `F`; a refusal needs an apply too. So the decision is state on the branch + worker ("the applier is behind"), set by any foreign commit or eligible refusal and cleared by a + request on a quiet branch, not a verdict on each delivery. A delivery is a hint: the worker reads + the remote's advertisement and judges from that, which handles late deliveries, rollbacks onto + our own older commits, and deliveries for our own replayed push. +5. **The hold does not wait for commit windows to close.** A busy user would hold a foreign push + back indefinitely. On a foreign commit the worker closes the windows, replays, pushes, then + requests. +6. **`onRefusal: PushEmptyCommit` becomes `onRefusal: Reapply`.** With the applier named, the + refusal asks that one applier to apply the tip, once the branch is quiet, instead of moving the + branch for everything watching it. +7. **Suppression is per applier, by construction.** A reconciler for another cluster is not + declared on this `GitTarget`, so we never ask it anything and its own triggers are its business. + +### 9.4 What changes for gitops-api + +- Your webhook stops waking Flux. Relay `{repository, branch, after}` to the tenant `GitProvider`'s + receiver (`type: generic-hmac`, path in `status.webhookPath`). +- Declare the tenant `Kustomization` as each `GitTarget`'s `kustomizationRef`, and drop any Flux + `Receiver` for the repository. +- The `lastAppliedRevision` gate goes. Flux still applies your saves eventually, through `CatchUp`, + but only after the branch has been quiet, which is what the gate was approximating. +- `GitRepository` and `Kustomization` intervals go long (`24h`); the `1m` poll goes. + +### 9.5 What we accept, and what is open + +- **GitOps Reverser becomes the fast path to a deployment.** While it is down, a Git-side push + reaches the cluster only through a manual sync, `flux reconcile`, or Flux's backstop interval. +- **An apply request cannot reserve time.** An edit made while a requested apply runs is still + reverted. That now happens only after a foreign push, a refusal, or a quiet period, not after + every commit we make. +- **Open:** an applier in a cluster other than the one it applies to (hub Argo CD), and routing + deliveries with more than one replica. Both are in the design's open questions. diff --git a/docs/design/push-notification-and-reconcile-trigger.md b/docs/design/push-notification-and-reconcile-trigger.md index 9bd51640..868630fa 100644 --- a/docs/design/push-notification-and-reconcile-trigger.md +++ b/docs/design/push-notification-and-reconcile-trigger.md @@ -1,1019 +1,475 @@ -# Push notification and the reconcile trigger - -> **design**: part shipped, the rest is a menu of options and none of them is chosen. -> Index: [`../INDEX.md`](../INDEX.md) -> Captured 2026-09-19, merged and rewritten 2026-09-21. This page replaces -> `inbound-push-notification.md` and `support-boundary/orchestrator-reconcile-trigger.md`, -> which argued two halves of one subject. -> Related: [`reconcile-triggering.md`](reconcile-triggering.md) (how our own controllers wake up; -> its §5 is superseded here), [`../bi-directional.md`](../bi-directional.md) (the user-facing -> model), [`support-boundary/orchestrator-knowledge-boundary.md`](support-boundary/orchestrator-knowledge-boundary.md) -> (the ownership model part 3 keeps running into), -> [`support-boundary/admission-consent.md`](support-boundary/admission-consent.md) (deciding -> *whether* a write happens, where this page is about what happens after). - -Reverser and the GitOps reconciler share a branch and tell each other almost nothing. This page is -about the messages that do not exist yet, what shipped instead, and what each remaining option -would cost. - -**Part 1** is built and measured. **Part 2** is the gap it leaves. **Part 3** is the menu, and -doing nothing is the first entry rather than a rhetorical one. +# Push notification and the apply trigger ---- +> **design**: decided 2026-10-16, not built. Index: [`../INDEX.md`](../INDEX.md) +> User-facing model: [`../bi-directional.md`](../bi-directional.md). Fields: +> [`../configuration.md`](../configuration.md#who-applies-this-folder-specapplier). The request that +> led here: [`bi-directional-right.md`](bi-directional-right.md). -## Part 1: what is built +GitOps Reverser and the applier (Flux or Argo CD) share a branch. This page is how they talk: +how GitOps Reverser learns that somebody else pushed, and how it decides when the applier applies. -### 1.1 The loop, and where it closes +**The decision: GitOps Reverser owns the apply trigger.** The applier keeps fetching, rendering, +applying and pruning. It stops applying at its own will: its timers, webhooks and automated sync +are switched off, and GitOps Reverser sends apply requests at moments it has checked. -```mermaid -flowchart LR - api(["API or operator edit"]) --> cluster[("Cluster")] - cluster -- "watch event" --> gr["GitOps Reverser"] - gr -- "commit, then compare-and-swap push" --> git[("Git branch")] - git -- "push webhook" --> recon["Flux or Argo CD
(selfHeal off)"] - recon -- "apply the fresh commit" --> cluster - - style gr fill:#e8f4fd,stroke:#2196f3 - style recon fill:#fff3e0,stroke:#ff9800 -``` +**Part 1** is the Git side as built, because the trigger rests on it. **Part 2** is why the trigger +has to move. **Part 3** is the design. **Part 4** is the reference: wire contracts, status, metrics, +tests. -That is the product: the cluster is the editing surface, Reverser mirrors it to Git, and the -reconciler is a **triggered applier** rather than an always-on loop. Every arrow exists today, and -the one from Git to the reconciler is a webhook the user configures. +--- + +## Part 1: how the branch worker reads the remote -### 1.2 Which side wins, before anything else +### 1.1 Which side wins -Every option in part 3 depends on this, so it goes first. +Everything else depends on this, so it goes first. -- **Cluster to Git.** The Kubernetes API is where edits are made. A write we captured from it is +- **Cluster to Git.** The Kubernetes API is where edits are made. A write captured from it is intent, and it wins over what Git holds. Replay re-plans the captured object onto the new tip and - writes it ([`../bi-directional.md`](../bi-directional.md)). -- **Git to cluster.** A push we did not author is new truth. We do not overrule it, and getting it - applied quickly is the point. + writes it. +- **Git to cluster.** A push GitOps Reverser did not author is new truth. It is not overruled, and + getting it applied promptly is the point. -Those read like a contradiction and are not, because they meet in exactly one cell. +They meet in one cell: -| | We hold a captured write for the object | We hold nothing | +| | A captured write is pending for the object | Nothing is pending | | --- | --- | --- | -| **A push changed it** | API wins, silently | **The hazard.** We must never publish here | +| **A push changed it** | API wins, silently | **The hazard.** Never publish here | | **A push did not** | Ordinary publication | Nothing to do | -The top-left cell is the only genuine conflict and it is decided: the API value wins, with no extra -signal raised. The cell that matters is the top right. We hold no intent there, so anything we -publish for that object is a stale reading rather than a decision, and publishing it reverts a push -for no reason at all. - -**Everything dangerous on this page lives in that one cell.** +The top-left cell is the only real conflict, and it is decided: the API value wins, with no extra +signal. The top-right cell is where danger lives. Nothing is pending, so anything published for +that object is a stale reading, and publishing it reverts a push for no reason. -### 1.3 The push already detects a moved branch - -[`validatePushState`](../../internal/git/git_atomic_push.go) opens a push session, reads the -remote's ref advertisement, and compares it against the SHA the cycle's commits were based on. A -mismatch is refused before a single object is uploaded, and the `packp.Command` carries -`Old: oldHash` so the server enforces the same condition independently. - -A cycle that commits **nothing** still reaches that push. A no-diff write is retained with a zero -`CommitSHA`, so the advertisement is read either way. That is what makes the next section safe. - -### 1.4 What shipped: the fetch in front of every cycle is gone - -```mermaid -sequenceDiagram - participant W as Branch worker - participant R as Git remote - - rect rgb(255, 235, 235) - Note over W,R: REMOVED: first commit of a cycle - W->>R: list refs - R-->>W: advertisement - W->>R: fetch objects - R-->>W: packfile - Note over W: reset worktree to the fetched tip - end - - Note over W: plan and commit locally - - rect rgb(232, 245, 233) - Note over W,R: KEPT: the push - W->>R: open receive-pack, read advertisement - R-->>W: current branch SHA - Note over W: SHA matches the commits' base,
so the write is safe to send - W->>R: packfile with Old = expected SHA - R-->>W: accepted - end -``` +**So a notification that the branch moved never triggers a cluster snapshot.** The cluster still +holds the values from before the push; a snapshot would publish them over it. A push says something +about Git and nothing about the cluster. -The green block already reads the branch SHA, which is the only thing the red block contributed on -an uncontended cycle. **A target that is publishing therefore reads the remote only through the -push it was making anyway**, which is what the whole of §1.6 measures. +### 1.2 The push is the read -One qualifier, added when the refresher shipped: an IDLE target now spends one ref advertisement -per `--git-refresh-interval` to re-prove where its branch is, so "no Git traffic at all" holds for -a publishing target always, and for an idle one only at `--git-refresh-interval=0`. Nothing here -changes for the active case, because a successful push renews the same knowledge and an active -branch therefore never schedules a refresh. +A publication cycle does not fetch before it plans. It plans on the checkout it has, and the push +reads the branch: [`validatePushState`](../../internal/git/git_atomic_push.go) opens the push +session, reads the remote's ref advertisement, and compares it with the commit the cycle's writes +were based on. A mismatch is refused before any object is uploaded, and the `packp.Command` carries +`Old` so the server enforces the same condition. A cycle that commits nothing still reaches the +push, so the advertisement is read either way. -### 1.5 The invariant that replaced it +That rests on one invariant: > **The worktree sits at the remote tip of the target branch, or the worker knows it does not.** -Three flags carry it, and they are three because each brackets a different span. Collapsing any two -is a bug that was found by trying. +`baseTrusted` says whether the worker is at the remote tip. A reset onto a fetched tip sets it, and +so does a push the server accepted, which is the stronger proof: the server has just moved the +branch on the worker's authority. Any push error or rejection clears it. `checkoutApplied` counts +how many pending writes the checkout holds on top of its root; unknown means a failed write left +the tree dirty, and a count below the pending writes means a reset discarded their commits and they +must be replayed. The loop's `materialize` is the one place that acts on it +([`gittarget-branch-worker-pending-writes.md`](gittarget-branch-worker-pending-writes.md)). -Trust is gained in two ways, and after the refresher shipped the push is the main one: a reset onto -a fetched tip, and a `PushAtomic` the server did not reject. The second is the stronger of the two — -a fetch says where the branch was when we looked, while an accepted push says the server has just -moved it there on our authority — which is why a push also renews `status.remote` and why an -actively-written branch never needs a refresh. +A cycle plans straight onto the worktree only when the base is trusted and the tree is clean; +otherwise it fetches, resets and replays first. -```mermaid -stateDiagram-v2 - direction LR - - state "Base at remote tip?" as Base { - [*] --> Untrusted - Untrusted --> Trusted: reset to the target branch - Untrusted --> Trusted: PushAtomic succeeded - Trusted --> Untrusted: push error or rejection - Trusted --> Untrusted: forced recheck - } - - state "Worktree clean?" as Tree { - [*] --> Clean - Clean --> Dirty: executePendingWrites error - Dirty --> Clean: reset (only this clears it) - } - - state "Retained writes real?" as Replay { - [*] --> Current - Current --> Stale: a reset discarded their commits - Stale --> Current: the replay rebuilt them - } -``` - -- **`baseTrusted`** answers "are we at the remote tip". A successful push sets it; any push error or - rejection clears it. -- **`worktreeDirty`** answers "did a failed write leave files nobody asked for". Only a reset clears - it, and a successful push must never clear it: a push can succeed while an earlier write's - leftovers are still staged, and treating those as one question commits them under an unrelated - author. -- **`replayRequired`** answers "do the retained writes still name commits that exist". A reset - discards the local commits behind retained writes; the replay rebuilds them. Die in between and - the next push sends nothing, returns nil, and the worker settles a `CommitRequest` as `Committed` - at a SHA that is on no remote. `worktreeDirty` cannot cover this, because the reset **clears** - `worktreeDirty` at exactly the moment the problem starts. - -The second and third questions are now one count, `checkoutApplied`: how many retained writes the -checkout holds on top of its root. Unknown is `worktreeDirty`; a count below the retained writes is -`replayRequired`, and a reset sets it to zero, so the reset no longer hides the problem it starts. -The loop's `materialize` is the one place that acts on it -([`gittarget-branch-worker-pending-writes.md`](gittarget-branch-worker-pending-writes.md), step 1). - -A cycle plans straight onto the worktree only when the base is trusted and the tree is clean: - -```mermaid -flowchart TD - START["First commit of a cycle"] --> RETAINED{"Retained writes
from this cycle?"} - RETAINED -->|yes| DIRTY1{"Worktree dirty?"} - DIRTY1 -->|no| PLAN["Plan on the worktree"] - DIRTY1 -->|yes| REPLAY["Reset, then replay
the retained writes"] - REPLAY --> PLAN - RETAINED -->|no| OK{"Base trusted
and tree clean?"} - OK -->|yes| PLAN - OK -->|no| FETCH["Fetch and reset"] - FETCH --> PLAN - - style PLAN fill:#e8f5e9,stroke:#43a047 -``` - -### 1.6 What it cost, measured +### 1.3 What it costs Every row comes from [`git-roundtrip-ledger.golden`](../../internal/git/testdata/git-roundtrip-ledger.golden), -produced against canonical git's `http-backend`. The unit is **HTTP requests to the Git host**. +measured against canonical git's `http-backend`. The unit is HTTP requests to the Git host. -| Situation | Before | Now | -| --- | --- | --- | -| Idle target | 0 | **0** | -| Uncontended publication | 4 | **2** | -| Publication that commits nothing | 3 | **1** | -| Contended publication, one rejection | 10 | **6** | -| Contended publication, two rejections | 16 | **10** | -| Resync snapshot | 4 | **4** | -| Forced recheck | 2 | **2** | -| Publication onto a branch the remote does not have | 4 | **2** | - -Three of those are worth reading twice. - -**A publication that commits nothing costs one request.** It opens the push conversation, reads the -advertisement, finds the remote correct, and sends nothing. That row *is* §1.3's argument, measured. - -**A rejection cost ten, not the eight an earlier draft predicted.** When the remote has moved, the -fetch that discovers it transfers objects, so it costs three requests rather than two. - -**The resync row does not move, deliberately.** A resync that finds nothing to change never reaches -a push, so nothing would catch a stale base. It keeps its fetch. - -**A branch the remote does not have is trusted right away**, which is the last row and a later -change than the rest of the table. A reset lands the worktree on whatever the remote has for the -branch, and for a branch it does not have, a worktree based on the default branch (or an empty one) -**is** that state. There is nothing left to learn, so the two `fetch-open` requests that cycle used -to pay could only re-learn the absence. The compare-and-swap is what makes it safe: a push onto a -branch we believe is absent declares `Old = zero`, which the server rejects if somebody created it -meanwhile, and the rejection invalidates and fetches. The cost of being wrong is one rejection, not -a bad write. - -### 1.7 The instrument - -`gitopsreverser_git_fetches_total{provider_namespace, provider_name, branch, reason}` counts every -call that runs `SmartFetch`. The label is the point: a bare total would fall when contention fell -and rise when a webhook arrived. - -| `reason` | Means | +| Situation | Requests | | --- | --- | -| `publication` | Head of a cycle. **Zero on a healthy steady-state target**, which is the claim §1.4 makes | -| `recovery` | The base was untrusted or the tree was dirty. Nonzero means trust is being lost | -| `contention` | The reset after a confirmed rejection. One rejection costs exactly one | -| `push_failure_probe` | A push that died before the remote said anything: credentials, connectivity | -| `forced_recheck` | An operator asked, or a resync refreshed | -| `bootstrap` | **Test-only today.** `prepareBootstrapRepository` has no production caller | +| Idle target | 0 | +| Uncontended publication | 2 | +| Publication that commits nothing | 1 | +| Contended publication, one rejection | 6 | +| Contended publication, two rejections | 10 | +| Resync snapshot | 4 | +| Forced recheck | 2 | +| Publication onto a branch the remote does not have | 2 | + +An idle target also spends one ref advertisement per `--git-refresh-interval` to re-prove where its +branch is, and a fetch only when the branch moved. A receiver delivery costs the same as that +refresh (§3.3). + +`gitopsreverser_git_fetches_total{reason}` counts every call that runs a `SmartFetch`. On a healthy +publishing target `reason="publication"` stays at zero; see +[interpreting-metrics.md](../interpreting-metrics.md#reading-git_fetches_total). --- -## Part 2: the gap +## Part 2: why the trigger has to move -### 2.1 Two messages that do not exist +An apply is safe only when nothing live is missing from the commit being applied. Between an edit +and its push, the edit is live and absent from Git: in an open commit window or in the pending +writes. An apply then writes Git's older value over it, GitOps Reverser mirrors that revert as the +applier's commit, and the edit is lost. -```mermaid -flowchart LR - api(["API or operator edit"]) --> cluster[("Cluster")] - cluster -- watch --> gr["GitOps Reverser"] - gr -- "commit + push" --> git[("Git branch")] - git -- "push webhook" --> recon["Flux or Argo CD"] - recon -- apply --> cluster - other(["Someone else pushes"]) --> git - - git -. "A: nobody tells Reverser
the branch moved" .-> gr - gr -. "B: refused write means no commit,
no push, no webhook, no revert" .-> recon - - style gr fill:#e8f4fd,stroke:#2196f3 - style recon fill:#fff3e0,stroke:#ff9800 -``` - -**A: nothing tells Reverser that a branch it tracks moved.** A *publishing* target finds out at its -next push and replays, which is correct and costs one wasted request. A *refused* target recovers on -its own, because it is not converged and requeues every 10 seconds with a forced re-read. The case -that does not recover is a **healthy, idle** target: converged, requeuing every 5 minutes without -touching Git, holding its previous answer about a folder somebody has since changed. - -Two honest qualifications. This gap predates §1.4, because a target with no writes never performed -the head-of-cycle fetch either. And on the traffic this system is built for it is mild. What changed -is the stakes: a notification is now the only thing that would move an idle target's view of Git. - -**B: a refused write produces no commit, so nothing reverts it.** With Argo CD `selfHeal: false`, -which bi-directional editing requires, the live edit stays `OutOfSync` until a human intervenes. -Under Flux it is reverted on the next apply interval, which a long interval leaves hours away. - -Note the shape of that sentence, because option 7 turns on it: the reconciler is idle here not -because it lacks permission to apply, but because **nothing has changed the revision**. - -### 2.2 Why the obvious fix for A is dangerous +Only the branch worker can see those edits. Every trigger the applier runs by itself fires without +that knowledge: -The chain to make an idle target re-read Git already exists and a human can drive it: - -```bash -kubectl annotate gittarget editing -n gitops-reverser \ - reconcile.configbutler.ai/requestedAt="$(date +%s)" --overwrite -``` - -That annotation runs a **full resync**: a cluster-to-Git snapshot with mark-and-sweep. Wiring a push -webhook to it would be the most natural thing in the world and would be a correctness bug. - -```mermaid -sequenceDiagram - actor Dev as Git-side author - participant Git - participant Rev as GitOps Reverser - participant Rec as Flux or Argo CD - participant K8s as Cluster - - Dev->>Git: push a fix - par one delivery each, nothing orders them - Git-->>Rev: push webhook - and - Git-->>Rec: push webhook - end - Note over Rev: needs a fetch and a plan - Note over Rec: needs a fetch, a render, and an apply - Rev->>K8s: snapshot the cluster - Note over K8s: still holds the PRE-push values - Rev->>Git: commit that snapshot over the fix - Rec->>K8s: apply... the reverted content - Note over Dev,K8s: the author watches their change
get undone by the mirror -``` +- **A Git-host webhook into the applier** fires on every push, GitOps Reverser's own included. Two + API edits a few seconds apart are enough: the first commit is fetched, the second edit lands, and + the first commit is applied over it. +- **Argo CD's timed refresh** resolves the branch tip every `timeout.reconciliation` (120s plus up + to 60s of jitter). Automated sync then runs whenever the tip is a commit it has not synced; with + `selfHeal: false` it skips only one it already synced + ([`alreadyAttemptedSync`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2648-L2670)). + Every push GitOps Reverser makes is a new commit, so two to three minutes after each push the + whole `Application` is synced over whatever is pending. +- **Flux's intervals** apply on every reconcile. kustomize-controller corrects drift each time it + runs, and a new `GitRepository` artifact wakes every `Kustomization` that reads it. -The argument is not that we are always first. It is that **nothing prevents it**, the window is a -render and an apply wide, and the failure is silent. This is §1.2's top-right cell: those pre-push -values are not intent, they are what the cluster happened to hold. +Filtering which webhook deliveries reach the applier removes the first trigger only. The timers +remain, and they cannot be filtered from outside. So the applier's own triggers are switched off, +and GitOps Reverser, which can see the pending edits, makes the requests. -**So the rule for anything built in part 3 is absolute: a notification that a branch moved must -never trigger a cluster snapshot.** A push tells you about Git. It says nothing about the cluster. - -The annotation stays exactly what it is, a human escape hatch for "go and look at Git now", where a -snapshot is what the operator is asking for. +That costs something real: **GitOps Reverser becomes the fast path to a deployment.** While it is +down, a Git-side push reaches the cluster only through a human or a long backstop timer. That is +accepted. Bi-directional editing already stops in that state. --- -## Part 3: the options - -Six of them, cheapest first. They are not mutually exclusive, and the order below is roughly the -order in which they could be built. - -### Option 0: do nothing - -**What it is.** Keep §1.4 as shipped. Operators use the annotation when they need a re-read, and -configure their Git host's webhook to point at Flux or Argo CD as they do today. - -**What it costs.** §2.1's two gaps stay open. An idle target holds a stale answer about its folder -for up to 5 minutes plus however long until somebody looks. A refused edit is never reverted on -Argo CD without a human. - -**What it buys.** No new inbound surface, no new secret to handle, no new outbound call, no new -single point of failure, and nothing extra for an operator to configure. +## Part 3: the design -**When it is the right answer.** For audit-only use it is straightforwardly correct: nobody is -editing through the API, so neither gap bites. For bi-directional use it is the honest default -until somebody reports the staleness as a real problem, and the counter in §1.7 is what turns that -report into a number instead of an anecdote. +### 3.1 The applier -**This is the baseline every option below has to beat.** Note that the measured argument for a -notification is weak on traffic alone: told in advance, the worker still has to fetch the moved -branch (three requests) and push (two). It saves exactly one request, the rejected advertisement. -The case for anything here rests on §2.1's staleness, not on §1.6's table. +`GitTarget.spec.applier` names exactly one applier, enforced with a CEL rule: -### Option 1: a maximum age on `baseTrusted` - -**What it is.** Not a new poller. The `GitTarget` already requeues on a 5-minute steady interval and -that pass currently publishes status without touching Git, so it is a scheduled tick looking for a -job. Give `baseTrusted` an age; when the tick finds it too old, clear it. The next cycle fetches. - -**It has to schedule a wake-up, not be consulted lazily.** An idle target is precisely the one with -no arriving writes, so an age tested only at the head of a cycle can never fire on the target it -exists to protect. - -**What it costs.** One fetch per target per interval on targets that are otherwise silent, whether -or not anything changed. That is the cost §1.4 removed, reintroduced at a cadence you choose -rather than per cycle. - -**What it buys.** Bounded staleness with **no Git-host configuration at all**, which no other option -here can say. It is also the safety net that turns a missed webhook into a latency problem rather -than a stuck one, which is the same argument Flux makes for keeping a short `GitRepository` -interval alongside a `Receiver`. +| Field | Applier | +| --- | --- | +| `kustomizationRef` | a Flux `Kustomization` whose `sourceRef` is a `GitRepository` | +| `applicationRef` | an Argo CD `Application` | +| `webhook` (`url`, `secretRef`) | anything else, through a signed POST | -**Consequence worth stating.** If any option below is built, this one should be too, expressed as -the same age rather than as a second mechanism. +plus `quietPeriod` (default `30s`, a Go duration string), used for `CatchUp` (§3.5). -**Built, then removed, and the reason is worth keeping.** A duration flag shipped in -[#383](https://github.com/ConfigButler/gitops-reverser/pull/383) and was taken out again. Expiring a -belief cannot reach the target it exists for: an idle target is converged, nothing publishes, and -so nothing ever spends the fetch the cleared flag merely permits. Making anything happen therefore -meant scheduling a full re-check, which drives a **resync** — a cluster snapshot with -mark-and-sweep — so the knob's real price was a snapshot per target per interval, reaching -freshness through the most expensive door in the building. No supported install could pay it -either: the chart builds `args:` as a closed allowlist with no escape hatch, so the flag was -unreachable from a chart install. +The reference is a declaration, not an inference. The earlier objection to naming an orchestrator +object was to **deriving** which object applies a folder: neither Flux nor Argo CD derives that +mapping, and a guess here would drive a write. A reference the user declares claims nothing we have +to guess. -**What replaced it** renews the observation instead of revoking it: a periodic refresh that reads -the remote's ref advertisement on an idle branch, fetches only when the branch actually moved, and -publishes what it saw on `GitTarget.status.remote`. The option's argument above survives intact — -bounded staleness with no Git-host configuration — but the mechanism under it is one advertisement -rather than a cluster snapshot, and a branch that is actively being written never schedules one at -all, because a successful push has already proved where the remote is. +It is looked up in the **source cluster**, the cluster the applier applies to, through the source +context the watches already use and with the `ClusterProvider`'s identity. An applier living in +another cluster (a hub Argo CD with its `Application` in a management cluster) is out of scope; +see §4.6. -### Option 2: tell the reconciler after we push +### 3.2 The worker state -**What it is.** After a successful `PushAtomic`, ask the downstream to reconcile. Outbound only. +The branch worker gains one piece of state per applier it serves: **the applier is behind**, +holding the commit and the reason it is owed. Three signals set it, and one action clears it: ```mermaid -flowchart LR - subgraph TODAY["Today"] - direction LR - r1["Reverser"] -- push --> g1[("Git")] - g1 -- "push webhook
(user must configure
at the Git host)" --> f1["Flux"] - end - - subgraph WITH["With option 2"] +stateDiagram-v2 direction LR - r2["Reverser"] -- push --> g2[("Git")] - r2 ==>|"reconcile now"| f2["Flux Receiver"] - g2 -. "webhook now optional" .-> f2 - end - - style r1 fill:#e8f4fd,stroke:#2196f3 - style r2 fill:#e8f4fd,stroke:#2196f3 + [*] --> Current + Current --> Behind: foreign commit seen + Current --> Behind: eligible refusal, onRefusal Reapply + Current --> Behind: own push, Flux or webhook applier + Behind --> Behind: newer reason or commit, coalesced + Behind --> Current: request sent on a quiet branch ``` -Flux ships the client for this. `flux trigger receiver` is an HMAC and a POST against a -`generic-hmac` `Receiver`, with the path derived from the token -(`cmd/flux/trigger_receiver.go` in flux2). It needs **no Flux Go types and no ownership claim**, -because the `Receiver` already names the resources to wake. - -**What it costs.** One optional configuration block (a URL, a secret, a flavor), one outbound HTTP -call per successful push, and the discipline in the four rules below. Argo CD is worse, because it -has no generic receiver type: reaching it means synthesizing a host-shaped signed push event, which -is the inverse of option 4 and shares its code. **Start with Flux.** - -**What it buys, and this is the part worth noticing.** The cluster-to-Git-to-cluster loop closes -**without the user configuring a Git host webhook at all.** Today somebody editing through the -Kubernetes API has to set up a webhook at GitHub only to watch their own edit come back. With this, -they do not. That is a quickstart simplification, not only a latency win, and it is the strongest -user-facing argument anywhere on this page. - -**Four rules it does not get to soften:** - -- **It requests reconciliation.** The POST says the reconciler was asked, never that the revision - was applied. No status, log line or metric may claim otherwise. -- **It is asynchronous and bounded,** off the publication path. -- **It retries and reports on its own.** -- **A failed notification never fails a successful publication.** The commit is in Git either way, - and the reconciler's timer is the backstop. - -**What it does not do.** Nothing for gap A. A push we made tells us nothing about a push somebody -else made. - -### Option 3: receive a push notification - -**What it is.** Our own validated endpoint. The Git host (or a relay in front of it) posts -`{repository, branch, after}`; we clear `baseTrusted` and record the believed tip. §4.1 is the wire -contract. - -**All it does is invalidate, and it must not fetch.** Fetching looks like the obvious thing and is -wrong under the budget: it would spend requests immediately on a target that may have nothing to -publish for an hour. Clearing the flag costs zero. The next publication cycle then finds an -untrusted base, fetches once with a real reason, and pushes onto the correct tip instead of -discovering the move through a rejection. +The worker serves every applier declared by a `GitTarget` on its branch, keyed by the applier +object, so two targets naming one `Application` share one entry and one request. -**Its mandatory companion.** The dedupe rule is one comparison, and the bookkeeping behind it is one -SHA. - -| `after` | Meaning | Action | -| --- | --- | --- | -| equals the believed tip | Our own push coming back | Ignore | -| anything else | Somebody moved the branch, or we cannot tell | **Invalidate** | - -That needs `lastCommitSHA` to advance **on a successful push**, which it does not today: it is -written only on fetch paths, so every one of our own publications would look foreign and earn a -fetch. Without that write the feature pays back the round trip it saved. - -**There is deliberately no `before` field.** A set of SHAs we published cannot distinguish our own -delayed delivery from somebody rolling the branch back through commits we published, because both -are pairs drawn from the same set: the information the distinction needs is **order**, which a set -has thrown away. If false invalidations ever measure as a problem, the escalation is an ordered -record, built with a number in hand. - -**What it costs.** An inbound endpoint, per-route secrets, repository-URL normalization, and the -rule from §2.2 written into the handler rather than assumed. - -**What it buys.** Gap A closes within a second of a push, for the price of one more webhook at the -Git host. §5.4 of the old triggering design called that config cost the thing this feature has to -justify, and on §1.6's numbers it does not justify it on traffic. - -**One thing refresh alone does not do.** `syncWithRemote` updates the checkout and the branch -metadata and stops. It does not re-evaluate the acceptance gate or re-resolve placement, because -those reach status today only via the snapshot that §2.2 forbids here. So an idle target would -fetch a fresh tree and report nothing about it. The round-trip-free answer is to **report the -staleness rather than resolve it**: set `status.remote.commit` to the commit the notification named -and leave `lastFetchedAt` where it was. The two disagreeing is the signal, and it is honest. - -### Option 4: take over the webhook and forward it - -**What it is.** The Git host talks only to us, and we forward each delivery onward to Flux or -Argo CD. Option 3 plus an outbound leg and a payload parser. - -```mermaid -flowchart LR - gh[("Git host")] -- "one webhook" --> rev["GitOps Reverser
relay"] - rev -- "verify, then" --> inv["invalidate baseTrusted
(zero round trips)"] - rev -- "forward the exact bytes" --> flux["Flux Receiver"] - rev -- "forward the exact bytes" --> argo["Argo CD /api/webhook"] +**A quiet branch** has no open commit window and no pending writes. It is the only moment a request +may go out, because it is the only moment nothing live is missing from the tip. The worker knows it +on its own goroutine, since every write and every window passes through its queue. - style rev fill:#e8f4fd,stroke:#2196f3 -``` +### 3.3 Learning that somebody else pushed -**The load-bearing reason is not timing: we never have to work out which object syncs the folder.** -Argo CD's `/api/webhook` matches a delivery against every `Application` whose source repository -matches; a Flux `Receiver` fans out to the objects in its `spec.resources`. Forwarding hands routing -to the tool that owns it. - -That matters because **neither orchestrator derives that mapping either.** Argo CD refreshes every -matching `Application` and narrows only when the user supplies the -`argocd.argoproj.io/manifest-generate-paths` annotation; with no annotation, or a payload that did -not list changed files, `AppFilesHaveChanged` returns true and it refreshes -(`util/app/path/path.go`). A Flux `Receiver` derives nothing at all: `spec.resources` is written by -hand. Both either ask the user or fan out, because a redundant refresh is cheap for whoever owns the -cache. An interpreter that derived the answer would be claiming something neither tool claims, to -drive a **write** rather than a refresh, against `resources:` lines and `ApplicationSet` generators -that move without telling us. A stale claim does not fail loudly; it triggers the wrong controller, -or none. - -This repository has already paid for modeling a tool instead of asking it: the write path stopped -re-implementing kustomize's transformers and now asks kustomize what a folder renders to, and three -shipped render bugs went with that change ([`../UPGRADING.md`](../UPGRADING.md)). - -**Forwarding must be transparent.** Each downstream authenticates differently: - -| Argo CD parser | What it verifies | +| Signal | How it reaches the worker | | --- | --- | -| GitHub, Gogs, Bitbucket Server | HMAC over the raw body | -| GitLab | a token compared against the configured secret | -| Azure DevOps | HTTP Basic | -| Bitbucket Cloud | a UUID | - -What they share is the property the relay needs: each checks the request **as the host sent it**, and -none asks the relay to hold a secret. So copy the bytes and the headers, sign nothing, re-serialize -nothing. Adoption is changing a URL at the Git host and keeping the secret. - -Three constraints follow. Cap the buffered body at or above the downstream's own limit (Argo CD -enforces `maxWebhookPayloadSizeB`, and a smaller cap makes the relay the limit). Take the forward -target from configuration and never from the payload. And parse host-native envelopes after all, -for our own `(repository, ref, after)`, behind a per-route declaration of which host's shape to -expect, never behind sniffing; a parse failure forwards anyway. - -**What it costs, and it is the largest cost on this page: we become the only trigger.** Today an -operator outage stops Git from being updated. Here it stops deployments. The reconciler's timer is -then the whole backstop, so it has to stay a real one. -[`../bi-directional.md`](../bi-directional.md) already has the right shape for Flux and for the -right reason: `GitRepository.spec.interval: 1m` with a long `Kustomization` interval, because a new -source revision wakes the dependent Kustomizations. That poll survives untouched. Argo CD's -`timeout.reconciliation` does not, since `0` disables the timed refresh outright, and **"long -intervals" must never be read as "no intervals"**. - -**What it buys.** Gap A, plus the ordering in §2.2 becomes structural instead of a rule, plus one -webhook at the Git host instead of two. - -**What it does not buy: being first in general.** A `GitRepository` poll, an Argo CD refresh timer, -a retry, or an operator running `flux reconcile` all reach Git without us. Forwarding removes the -race between two webhook deliveries, which is the one race that had no other answer. **Nothing may -depend on being first.** - -### Option 5: hold the push while an apply is in flight - -**What it is.** On a foreign push, stop producing commits until the apply has had its chance, since -anything produced there is replayed anyway. - -**It cannot carry correctness, and this is the important part.** Four reasons, all in the tree -today: - -- **Commit windows do not merge across authors.** `canAppend` - ([`open_window.go`](../../internal/git/open_window.go)) requires a matching author, attribution and - target, so an apply under the orchestrator's service account **closes** the user's window and - opens its own rather than collapsing into it. -- **Window timers fire during a hold.** A window closes on silence, not on our permission. -- **Retained writes sit outside any window** and a hold does not reach them. -- **Nothing observes an apply converging.** Knowing a revision landed means comparing live objects - against a rendered revision, deletions included. That is its own piece of work, and it is not - `skipUnchangedLiveUpdate`, which caches the last sanitized hash per object and compares successive - **live** events, never Git. - -**So it is an optimization.** A bounded pause on the **push**, never on the watch, expiring on a -timer, with compare-and-swap and replay unchanged underneath. The test of whether it stayed one is -that skipping it entirely changes no outcome. - -**What it buys.** One doomed commit and one doomed push per foreign push. - -**What it costs.** Latency on unrelated live edits for the length of the hold, unless it is scoped -to the objects the push changed, which needs a local diff of the two tips. - -**Do not stop the event stream to achieve this.** Dropping watch events means re-deriving them, and -re-deriving is a resync, which is the cluster snapshot §2.2 forbids next to a push. - -### Option 6: trigger a reconcile to revert a refused edit - -**What it is.** Gap B. When the flush oracle refuses a write, ask the reconciler to re-apply the -governing object, which reverts the live edit that has no home in Git. - -```mermaid -sequenceDiagram - actor User - participant K8s as Cluster - participant Rev as GitOps Reverser - participant Git - participant Rec as Flux or Argo CD - - User->>K8s: edit a base-owned field - K8s-->>Rev: watch event - Note over Rev: the flush oracle REFUSES:
no legal destination in Git - Rev-->>K8s: GitPathAccepted=False - Note over Git: nothing committed, no push, no webhook - Note over Rec: selfHeal off, so it reverts nothing on its own - Rev->>Rec: TRIGGER a reconcile of the governing object - Rec->>Git: fetch the current revision (unchanged) - Rec->>K8s: re-apply, and the refused edit is reverted +| A receiver delivery (§4.1) | Enqueued as an immediate refresh of that branch | +| A rejected compare-and-swap push | Already on the worker; the replay that follows is the response | +| The periodic refresh | Already on the worker, every `--git-refresh-interval` on an idle branch | + +All three end in the same comparison: the remote's tip against the last commit the worker pushed. +Equal means its own work. Anything else is foreign, including a tip the worker cannot classify. + +**A delivery is a hint, never a fact.** The worker does not trust the commit a delivery names. On +an idle branch it reads the advertisement (one request), compares, and fetches only when the tip is +foreign, exactly as the refresh does. With work pending it does not read anything first: it closes +the open windows and publishes, and the compare-and-swap discovers the move. That makes the hard +cases come out right without bookkeeping: + +- **A late delivery for an earlier push of the worker's own.** The advertisement shows the current + tip, the worker's latest push, so it is own work. +- **A rollback onto one of the worker's older commits.** The tip is not the last push, so it is + foreign, and the rollback is applied. +- **A delivery that arrives before `PushAtomic` returns.** The worker records the commit it is + pushing before it sends, so the comparison is against the commit in flight. +- **The delivery for the replayed push `F + W`.** It names the worker's own push and is dropped, + which is correct: the foreign commit `F` already set the applier behind, and the request is + owed whether or not this delivery arrives. + +A set of SHAs the worker has pushed is deliberately not kept. It cannot tell a late delivery from a +rollback, because the distinction needs order, and the advertisement already supplies order for +one request. + +### 3.4 Foreign commits with work pending + +When the applier falls behind on a foreign commit while windows are open or writes are pending, the +worker does not wait for the windows' timers. A window closes on silence, so a user who keeps +editing would hold a Git-side change back indefinitely. Instead: + +1. Close every open window on the branch, as a resync boundary already does. +2. Publish. The push is rejected, the worker fetches the foreign commit, replays the pending writes + onto it, and pushes `F + W`. +3. The branch is quiet. Request the apply of `F + W`. + +The applied commit carries the API side's answer, so for `W`'s objects the apply is a no-op and +only `F`'s changes land: §1.1's top-left cell, enforced in the cluster as well as in Git. + +If publishing fails, the request waits with the pending writes. It is never sent past them: +applying `F` would revert `W` live, the worker would capture that revert as a newer write, and `W` +would be lost even after the push recovers. The usual cause is an unreachable Git host, which the +applier could not fetch from either. + +### 3.5 The three reasons + +| `reason` | Set by | Requested commit | Sent to | +| --- | --- | --- | --- | +| `ForeignCommit` | §3.3 | The tip once the branch is quiet, after any replay | Every applier on the branch | +| `Refusal` | An eligible refusal (§3.7) on a target with `onRefusal: Reapply` | The tip once the branch is quiet | That target's applier | +| `CatchUp` | An accepted push of the worker's own | The tip once the branch has been quiet for `quietPeriod` | `kustomizationRef` and `webhook` appliers | + +**Why Flux gets `CatchUp` and Argo CD does not.** With automated sync disabled Argo CD applies +nothing on its own, so an `Application` that has not seen GitOps Reverser's own commits has nothing +stale to apply; and every sync runs the `Application`'s hooks, so syncing for no change would run +them for no reason. Flux has no refresh-only mode: its timers and a controller restart apply +whatever artifact the `GitRepository` holds, and with a long interval that artifact falls behind. +Applying a stale artifact reverts **published** edits, not only pending ones, which is far worse. +`CatchUp` keeps the artifact at the tip, so any apply Flux starts on its own applies values the +cluster already holds. + +`quietPeriod` exists because a quiet branch is a moment, not a guarantee. An editor who saves twice +a few seconds apart produces a quiet instant between the first push and the second write; a catch-up +sent then is exactly the race of Part 2. Waiting `quietPeriod`, restarted by every write, keeps the +catch-up out of an editing session. `ForeignCommit` and `Refusal` do not wait it out, since a +Git-side change should land promptly and §3.4 already published everything pending. + +**Never sent:** a request for the worker's own commits to an `applicationRef`; any request while +windows are open or writes are pending; a `Refusal` for a suspended target. + +A suspended target's applier still receives `ForeignCommit`. Suspend stops writes; with the +applier's own triggers off, withholding the request would stall Git-side deployments too. + +### 3.6 What a request does + +**Flux (`kustomizationRef`).** The request `flux reconcile kustomization --with-source` makes: +stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository` named by the `Kustomization`'s +`sourceRef`, wait until its artifact names the requested commit or a later one, then stamp the +`Kustomization` unless it already started reconciling that artifact. A `Refusal` needs the second +stamp, since a reapply at an unchanged commit produces no new artifact to wake it. A `CatchUp` stamps +only the `GitRepository`. Permissions: `get` and `patch` on `gitrepositories` and `kustomizations`. + +**Argo CD (`applicationRef`).** Set `operation` on the `Application` as Argo CD's own automated +sync builds it ([`appcontroller.go`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2628-L2641)), +pinned to one commit: + +```yaml +operation: + initiatedBy: + username: gitops-reverser + sync: + revision: + prune: + syncOptions: + source: # or sources and revisions, for a multi-source Application + retry: ``` -**The operator is uniquely entitled to do this.** Argo CD cannot distinguish authorized drift from -unauthorized, which is why `selfHeal` has to be off. At refusal time we have exactly that signal, -because a refused edit is by construction the drift with no home in Git. So this is the targeted, -per-write substitute for the blanket self-heal that had to be switched off. - -**What it costs, and why it is last.** This is the one option that **cannot** be done by forwarding, -because a refusal produces no push to forward. It needs the ownership interpreters: a claim of the -form "object O reconciles path P", plus the ability to patch O. Flux is a one-annotation trigger -with a correct outcome (`reconcile.fluxcd.io/requestedAt` on the `Kustomization`; a server-side -apply reverts drift as a side effect). **Argo CD needs a sync operation, not a refresh**, because a -refresh with `selfHeal` off only marks `OutOfSync`. That is a write into somebody else's controller -and has to be scoped so it can only ever revert the specific refused object. - -**What it buys.** The only thing that ever reverts a refused edit on Argo CD without a human. - -**Hard boundaries if it is built.** Off by default, per `GitTarget`. Never on a cluster we merely -mirror: we do not get to drive somebody's orchestrator because our mirror is lossy. And an absent -orchestrator is a no-op, not an error. - -### Option 7: push an empty commit - -**What it is.** Gap B again, with none of option 6's machinery. On a refusal, commit nothing and -push it: `git commit --allow-empty` with a message naming what was refused. The branch moves, both -reconcilers notice a new revision, and each applies the same desired state it already had, which -reverts the drift as a side effect. +Pinning the commit means the sync applies exactly what the worker checked, not whatever the tip has +become by the time Argo CD runs it. An operation already in progress is waited for, never replaced, +which is the rule Argo CD's own `SetAppOperation` enforces. `automated.enabled: false` (Argo CD 3.1 +and later) is what keeps `prune` configured while automated sync is off; without it the operation +does not prune. Permissions: `get` and `update` on `applications`. + +**Webhook.** POST `{"repository", "branch", "commit", "reason"}` with +`X-Signature: sha256=`, keyed by the Secret's `token`. That is the +header Flux's `generic-hmac` `Receiver` checks, so a webhook applier can also point straight at +one. A third-party tool needs a small adapter that maps the request to its own trigger. There is no +in-process plugin, because it would run with every credential the operator holds. + +**For all three:** + +- A request asks and never reports. Nothing GitOps Reverser publishes claims the apply happened; + the applier's own status says that. +- It runs off the worker's queue, in `internal/applier`, so a slow applier never holds the branch. +- It retries with backoff, and a newer request for the same applier supersedes one not yet sent. +- A failed request never fails the publication that caused it. +- Each request records an `ApplyRequested` Event and updates `status.applier`. A request that + keeps failing records `ApplyRequestFailed`. + +### 3.7 Refusals: `onRefusal: Reapply` + +A refused edit has no legal destination in Git, so it stays live until something applies Git over +it. `Reapply` asks the target's applier to apply the current tip, which reverts it. It replaces the +earlier `PushEmptyCommit`, which moved the branch with an empty commit so that appliers nobody named +would notice a new commit. With the applier named, that indirection only added blast radius: an +empty commit woke everything watching the branch, while a request wakes one applier. + +The eligibility guards carry over unchanged, because they are about what reapplying does, not about +how it is asked for: + +- Only a write-boundary refusal of an edit to a document the folder already holds. Never a + folder-level refusal, which only a human can fix. +- Never when the same flush removes a document anywhere: reapplying would put back an object + somebody deliberately deleted, or prune one somebody else is deleting. Eligibility is per + document. +- A standing refusal is requested once. The digest of the refused objects and issues is kept, an + identical observation is dropped, and the memory clears when a write of that type is accepted. +- Requests are coalesced per target, not rate-limited away: a refusal arriving while one is + outstanding rides the next request. +- A target that cannot be read is treated as `Ignore`. + +Waiting for a quiet branch is the improvement over the empty commit: an allowed edit still in a +commit window is published before the apply, so it is not reverted with the refused one. + +### 3.8 Validating the applier + +The `GitTarget` controller reads the applier on its reconcile tick and sets `ApplierConfigured`, a +gate: + +| Reason | Check | +| --- | --- | +| `ApplierNotFound` | The object exists in the source cluster, and its kind is served there | +| `SourceMismatch` | The `GitRepository` URL or the `Application`'s `repoURL` matches `GitProvider.spec.url` after normalization, and its branch or `targetRevision` is `spec.branch` | +| `AutomatedSyncEnabled` | The `Application` has automated sync off | +| `Forbidden` | The identity can patch it (a `SelfSubjectAccessReview` in the source cluster) | -```mermaid -sequenceDiagram - participant Rev as GitOps Reverser - participant Git - participant Rec as Flux or Argo CD - participant K8s as Cluster - - Note over Rev: a write is REFUSED:
no legal destination in Git - Rev->>Git: empty commit, message names the refusal - Note over Git: content identical, SHA is new - Git-->>Rec: new revision (webhook, or the next poll) - Rec->>Rec: build: manifests unchanged - Rec->>K8s: apply anyway - Note over K8s: the refused live edit is reverted - K8s-->>Rev: watch event, content now matches Git - Note over Rev: no commit, so no loop -``` +It is read, not watched, so a cluster without the Flux or Argo CD CRDs needs no informer for them. +`AutomatedSyncEnabled` still sends requests: they are correct, only the guarantee is missing. +Intervals are not checked; a short Flux interval is legal, and the guide says what it costs. -**It is the same move as option 4.** We do not name the object that syncs the folder, we move the -branch and let each tool decide what that means. Option 6 has to patch a specific `Kustomization` or -`Application`; this patches nothing and needs no claim about anybody's topology. - -**It works, and here is why, per tool.** - -*Flux is unconditional.* A new commit is a new `GitRepository` artifact revision, which wakes the -dependent Kustomizations, and kustomize-controller server-side-applies the built manifests, which -corrects drift whether or not the build changed. - -*Argo CD works by default and has one exception worth knowing.* The `selfHeal: false` skip in -`autoSync` sits inside `if alreadyAttempted`, and `alreadyAttemptedSync` compares the desired -revision against the last sync's revision (`controller/appcontroller.go`). A new SHA makes them -differ, so the skip never runs and automated sync proceeds. **This is the whole trick: `selfHeal` -gates drift at an unchanged revision, and an empty commit is a changed revision.** - -The exception is `argocd.argoproj.io/manifest-generate-paths`. With it set, `evaluateRevisionChanges` -asks the repo server whether any file under the refresh paths changed between the two revisions -(`controller/state.go`). For an empty commit nothing did, so `revisionsMayHaveChanges` is false, the -revision comparison is skipped, and the sync is treated as already attempted. **An empty commit is -ignored by an Application carrying that annotation.** It is the same annotation option 4 discusses, -which is a coincidence worth remembering rather than a connection. - -**What it buys, and it is a lot for the price.** Option 6's entire cost disappears: no ownership -interpreters, no patching another controller's object, and above all **no Argo CD sync authority**, -which was its worst problem. It needs no webhook, no secret, no URL, and no new configuration of any -kind, because we already hold push credentials. It is the only option on this page that adds zero -configuration surface. The commit message is also an audit trail of refusals, in the place an -operator is already looking. - -**Four costs, and the third is the one to be careful about.** - -- **The blast radius is the branch, not the object.** Everything watching that repository re-applies. - Option 6 wakes one object; this wakes all of them. That is usually harmless and occasionally not. -- **It can revert live edits we have not published yet.** An allowed edit sitting in an open commit - window is reverted along with the refused one. Landing pending intent before the empty commit - shrinks the window to the length of the apply; it does not close it. -- **Pruning is not ours to hurry, and that decides the scope.** Re-applying only corrects drift on - an object Git manages. For one it does not, re-applying does nothing (Flux prunes from its - inventory and Argo CD from the resources it tracks, so a live-created object is in neither), or - prunes it, when the object was managed and has since been removed from Git. The two are - indistinguishable from here and only one is harmless, so **the commit fires only when the folder - already holds a file for every refused object**. An earlier version of this page said the action - deletes such objects and left that as a warning; it is now a fence. -- **The blast radius still reaches other people's prunes.** The commit moves the branch, so - everything watching it re-applies, including another target's pending delete. The fence bounds - what we commit FOR, not what a moved branch causes, and nothing in a branch-wide trigger can. -- **A flapping refusal writes a stream of empty commits.** Debounce per object, and cap it. - -**What it does not do.** Nothing for gap A, which is about learning that somebody else pushed. - -**How it degrades.** Without a Git-host webhook it still works, on the reconciler's own clock: -Flux notices within its `GitRepository` interval, Argo CD within `timeout.reconciliation`. So it -needs no webhook to be correct, only to be fast. - -**Where this leaves option 6.** Mostly superseded. It survives only where waking exactly one object -matters and waking the branch is unacceptable, and that case now has to argue for the interpreters -on its own rather than inheriting the need from gap B. - -**Built, as `GitTarget.spec.onRefusal: PushEmptyCommit`,** defaulting to `Ignore`. Four guards ship -with it, and the first two are the ones this section argued for: - -- **Off unless asked for**, because of the `prune` row above. -- **Only for a refusal carrying evidence the document exists.** The refusal KIND is not enough, and - a review found the hole: a render refusal fires for a brand-new resource an `images:` entry would - override, and a write that is REMOVING a document had one before it. Both would have been - committed for. The evidence is set where the pre-write buffer is in hand and carried on the issue. - It is evidence about the DOCUMENT, not the file, which a second review found the hole in too: - removing one document from a two-document file leaves the file present, so file survival answered - "an edit to a surviving document" for what was plainly a removal. A flush that removes a document - by any route — an explicit `DELETE`, a mark-and-sweep drop, a `$patch: delete` authored into an - overlay — earns no commit at all, because the refusal means Git still holds the document and - re-applying would put back an object somebody deliberately deleted. -- **An unchanged refusal is one trigger, not one per recheck.** The rate limit decides how often a - commit may be made; it cannot tell whether the refusal is the same one. A forced recheck re-lists - every interval, so a refused edit nobody corrects is observed over and over, and each observation - used to be another commit. Each commit carries the digest of what it covered — the objects the - refused write would have produced, and the issues raised about them — and an observation matching - it is dropped. The digest is CONTENT: a re-list gives every snapshot a fresh collection version, - so a version would make every recheck look new. The memory is dropped again as soon as a write - for that target is ACCEPTED, which is what makes an identical edit, re-made after the reconciler - reverted the first one, a new trigger rather than a repeat. -- **Rate limited per target, and COALESCED rather than dropped.** A controller rewriting a - base-owned field refuses on every one of its own reconciles, and every commit wakes every - reconciler watching the branch. But a refusal discarded inside that window is lost: the reconcile - an earlier commit triggered may already have finished, so nothing would ever cover it. One - trailing commit covers every refusal that arrived during the window, and re-checks consent when - it fires. -- **Never for a suspended target.** An empty commit is a write, and it is the one write that - reaches outside this operator, so the state that means "write nothing" has to stop it. -- **A GitTarget that cannot be read is treated as opted out.** Missing evidence is not consent for - an action that deletes where the reconciler prunes. - -The commit message explains its own empty diff, so `git log` does not show what looks like a stray -no-op. - -### Option 8: hand the action to the installer - -**What it is.** Stop deciding what a refusal should cause. Emit the fact, with enough detail to act -on, and let whoever installed the operator decide. We then need to know nothing about Argo CD or -Flux for gap B at all. - -**The instinct is right and it is the same one behind options 4 and 7:** do not model somebody -else's system, hand them the decision. What it turns on is the *mechanism*, and the three candidates -are not equivalent. - -#### 8a. Execute a script mounted from a ConfigMap: declined - -The obvious shape: mount a script, run it on refusal, pass the `GitTarget` and the destination as -arguments. It is declined, and recorded here so it is not re-proposed. - -**It is arbitrary code execution as the operator.** The operator process holds Git push credentials, -SSH private keys, SOPS and age keys, and the kubeconfigs of every mirrored remote cluster, and its -service account watches the whole cluster. A script mounted from a ConfigMap runs with all of that. -Anyone who can write that one ConfigMap can exfiltrate every credential the operator holds. That is -a privilege escalation from "edit a ConfigMap in one namespace" to "own every Git repository and -cluster this operator touches", and no amount of care in the handler changes it. - -**Argo CD already ran this experiment.** Config management plugins were configured in `argocd-cm`, -were deprecated in 2.4 and **removed completely in 2.8**, and the replacement is a sidecar container -with its own image and security context, not a mounted script. Flux does not offer the pattern at -all. - -Three smaller objections, any one of which would still need answering: - -- **It blocks the branch queue.** Publication for a branch is one serialized worker. A script that - hangs stalls every write for that branch, so this needs timeouts, concurrency limits and an answer - for "the script is broken" before it is a feature. -- **It fixes the base image.** The runtime is `distroless/static:debug`, whose shell is busybox from - the debug tag. Depending on that for a product feature pins the image to `:debug` and turns every - future attack-surface review into a conversation about it. -- **We cannot test the outcome.** Our e2e could assert that the hook was called and nothing about - what it did, so the product's promise for gap B would shrink to "we told someone". - -#### 8b. Emit a structured event: recommended, and nearly free - -The operator already has an `EventRecorder` and already emits Events on condition transitions -([`status.go`](../../internal/controller/status.go)). A refusal already sets `GitPathAccepted=False`. -What is missing is an Event carrying enough to act on. - -**The payload has to say more than which target refused.** A handler that knows only the `GitTarget` -and the destination cannot do anything specific. It needs the refused object (group, kind, namespace, -name), the reason, and the author, because the useful actions are all per-object: revert this one -resource, notify the person who made the edit, open a ticket naming the field. - -The installer then wires whatever they want to it, with no new privilege anywhere: a small -controller watching Events, a Prometheus alert into Alertmanager, or a Flux `Alert` in -notification-controller. This is the idiomatic Kubernetes seam, it costs us one call, and it is -testable on our side because "we emitted this Event with these fields" is an assertion. - -The honest trade against 8a: the installer has to run *something*, rather than dropping a shell -script into a ConfigMap. That is the price of not putting an RCE in the operator. - -#### 8c. Notify an endpoint the installer configures - -The same idea over HTTP, and it is **option 2's outbound leg with a second event type**: the same -configuration object, the same HMAC, the same retry and the same rule that a failed notification -never fails a publication. Point it at a Flux `Receiver` after a push, or at the installer's own -endpoint after a refusal. - -Worth building only once option 2 exists, at which point it is small. Until then 8b covers the same -ground without any new machinery. - -#### How this composes - -Option 7 is the sensible **default** action for a refusal and option 8 is the **escape hatch** for -installers who want something else, so they are not competitors. Between them, gap B is covered -without the operator ever naming a `Kustomization` or an `Application`, which is what option 6 -needed the ownership interpreters for. **On this reading option 6 may never be built at all**, and -the interpreters would then have to justify themselves on some other feature rather than on this one. - -### 3.7 Summary - -| | Closes gap | New inbound surface | New outbound call | Git-host config | Needs ownership interpreters | -| --- | --- | --- | --- | --- | --- | -| **0. Do nothing** | neither | no | no | unchanged | no | -| **1. Max age on `baseTrusted`** | A, bounded | no | no | **none** | no | -| **2. Tell the reconciler after we push** | neither | no | yes | **removes one** | no | -| **3. Receive a notification** | A | yes | no | adds one | no | -| **4. Take over and forward** | A | yes | yes | **replaces one** | no | -| **5. Hold the push** | neither | no | no | unchanged | no | -| **6. Trigger on refusal** | B | no | yes | unchanged | **yes** | -| **7. Push an empty commit** | B | no | no | **none** | no | -| **8b. Emit a structured Event** | B, by delegation | no | no | **none** | no | -| **8c. Notify an endpoint** | B, by delegation | no | yes | **none** | no | - -Options 2, 3 and 4 share one configuration object and one piece of bookkeeping (`lastCommitSHA` -advancing on a successful push), so whichever is built first should shape both. - -**Option 7 is the cheapest thing on this page and it closes gap B on its own**, which is not where -this page expected to end up: gap B was the one that looked like it needed the ownership -interpreters. It adds no configuration at all. Read its four costs before believing that, in -particular what it does under `prune: true`. - -**Options 7 and 8 together are why option 6 may never be built.** One gives a sensible default -action and the other gives installers a seam for anything else, and neither requires the operator to -name an orchestrator object. Option 6 remains the only way to wake exactly one object rather than -the branch, and that is now the whole of its case. +URL normalization is the receiver's (§4.1): case-insensitive host and path, no trailing `.git`, +`git@host:org/repo` equal to `https://host/org/repo`, default ports ignored. --- ## Part 4: reference -### 4.1 The wire contract for option 3 - -Two readers need this before any code exists: whoever builds the handler, and whoever points -something at it. +### 4.1 The receiver -**Where it listens.** The operator runs two HTTPS listeners: the controller-runtime webhook server -(admission and conversion, called by the API server) and the audit server (its own `Service` and -certificate, called by the API server's audit backend, possibly off-cluster). **The receiver belongs -on the audit listener.** It is the same shape of traffic: an inbound POST from something that is not -the API server, authenticated by a shared secret. Putting it on the admission server would expose -the listener the API server trusts to a Git host. That is a recommendation; an implementer who -disagrees should record why here rather than landing it on the admission server because `Register` -was easier to call. +`GitProvider.spec.receiver` (`type`, `secretRef`), named after Flux's `Receiver`, served on its own +listener (`--receiver-bind-address`, Helm `receiver.bindAddress`, off by default). It cannot share +the audit listener, which requires a client certificate signed by the audit CA that no Git host can +present. ```text -POST /git-push/ -Content-Type: application/json -X-Reverser-Signature: sha256= +POST /git-push// ``` -```json -{ - "repository": "https://github.com/acme/infra.git", - "branch": "main", - "after": "9f7fc13494903993371f49caa67bd08051987cd3" -} -``` - -| Field | Meaning | -| --- | --- | -| `repository` | In any form the host uses. Matched against `GitProvider.spec.url` after normalization, never by string equality | -| `branch` | The short name, `main`, not `refs/heads/main` | -| `after` | The SHA the branch now points at, or forty zeros for a deletion | +`GitProvider.status.webhookPath` publishes the path. -`` names a receiver configuration the way `/audit-webhook/` does today, so several -hosts or organizations can deliver to one operator with separate secrets. It is a path segment so a -proxy can route on it and a log line identifies the sender without a body. - -**A host-native payload is not accepted here.** The three fields above are trivially derivable from -every host's envelope, and whoever wires a host up writes that mapping once. (Option 4 changes this, -because there we *are* the thing in front.) +| `type` | Verifies | Body read for | +| --- | --- | --- | +| `github` | `X-Hub-Signature-256`, HMAC-SHA256 of the raw body | `ref`, `after` | +| `gitlab` | `X-Gitlab-Token` equals the token | `ref`, `after` | +| `gitea` | `X-Gitea-Signature`, HMAC-SHA256 of the raw body | `ref`, `after` | +| `generic-hmac` | `X-Signature: sha256=`, HMAC-SHA256 of the raw body | `repository`, `branch`, `after` | -**A deletion is a movement like any other.** Forty zeros says the branch is gone, and the handler -invalidates as it would for any value it does not believe. +The token is the Secret's `token` key, the key a Flux `Receiver` Secret uses. -**Authentication.** Hex HMAC-SHA256 of the exact body, keyed by the route's secret, compared in -constant time and computed over the raw bytes before any JSON decoding: re-serializing changes them. -Three rules that do not soften. An unsigned or wrongly signed request is refused and nothing about it -is acted on, because an endpoint accepting anonymous "this moved" claims is a way to make a target -fetch on demand. A route with no configured secret does not serve. And verification happens before -the body is parsed for anything else, including the repository lookup, so an unsigned request cannot -probe which repositories this operator tracks. +**Authentication comes first.** The signature is verified over the raw bytes before the body is +parsed for anything, in constant time. An unsigned or wrongly signed request is refused and nothing +about it is acted on, so the endpoint cannot be used to make a target do work on demand or to probe +which branches are tracked. A route whose `GitProvider` has no `spec.receiver` does not serve. | Status | When | | --- | --- | -| `202 Accepted` | Verified and applied, **including when it matched no branch worker** | -| `400` | Not JSON, or a required field missing or malformed | +| `202 Accepted` | Verified, including when it named a branch no `GitTarget` writes, a tag, or another repository's events on a shared hook | +| `400` | Verified but not parseable, or a required field missing | | `401` | Signature absent or wrong | | `404` | No such route | -| `503` | Not ready to serve | - -**A delivery that matches nothing is a success.** A `GitTarget` may be suspended, the branch may be -one nobody mirrors, or the repository may not be ours. None is the caller's fault, and an error -teaches a Git host to disable the hook. The body says `{"matched": 0}`, which is what makes a -misconfigured hook debuggable and is the argument for a counter labeled by route. - -**Normalization is where matching goes wrong**, so state it: compare case-insensitively on host and -path, strip a trailing `.git`, strip a leading `git@` or `ssh://git@`, and ignore a default port. -`git@github.com:acme/infra.git` and `https://github.com/acme/infra` are the same repository. - -**Delivery is unreliable and the handler is built for that.** Deliveries arrive out of order, twice, -or never. Invalidation is idempotent, so applying it twice is applying it once. A missed delivery -costs freshness, never correctness, because the compare-and-swap is still what makes a stale base -safe. The tests are reordered delivery, dropped delivery, a duplicate, and a rollback that passes -only through SHAs we published. - -**Calling it by hand:** - -```bash -BODY='{"repository":"https://github.com/acme/infra.git","branch":"main","after":"9f7fc134..."}' -SIG=$(printf '%s' "$BODY" | openssl dgst -sha256 -hmac "$RECEIVER_SECRET" -hex | awk '{print $2}') - -curl -sS -X POST "https://reverser.example.com/git-push/acme" \ - -H 'Content-Type: application/json' \ - -H "X-Reverser-Signature: sha256=$SIG" \ - --data "$BODY" +| `503` | Not the elected replica, or not ready | + +A verified delivery that matches nothing is a success, because it is not the host's fault and an +error teaches a host to disable the hook. A `generic-hmac` body naming a repository other than +`spec.url` is ignored, not refused. + +**Delivery is unreliable, and correctness does not depend on it.** Deliveries arrive late, twice, +out of order or never. Each one is only a request to look (§3.3), so a duplicate is one more +advertisement, and a missed one costs latency until the refresh or the next push finds the move. + +### 4.2 Status + +```yaml +status: + conditions: + - type: ApplierConfigured + status: "True" + reason: Succeeded + applier: + requestedCommit: 4f2c1ab9... + requestedAt: "2026-10-20T10:14:03Z" + reason: ForeignCommit # ForeignCommit | Refusal | CatchUp + remote: + commit: 4f2c1ab9... + lastVerifiedAt: "2026-10-20T10:14:02Z" + verifiedBy: Push # Push | Fetch ``` -### 4.2 `status.remote` +`status.applier` is an observation of what was requested, sampled on the target's reconcile like +`status.remote`. It never says what was applied. -An operator who can no longer assume a fetch per cycle needs to see when the remote was last read. -The worker tracks it and does not surface it. +### 4.3 Metrics -| Field | Source | Meaning | +| Family | Labels | Reads as | | --- | --- | --- | -| `revision` | `lastCommitSHA` | The branch head as of the last read | -| `lastFetchedAt` | `lastFetchTime` | When that read happened | -| `branchExists` | `branchExists` | False while the branch is unborn | - -Name it fetch rather than pull: nothing merges. The doc comment should say it dates the **read**, so -a timestamp well in the past means nothing has looked, which after §1.4 is the normal state of a -healthy quiet target. A successful push advances `revision` too, which is the same write option 3 -depends on. - -### 4.3 What has to be proven - -The [bi-directional corner](../../test/e2e/flux_bi_directional_e2e_test.go) is the only place with -both reconcilers against a real remote. Whatever is built from part 3 needs these, and the last one -must never be allowed to lapse: - -0. **A new revision is enough** (option 7's load-bearing claim, and the only part of it real Flux - can answer). Drift a live object, push a commit that changes no file, wake only the - `GitRepository`, and assert the `Kustomization` applies on its own and reverts the drift. The - `GitTarget` is suspended first, or the operator publishes the drift and there is nothing left to - revert; and the `Kustomization` is never reconciled by hand, or it would apply for a reason that - has nothing to do with the new revision. **Built**, in the Flux corner. -0b. **A real refusal producing the operator's own empty commit.** Not the foreign-file refusal an - earlier version of this page described: that is folder-level, the filter excludes it, and it is - normally found by the per-type reconcile rather than by a write. The spec uses a **write-boundary** - refusal instead, from corpus shape 8: an overlay whose Deployment comes from `../../base`, and a - drifted env var that no overlay declaration can express. It asserts the refusal kind and message, - the commit, its empty diff, and that the commit names the `GitTarget`. It then asserts settling - **past** the one-minute trailing interval, and that a corrected object leaves the branch alone, - because a shorter window only proves the rate limit works. **Built**, as its own manager spec, - not in the Flux corner. -0c. **A refusal alongside accepted work, under contention.** Deliberately NOT an e2e: the property - is about arrival order between our push and somebody else's, which the corner does not control. - Over a real Git server the contending commit lands between our commit and our push, so the - rejection is deterministic, and the replay must keep the accepted write, bring the empty commit - back still empty, and rebase onto the other writer rather than over them. **Built**, in - `internal/git`. -1. **The hazard cell.** A push changes an object we hold no pending write for. It survives in Git - and reaches the cluster. This is §2.2 written as a test. -2. **The contested cell, as decided.** A push changes an object we do hold a captured write for, and - the API value wins. This pins §1.2 rather than discovering it. -3. **Somebody else gets there first.** The reconciler learns the revision from its own poll while a - hold is in effect or before a relay forwards. Nothing breaks, because nothing depends on order. -4. **The webhook is absent.** The same foreign push with no notification delivered at all; assert - the worker still recovers on its next push through rejection, replay and a clean landing. This is - the guarantee that a missing or broken webhook degrades to correct-but-slower rather than to data - loss, and it is the one that must stay green for the life of any feature here. - -### 4.4 Open questions - -1. **Which object carries a receiver's configuration?** A push moves a branch and several targets - can track it. The worker layer already maps repo-and-branch to targets, so this does not block - the handler, but the secret has to hang somewhere and the candidates differ in blast radius. -2. **Under option 4, serve from every replica or only the leader?** Forwarding is idempotent and - needs no worker; the barrier needs the replica that owns the branch worker. -3. **Under option 4, what about repositories we do not track?** The host has one webhook now, so - deliveries arrive for repositories only the reconciler cares about. They must be forwarded, which - makes the configuration per route rather than per `GitProvider`. -4. **Should `baseTrusted` survive a worker restart?** **Answered: no, and the question is closed.** - Repositories are cloned into an empty directory, so every restart starts from a fresh checkout - with nothing to carry forward. Keeping the flag alive across a pod replacement would mean making - the clone durable first, and one fetch is not expensive enough to justify that. -5. **How confident must an ownership claim be before we *write* on it?** Option 6 only. A wrong - claim triggers the wrong controller, and this is a higher bar than a claim used to read. -6. **Does the default-branch fallback deserve better than untrusted?** **Answered: yes, and it is - now trusted.** See §1.6's last row: it was four requests per cycle and is two. +| `gitopsreverser_apply_requests_total` | `applier_kind`, `reason`, `outcome` | Requests decided. `outcome`: `sent`, `failed`, `superseded` | +| `gitopsreverser_receiver_deliveries_total` | `type`, `outcome` | Deliveries. `outcome`: `foreign`, `own`, `ignored`, `rejected` | +| `gitopsreverser_git_fetches_total` | `reason` | Unchanged. A delivery that finds a foreign tip on an idle branch fetches under `refresh` | + +These are recorded in [interpreting-metrics.md](../interpreting-metrics.md) when they ship. + +### 4.4 Chart + +- `receiver.bindAddress` and a `gitops-reverser-receiver` Service when it is set. Exposing it is the + installer's Ingress or Gateway route. +- RBAC for the operator's own cluster: `get` and `patch` on `kustomize.toolkit.fluxcd.io` + `kustomizations` and `source.toolkit.fluxcd.io` `gitrepositories`; `get` and `update` on + `argoproj.io` `applications`. Off unless asked for, since most installs run one of the two. + +### 4.5 What has to be proven + +In the bi-directional corner, against real Flux and Argo CD configured as the guide says: + +1. **The original race.** Patch one field to `1`, then to `2` within two seconds of the first push. + After both commits and two applier intervals, the live value and Git hold `2`, and no commit + after the second patch is authored by the applier. Run for both appliers. +2. **Argo CD's timer does nothing.** With `timeout.reconciliation` at 10 seconds and edits in + flight, the `Application`'s operation history holds only GitOps Reverser's syncs. +3. **A foreign push with an edit pending.** Both land, and the applier applies the replayed commit. +4. **Catch-up.** After `quietPeriod` the `GitRepository` artifact holds the tip, and a forced + `Kustomization` reconcile changes nothing and produces no commit. +5. **Refusal.** An eligible refused edit is reverted by `Reapply`, only its own applier is asked, + and an allowed edit made just before it reaches Git. +6. **No receiver.** A foreign push with no delivery at all is still applied once the refresh finds + it. This is the guarantee that a missing webhook degrades to latency, never to loss, and it must + stay green for the life of the feature. + +In `internal/git` over a real Git server (not `file://`, whose in-process receive-pack never +compares `Old`): the delivery cases of §3.3, the window-closing of §3.4, coalescing and +superseding, and that no request leaves a branch with pending writes. + +### 4.6 Open questions + +1. **An applier in another cluster.** Hub Argo CD keeps `Application`s in a management cluster. + Supporting it means a second identity, the control cluster's or a kubeconfig, on + `spec.applier`. Deferred until someone runs that topology. +2. **High availability.** The decision lives on the replica that owns the branch worker, so a + delivery reaching another replica answers `503` today. With multiple replicas, deliveries need + routing to the owner, or a shared queue in Redis. +3. **Path scoping for `ForeignCommit`.** Every applier on the branch is asked, whether or not the + foreign commit touched its folder. That is safe, since the branch is quiet, but it runs every + `Application`'s hooks. Narrowing it needs the applier's own source paths, which are not ours to + interpret; revisit only with a measured complaint. + +### 4.7 Build order + +1. Worker state, the quiet-branch rule and `status.applier`, with the Flux client. This alone fixes + the original race for Flux users. +2. The receiver, on its own listener. +3. The Argo CD client, and the guide's Argo CD configuration. +4. `onRefusal: Reapply` in place of `PushEmptyCommit`, with the upgrade note. +5. The webhook applier. From 75eac7a481508986b729d07c645f9466df89032e Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 08:59:14 +0000 Subject: [PATCH 04/36] feat!: GitOps Reverser decides when Flux applies a folder A GitTarget can now name the Flux Kustomization that applies its folder (spec.applier.kustomizationRef). The branch worker then decides when it applies: only on a quiet branch, with no open commit window and no pending writes, so an apply never reverts an edit that is live and not yet pushed. It asks for three reasons: - ForeignCommit: the branch gained a commit this worker did not push. Open windows are closed and pending writes replayed onto it first, so Flux applies a commit that already carries the API side's answer. - Refusal: a refused edit on a target with spec.onRefusal: Reapply. - CatchUp: after this worker's own pushes, once the branch has been quiet for spec.applier.quietPeriod (30s), so Flux's artifact never falls behind and its own timers and restarts apply values the cluster already holds. A request stamps reconcile.fluxcd.io/requestedAt on the GitRepository, or on the Kustomization when the artifact already holds the commit, as `flux reconcile --with-source` does. It is delivered off the worker's queue by a dispatcher in internal/applier, retried, superseded by newer requests, recorded as an ApplyRequested Event and in status.applier. The ApplierConfigured condition checks the Kustomization in the source cluster. GitProvider.spec.receiver accepts the Git host's push webhook (github, gitlab, gitea, generic-hmac) on a new --receiver-bind-address listener and publishes status.webhookPath. A delivery is a hint: the worker reads the advertisement and judges from the remote whether the branch moved. BREAKING CHANGE: spec.onRefusal PushEmptyCommit is replaced by Reapply, which requires spec.applier and asks that one Kustomization to reapply instead of moving the branch with an empty commit. A stored PushEmptyCommit reads as Ignore. See docs/UPGRADING.md. Co-Authored-By: Claude Opus 5.5 (1M context) --- api/v1alpha3/gitprovider_types.go | 54 ++ api/v1alpha3/gittarget_types.go | 155 +++-- api/v1alpha3/zz_generated.deepcopy.go | 71 +++ charts/gitops-reverser/README.md | 3 + charts/gitops-reverser/templates/_helpers.tpl | 24 + .../templates/applier-rbac.yaml | 54 ++ .../gitops-reverser/templates/deployment.yaml | 8 + .../gitops-reverser/templates/services.yaml | 24 + charts/gitops-reverser/values.schema.json | 29 +- charts/gitops-reverser/values.yaml | 26 + cmd/main.go | 16 + cmd/main_chart_receiver_test.go | 198 +++++++ cmd/receiver.go | 156 +++++ cmd/receiver_test.go | 97 ++++ .../bases/configbutler.ai_gitproviders.yaml | 40 ++ .../crd/bases/configbutler.ai_gittargets.yaml | 120 +++- config/deployment.yaml | 6 + config/kustomization.yaml | 1 + config/rbac/flux-applier-role.yaml | 36 ++ config/rbac/flux-applier-rolebinding.yaml | 15 + config/rbac/kustomization.yaml | 3 + config/receiver-service.yaml | 20 + docs/UPGRADING.md | 23 + docs/architecture.md | 62 +- docs/bi-directional.md | 107 ++-- docs/configuration.md | 114 ++-- docs/design/branch-worker-event-model.md | 2 +- docs/design/collection-terminology-rename.md | 8 +- docs/design/label-selection-follow-ups.md | 2 +- ...push-notification-and-reconcile-trigger.md | 50 +- docs/interpreting-metrics.md | 27 +- hack/crdfields/fields.yaml | 11 + internal/applier/dispatcher.go | 546 ++++++++++++++++++ internal/applier/dispatcher_test.go | 397 +++++++++++++ internal/applier/flux.go | 433 ++++++++++++++ internal/applier/flux_test.go | 494 ++++++++++++++++ internal/controller/gitprovider_controller.go | 14 + .../controller/gitprovider_controller_test.go | 38 ++ .../gitprovider_controller_unit_test.go | 22 + internal/controller/gittarget_applier.go | 130 +++++ internal/controller/gittarget_applier_test.go | 204 +++++++ internal/controller/gittarget_controller.go | 17 + internal/git/apply_request.go | 30 + internal/git/apply_trigger.go | 335 +++++++++++ internal/git/apply_trigger_test.go | 368 ++++++++++++ internal/git/branch_worker.go | 60 +- internal/git/commit_executor.go | 3 - internal/git/dirty_worktree_recovery_test.go | 12 - internal/git/pending_writes_loop.go | 51 +- internal/git/plan_flush.go | 11 +- internal/git/push_notification.go | 19 + internal/git/refresh.go | 9 + internal/git/refusal_observation_test.go | 226 ++++---- .../{refusal_touch.go => refusal_reapply.go} | 270 +++------ ..._touch_test.go => refusal_reapply_test.go} | 359 ++++++------ internal/git/types.go | 8 +- internal/git/worker_manager.go | 26 + internal/repourl/repourl.go | 107 ++++ internal/repourl/repourl_test.go | 69 +++ internal/telemetry/apply_requests.go | 28 + internal/telemetry/exporter.go | 2 + internal/telemetry/receiver.go | 28 + internal/watch/cluster_context.go | 35 ++ internal/watch/source_dynamic_client_test.go | 59 ++ internal/webhook/receiver.go | 521 +++++++++++++++++ internal/webhook/receiver_test.go | 483 ++++++++++++++++ test/e2e/flux_bi_directional_e2e_test.go | 106 +++- ...2e_test.go => refusal_reapply_e2e_test.go} | 226 ++++---- 68 files changed, 6373 insertions(+), 935 deletions(-) create mode 100644 charts/gitops-reverser/templates/applier-rbac.yaml create mode 100644 cmd/main_chart_receiver_test.go create mode 100644 cmd/receiver.go create mode 100644 cmd/receiver_test.go create mode 100644 config/rbac/flux-applier-role.yaml create mode 100644 config/rbac/flux-applier-rolebinding.yaml create mode 100644 config/receiver-service.yaml create mode 100644 internal/applier/dispatcher.go create mode 100644 internal/applier/dispatcher_test.go create mode 100644 internal/applier/flux.go create mode 100644 internal/applier/flux_test.go create mode 100644 internal/controller/gittarget_applier.go create mode 100644 internal/controller/gittarget_applier_test.go create mode 100644 internal/git/apply_request.go create mode 100644 internal/git/apply_trigger.go create mode 100644 internal/git/apply_trigger_test.go create mode 100644 internal/git/push_notification.go rename internal/git/{refusal_touch.go => refusal_reapply.go} (63%) rename internal/git/{refusal_touch_test.go => refusal_reapply_test.go} (53%) create mode 100644 internal/repourl/repourl.go create mode 100644 internal/repourl/repourl_test.go create mode 100644 internal/telemetry/apply_requests.go create mode 100644 internal/telemetry/receiver.go create mode 100644 internal/watch/source_dynamic_client_test.go create mode 100644 internal/webhook/receiver.go create mode 100644 internal/webhook/receiver_test.go rename test/e2e/{refusal_empty_commit_e2e_test.go => refusal_reapply_e2e_test.go} (55%) diff --git a/api/v1alpha3/gitprovider_types.go b/api/v1alpha3/gitprovider_types.go index 59581a95..649181ca 100644 --- a/api/v1alpha3/gitprovider_types.go +++ b/api/v1alpha3/gitprovider_types.go @@ -54,6 +54,55 @@ type GitProviderSpec struct { // formatting moved to GitTarget.spec.commit.message. // +optional Commit *CommitSpec `json:"commit,omitempty"` + + // The names are Flux's Receiver's, type values included, so a Flux user already knows them. + + // Receiver lets the Git host tell GitOps Reverser that somebody pushed. Point the host's push + // webhook at the receiver listener plus status.webhookPath, with the Secret's token as the + // webhook secret. A delivery is only a hint: it makes GitOps Reverser read where the branch is + // on the remote, and what the remote says decides whether a GitTarget's applier is asked to + // apply. Without a receiver, a foreign push is still found, by the next rejected push or by + // the periodic refresh. + // +optional + Receiver *GitProviderReceiver `json:"receiver,omitempty"` +} + +// GitProviderReceiver is the push-notification endpoint of a GitProvider. +type GitProviderReceiver struct { + // Type is the payload and signature the Git host sends. "github" checks X-Hub-Signature-256, + // "gitlab" checks X-Gitlab-Token, "gitea" checks X-Gitea-Signature, and "generic-hmac" checks + // X-Signature: sha256= over a {"repository", "branch", "after"} body, + // for a relay of your own. + // +required + // +kubebuilder:validation:Enum=github;gitlab;gitea;generic-hmac + Type ReceiverType `json:"type"` + + // SecretRef names a Secret in this namespace whose "token" key is the webhook secret. + // +required + SecretRef meta.LocalObjectReference `json:"secretRef"` +} + +// ReceiverType is the payload format a receiver accepts. The values are Flux's Receiver types. +type ReceiverType string + +const ( + // ReceiverTypeGitHub is a GitHub push event. + ReceiverTypeGitHub ReceiverType = "github" + // ReceiverTypeGitLab is a GitLab push event. + ReceiverTypeGitLab ReceiverType = "gitlab" + // ReceiverTypeGitea is a Gitea push event. + ReceiverTypeGitea ReceiverType = "gitea" + // ReceiverTypeGenericHMAC is a {"repository", "branch", "after"} body signed as Flux's + // generic-hmac Receiver expects. + ReceiverTypeGenericHMAC ReceiverType = "generic-hmac" +) + +// ReceiverTokenKey is the Secret key holding a receiver's token, the key a Flux Receiver Secret uses. +const ReceiverTokenKey = "token" + +// ReceiverPath is the path a GitProvider's receiver is served on. +func ReceiverPath(namespace, name string) string { + return "/git-push/" + namespace + "/" + name } // KnownHostsReference points at a namespace-local ConfigMap or Secret that holds SSH known_hosts @@ -166,6 +215,11 @@ type GitProviderStatus struct { // that cadence. Providers are few and the check was already running; this adds no request. // +optional LastVerifiedAt *metav1.Time `json:"lastVerifiedAt,omitempty"` + + // WebhookPath is the path spec.receiver is served on, under the receiver listener. Empty + // without spec.receiver. + // +optional + WebhookPath string `json:"webhookPath,omitempty"` } // GitProviderBranchStatus is one branch of this repository that GitTargets are configured to diff --git a/api/v1alpha3/gittarget_types.go b/api/v1alpha3/gittarget_types.go index 7633ebb5..291b05ef 100644 --- a/api/v1alpha3/gittarget_types.go +++ b/api/v1alpha3/gittarget_types.go @@ -3,6 +3,8 @@ package v1alpha3 import ( + "time" + meta "github.com/fluxcd/pkg/apis/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) @@ -35,6 +37,7 @@ import ( // gitProviderRef/branch/path above — and defaults to a ClusterProvider named "default", so it is // always populated (never nil) and always jumpable. // +kubebuilder:validation:XValidation:rule="self.clusterProviderRef == oldSelf.clusterProviderRef",message="spec.clusterProviderRef is immutable; delete and recreate the GitTarget to change the cluster it mirrors" +// +kubebuilder:validation:XValidation:rule="!has(self.onRefusal) || self.onRefusal != 'Reapply' || has(self.applier)",message="spec.onRefusal: Reapply needs spec.applier, the object asked to reapply" type GitTargetSpec struct { // GitProviderRef names the GitProvider that backs this target, in this GitTarget's own // namespace. Many GitTargets may name the same GitProvider. @@ -85,52 +88,39 @@ type GitTargetSpec struct { // +optional Encryption *EncryptionSpec `json:"encryption,omitempty"` - // Why this is an empty commit rather than a direct trigger of the reconciler: naming the - // Kustomization or Application that renders a folder would mean deriving a mapping neither - // Flux nor Argo CD derives (Argo refreshes every Application whose repository matches and - // narrows only on an annotation; a Flux Receiver names its resources by hand), and then - // writing to somebody else's object on the strength of it. Moving the branch asks each tool - // instead of modelling it. See docs/design/push-notification-and-reconcile-trigger.md, part 3 - // option 7. + // Why a reference rather than an inference: deriving which Kustomization applies a folder would + // be guessing on Flux's behalf, and the guess would drive a write. A reference the user declares + // claims nothing. See docs/design/push-notification-and-reconcile-trigger.md §3.1. + + // Applier names the object that applies this folder back to its source cluster. With it set, + // GitOps Reverser decides when that object applies: it asks for an apply only when the branch + // holds no open commit window and no pending writes, after a commit it did not push, after a + // refused edit (spec.onRefusal: Reapply), and, once the branch has been quiet for quietPeriod, + // to keep the applier current with its own pushes. Switch off the applier's own triggers (short + // intervals, Git-host webhooks into it), or they go on applying without that check. The + // reference is looked up in the source cluster, with the identity the ClusterProvider uses. + // +optional + Applier *GitTargetApplier `json:"applier,omitempty"` // OnRefusal decides what happens when a live edit is refused because it has no legal - // destination in the Git folder. A refusal produces no commit, so nothing normally reverts - // the edit: with Argo CD selfHeal off it stays OutOfSync until a human acts, and under Flux - // it survives until the next apply interval. + // destination in the Git folder. A refusal produces no commit, so nothing reverts the edit + // until something applies Git over it. // // "Ignore", the default, records the refusal in conditions and events and does nothing else. // - // "PushEmptyCommit" pushes a commit that changes no file, which moves the branch. Flux then - // sees a new artifact revision and re-applies; Argo CD sees a revision it has not synced, so - // its selfHeal=false skip does not apply and automated sync reverts the edit. - // - // It fires only when the refused write was an edit to a document the folder ALREADY holds and - // is not removing. An object Git does not manage is left alone deliberately: re-applying would - // do nothing to it (Flux prunes from its inventory and Argo CD from the resources it tracks, - // and a live-created object is in neither), or, if it was managed and has since been removed - // from Git, would prune it. That second one is the reconciler's decision to take on its own - // schedule, not this operator's to hurry. Eligibility is per document, not per file: a write - // that removes a document from a file other documents keep alive is still a removal, and a - // flush that removes one anywhere is excluded whole. + // "Reapply" asks this target's applier (spec.applier, which it requires) to apply the current + // tip once the branch holds no open window and no pending writes, which reverts the refused + // edit and nothing that is still waiting to be published. No other folder on the branch is + // woken. // - // A standing refusal earns one commit, not one per recheck. The target re-reads its folder and - // refuses again for as long as nobody corrects the edit, and an observation of the same objects - // with the same issues is already covered by the commit that was made for it. A different - // refused edit is a new commit, as is the same edit re-made after the target accepted a write - // in between. - // - // It also fires only for a refusal where the folder itself is accepted and this one write had - // nowhere to land. A folder-level refusal (unparseable YAML, a foreign file) is left alone: an - // empty commit cannot repair a folder, and the reconciler may be mid-way through its own - // corrections there. - // - // Two consequences remain to weigh. The commit wakes EVERYTHING watching the branch, not just - // the refused object, so it can hurry along unrelated work including another target's pending - // prune. And it can revert allowed live edits that are still waiting in the commit window. An - // Argo CD Application carrying argocd.argoproj.io/manifest-generate-paths ignores the commit - // entirely, because no file under its refresh paths changed. - // +optional - // +kubebuilder:validation:Enum=Ignore;PushEmptyCommit + // It fires only for an edit to a document the folder ALREADY holds, in a write that removes no + // document anywhere: re-applying would otherwise prune, or put back, an object somebody else is + // deleting. A folder-level refusal (unparseable YAML, a foreign file) never fires it, because + // re-applying Git cannot repair a folder. A standing refusal is requested once, not once per + // recheck; a different refused edit, or the same edit re-made after a write of that type was + // accepted, is a new request. + // +optional + // +kubebuilder:validation:Enum=Ignore;Reapply // +kubebuilder:default=Ignore OnRefusal RefusalAction `json:"onRefusal,omitempty"` @@ -450,6 +440,11 @@ type GitTargetStatus struct { // one this target points at. // +optional Remote *GitTargetRemoteStatus `json:"remote,omitempty"` + + // Applier is the last apply request made to spec.applier: the commit, when, and why. Absent + // until the first request. + // +optional + Applier *GitTargetApplierStatus `json:"applier,omitempty"` } // GitTargetRemoteStatus is the answer to "where is my branch, and when did we last prove it". @@ -636,6 +631,7 @@ type GitTargetRetentionStatus struct { // +kubebuilder:printcolumn:name="LayoutResolved",type=string,JSONPath=`.status.conditions[?(@.type=="LayoutResolved")].reason`,priority=1 // +kubebuilder:printcolumn:name="GitPathAccepted",type=string,JSONPath=`.status.conditions[?(@.type=="GitPathAccepted")].status`,priority=1 // +kubebuilder:printcolumn:name="RenderMatchesLive",type=string,JSONPath=`.status.conditions[?(@.type=="RenderMatchesLive")].status`,priority=1 +// +kubebuilder:printcolumn:name="ApplierConfigured",type=string,JSONPath=`.status.conditions[?(@.type=="ApplierConfigured")].status`,priority=1 // +kubebuilder:printcolumn:name="StreamsRunning",type=string,JSONPath=`.status.conditions[?(@.type=="StreamsRunning")].status`,priority=1 // +kubebuilder:printcolumn:name="SourceReachable",type=string,JSONPath=`.status.conditions[?(@.type=="SourceClusterReachable")].reason`,priority=1 // +kubebuilder:printcolumn:name="Encryption",type=string,JSONPath=`.spec.encryption.provider`,priority=1 @@ -704,17 +700,82 @@ type RefusalAction string const ( // RefusalActionIgnore records the refusal and does nothing else. It is the default. RefusalActionIgnore RefusalAction = "Ignore" - // RefusalActionPushEmptyCommit pushes a commit that changes no file, so the branch moves and - // the reconciler re-applies the desired state, reverting the refused edit. - RefusalActionPushEmptyCommit RefusalAction = "PushEmptyCommit" + // RefusalActionReapply asks the target's applier to apply the current tip, reverting the + // refused edit. + RefusalActionReapply RefusalAction = "Reapply" ) +// GitTargetApplier names the object that applies a GitTarget's folder to its source cluster. +type GitTargetApplier struct { + // KustomizationRef is a Flux Kustomization whose spec.sourceRef is a GitRepository on this + // target's repository and branch. Its namespace defaults to the GitTarget's namespace. GitOps + // Reverser asks it to apply the way `flux reconcile kustomization --with-source` does: it + // stamps reconcile.fluxcd.io/requestedAt on the GitRepository, and on the Kustomization too + // when the GitRepository's artifact already holds the commit. + // +required + KustomizationRef meta.NamespacedObjectReference `json:"kustomizationRef"` + + // QuietPeriod is how long the branch must hold no open commit window and no pending writes, + // after GitOps Reverser's own pushes, before the applier is asked to catch up with them. Flux + // applies on its own timers and on a controller restart from whatever artifact it holds, so + // an artifact left behind those pushes would revert edits that are already in Git. Every new + // write restarts the wait. A Go duration of at most "1h". Defaults to "30s". + // +optional + // +kubebuilder:validation:Type=string + // +kubebuilder:validation:Pattern="^([0-9]+(\\.[0-9]+)?(ns|us|µs|μs|ms|s|m|h))+$" + // +kubebuilder:default="30s" + // +kubebuilder:validation:XValidation:rule="duration(self) <= duration('1h')",message="spec.applier.quietPeriod must be a Go duration of at most 1h" + QuietPeriod *metav1.Duration `json:"quietPeriod,omitempty"` +} + +// DefaultApplierQuietPeriod is spec.applier.quietPeriod when it is omitted. +const DefaultApplierQuietPeriod = 30 * time.Second + +// EffectiveQuietPeriod is spec.applier.quietPeriod, resolving an omitted field to the default. +func (a *GitTargetApplier) EffectiveQuietPeriod() time.Duration { + if a == nil || a.QuietPeriod == nil { + return DefaultApplierQuietPeriod + } + return a.QuietPeriod.Duration +} + +// ApplyReason is why GitOps Reverser asked an applier to apply. +// +kubebuilder:validation:Enum=ForeignCommit;Refusal;CatchUp +type ApplyReason string + +const ( + // ApplyReasonForeignCommit is a commit GitOps Reverser did not push. + ApplyReasonForeignCommit ApplyReason = "ForeignCommit" + // ApplyReasonRefusal is a refused edit on a target with spec.onRefusal: Reapply. + ApplyReasonRefusal ApplyReason = "Refusal" + // ApplyReasonCatchUp keeps the applier current with GitOps Reverser's own pushes. + ApplyReasonCatchUp ApplyReason = "CatchUp" +) + +// GitTargetApplierStatus is the last apply request GitOps Reverser made for this target. +type GitTargetApplierStatus struct { + // RequestedCommit is the commit the applier was asked to apply. + // +optional + RequestedCommit string `json:"requestedCommit,omitempty"` + + // RequestedAt is when the request was made. It says the applier was asked, never that it + // applied: read the applier's own status for that. + // +optional + RequestedAt *metav1.Time `json:"requestedAt,omitempty"` + + // Reason is why it was asked. + // +optional + Reason ApplyReason `json:"reason,omitempty"` +} + // EffectiveOnRefusal returns the refusal action, resolving an omitted field to the default. The // CRD default makes this redundant for objects that went through the API server, and not for a -// literal built in a test or by an older client. +// literal built in a test or by an older client. A value this release does not know, such as the +// retired "PushEmptyCommit" still stored on an object, is Ignore: an action nobody can name now is +// not consent to one. func (g *GitTarget) EffectiveOnRefusal() RefusalAction { - if g.Spec.OnRefusal == "" { - return RefusalActionIgnore + if g.Spec.OnRefusal == RefusalActionReapply { + return RefusalActionReapply } - return g.Spec.OnRefusal + return RefusalActionIgnore } diff --git a/api/v1alpha3/zz_generated.deepcopy.go b/api/v1alpha3/zz_generated.deepcopy.go index 890184c4..1220014b 100644 --- a/api/v1alpha3/zz_generated.deepcopy.go +++ b/api/v1alpha3/zz_generated.deepcopy.go @@ -681,6 +681,22 @@ func (in *GitProviderList) DeepCopyObject() runtime.Object { return nil } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GitProviderReceiver) DeepCopyInto(out *GitProviderReceiver) { + *out = *in + out.SecretRef = in.SecretRef +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GitProviderReceiver. +func (in *GitProviderReceiver) DeepCopy() *GitProviderReceiver { + if in == nil { + return nil + } + out := new(GitProviderReceiver) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *GitProviderSpec) DeepCopyInto(out *GitProviderSpec) { *out = *in @@ -704,6 +720,11 @@ func (in *GitProviderSpec) DeepCopyInto(out *GitProviderSpec) { *out = new(CommitSpec) (*in).DeepCopyInto(*out) } + if in.Receiver != nil { + in, out := &in.Receiver, &out.Receiver + *out = new(GitProviderReceiver) + **out = **in + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GitProviderSpec. @@ -774,6 +795,46 @@ func (in *GitTarget) DeepCopyObject() runtime.Object { return nil } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GitTargetApplier) DeepCopyInto(out *GitTargetApplier) { + *out = *in + out.KustomizationRef = in.KustomizationRef + if in.QuietPeriod != nil { + in, out := &in.QuietPeriod, &out.QuietPeriod + *out = new(v1.Duration) + **out = **in + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GitTargetApplier. +func (in *GitTargetApplier) DeepCopy() *GitTargetApplier { + if in == nil { + return nil + } + out := new(GitTargetApplier) + in.DeepCopyInto(out) + return out +} + +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *GitTargetApplierStatus) DeepCopyInto(out *GitTargetApplierStatus) { + *out = *in + if in.RequestedAt != nil { + in, out := &in.RequestedAt, &out.RequestedAt + *out = (*in).DeepCopy() + } +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GitTargetApplierStatus. +func (in *GitTargetApplierStatus) DeepCopy() *GitTargetApplierStatus { + if in == nil { + return nil + } + out := new(GitTargetApplierStatus) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *GitTargetCommitSpec) DeepCopyInto(out *GitTargetCommitSpec) { *out = *in @@ -944,6 +1005,11 @@ func (in *GitTargetSpec) DeepCopyInto(out *GitTargetSpec) { *out = new(EncryptionSpec) (*in).DeepCopyInto(*out) } + if in.Applier != nil { + in, out := &in.Applier, &out.Applier + *out = new(GitTargetApplier) + (*in).DeepCopyInto(*out) + } if in.Placement != nil { in, out := &in.Placement, &out.Placement *out = new(GitTargetPlacementSpec) @@ -1011,6 +1077,11 @@ func (in *GitTargetStatus) DeepCopyInto(out *GitTargetStatus) { *out = new(GitTargetRemoteStatus) (*in).DeepCopyInto(*out) } + if in.Applier != nil { + in, out := &in.Applier, &out.Applier + *out = new(GitTargetApplierStatus) + (*in).DeepCopyInto(*out) + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GitTargetStatus. diff --git a/charts/gitops-reverser/README.md b/charts/gitops-reverser/README.md index e61bf2f0..14f4e57e 100644 --- a/charts/gitops-reverser/README.md +++ b/charts/gitops-reverser/README.md @@ -180,6 +180,8 @@ nodeSelector: | `controllerManager.branchBufferMaxSize` | Maximum in-memory event buffer per branch worker. Bounds the open commit window plus the writes retained for replay until a push succeeds. While the remote cannot be reached, retained writes that reach it close the worker to new writes, saves and resyncs until they are published; the watch delivers a refused event again | `8Mi` | | `controllerManager.commitRequestTTL` | How long a `CommitRequest` is kept once it finishes. Written onto the request as the `configbutler.ai/delete-after` annotation; remove or edit that to keep one. `"0"` writes none | `48h` | | `git.refreshInterval` | How often an idle `GitTarget` re-proves where its branch is on the Git remote. `"0"` disables it | `10m` | +| `receiver.bindAddress` | `host:port` the push receiver listens on, over plain HTTP (`--receiver-bind-address`), for Git hosts' push webhooks to `GitProvider.status.webhookPath`. Empty or `"0"` serves no receiver. When set, the chart adds a `-receiver` Service; expose it with an Ingress or Gateway route of your own that terminates TLS. See [push receiver settings](../../docs/configuration.md#push-receiver-settings) | `""` | +| `applierAccess.flux` | Grant the operator `get`, `list`, `watch`, `patch` on Flux `Kustomizations` and `GitRepositories`, and `create` on `SelfSubjectAccessReviews`, so it can ask Flux to apply a commit somebody else pushed | `false` | | `auditService.type` | Service type for the dedicated audit Service | `NodePort` | | `auditService.nodePort` | Fixed NodePort for the audit Service when `auditService.type=NodePort` | `30444` | | `auditService.clusterIP` | Optional fixed ClusterIP for the dedicated audit Service | `""` | @@ -220,6 +222,7 @@ nodeSelector: | `servers.metrics.tls.secretNameOverride` | Override Secret name for metrics TLS cert/key | `-metrics-server-cert` | | `service.clusterIP` | Optional fixed ClusterIP for single controller Service | `""` | | `service.ports.audit` | Service port for audit ingress when `attribution.enabled=true` | `9444` | +| `service.ports.receiver` | Port of the `-receiver` Service when `receiver.bindAddress` is set | `80` | | `service.ports.metrics` | Service port for metrics | `8080` | | `servers.healthProbe.bindAddress` | Liveness/readiness probe bind address (`--health-probe-bind-address`) | `:8081` | | `servers.admission.port` | Admission webhook container/Service port | `9443` | diff --git a/charts/gitops-reverser/templates/_helpers.tpl b/charts/gitops-reverser/templates/_helpers.tpl index 6bebf60d..71576c9b 100644 --- a/charts/gitops-reverser/templates/_helpers.tpl +++ b/charts/gitops-reverser/templates/_helpers.tpl @@ -107,3 +107,27 @@ as the Certificate name and in the webhook's cert-manager.io/inject-ca-from anno {{- define "gitops-reverser.admissionServerCertName" -}} {{- printf "%s-admission-server-cert" (include "gitops-reverser.fullname" .) -}} {{- end }} + +{{/* +Non-empty when the push receiver listener is on: receiver.bindAddress is set and is not the "0" +the binary reads as off. Used as `if include ...`, so it renders "true" or nothing. +*/}} +{{- define "gitops-reverser.receiverEnabled" -}} +{{- $addr := .Values.receiver.bindAddress | default "" | toString | trim -}} +{{- if and $addr (ne $addr "0") -}}true{{- end -}} +{{- end }} + +{{/* +The port in receiver.bindAddress ("host:port"; an empty host or an IPv6 "[::]" host included), so +the containerPort and the Service's targetPort cannot drift from what the binary binds. +*/}} +{{- define "gitops-reverser.receiverPort" -}} +{{- splitList ":" (.Values.receiver.bindAddress | toString | trim) | last | int -}} +{{- end }} + +{{/* +Name of the push receiver Service. +*/}} +{{- define "gitops-reverser.receiverServiceName" -}} +{{- printf "%s-receiver" (include "gitops-reverser.fullname" .) | trunc 63 | trimSuffix "-" }} +{{- end }} diff --git a/charts/gitops-reverser/templates/applier-rbac.yaml b/charts/gitops-reverser/templates/applier-rbac.yaml new file mode 100644 index 00000000..9a0f1fe6 --- /dev/null +++ b/charts/gitops-reverser/templates/applier-rbac.yaml @@ -0,0 +1,54 @@ +{{- if and .Values.rbac.create .Values.applierAccess.flux }} +# What GitOps Reverser needs to ask Flux to apply a commit after somebody else pushed to a branch a +# GitTarget writes: read and patch the Kustomization and its GitRepository (an apply request is a +# reconcile request on them), and create SelfSubjectAccessReviews so the operator can check it holds +# that access before relying on it. Separate from the manager role because it is optional and +# names another project's API: an install without Flux grants nothing on it. See +# docs/design/push-notification-and-reconcile-trigger.md §4.4. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "gitops-reverser.fullname" . }}-applier-flux + labels: + {{- include "gitops-reverser.labels" . | nindent 4 }} +rules: +- apiGroups: + - kustomize.toolkit.fluxcd.io + resources: + - kustomizations + verbs: + - get + - list + - watch + - patch +- apiGroups: + - source.toolkit.fluxcd.io + resources: + - gitrepositories + verbs: + - get + - list + - watch + - patch +- apiGroups: + - authorization.k8s.io + resources: + - selfsubjectaccessreviews + verbs: + - create +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "gitops-reverser.fullname" . }}-applier-flux + labels: + {{- include "gitops-reverser.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "gitops-reverser.fullname" . }}-applier-flux +subjects: +- kind: ServiceAccount + name: {{ include "gitops-reverser.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/charts/gitops-reverser/templates/deployment.yaml b/charts/gitops-reverser/templates/deployment.yaml index f8258d0f..cf3fd7b0 100644 --- a/charts/gitops-reverser/templates/deployment.yaml +++ b/charts/gitops-reverser/templates/deployment.yaml @@ -109,6 +109,9 @@ spec: - --branch-buffer-max-size={{ .Values.controllerManager.branchBufferMaxSize }} - --git-refresh-interval={{ .Values.git.refreshInterval }} - --commit-request-ttl={{ .Values.controllerManager.commitRequestTTL }} + {{- if include "gitops-reverser.receiverEnabled" . }} + - {{ printf "--receiver-bind-address=%s" (.Values.receiver.bindAddress | toString | trim) | quote }} + {{- end }} {{- if .Values.logging.level }} - --zap-log-level={{ .Values.logging.level }} {{- end }} @@ -135,6 +138,11 @@ spec: containerPort: {{ .Values.servers.admission.port }} protocol: TCP {{- end }} + {{- if include "gitops-reverser.receiverEnabled" . }} + - name: receiver + containerPort: {{ include "gitops-reverser.receiverPort" . }} + protocol: TCP + {{- end }} env: - name: POD_NAME valueFrom: diff --git a/charts/gitops-reverser/templates/services.yaml b/charts/gitops-reverser/templates/services.yaml index 3d0ddff2..16a23e1b 100644 --- a/charts/gitops-reverser/templates/services.yaml +++ b/charts/gitops-reverser/templates/services.yaml @@ -55,3 +55,27 @@ spec: {{- include "gitops-reverser.selectorLabels" . | nindent 4 }} control-plane: gitops-reverser {{- end }} +{{- if include "gitops-reverser.receiverEnabled" . }} +--- +# The push receiver: plain HTTP for Git hosts' push webhooks. ClusterIP only; expose it with an +# Ingress or Gateway route of your own that terminates TLS, the way Flux's webhook-receiver is. +# Only the elected replica serves it (a delivery drives a branch worker). +apiVersion: v1 +kind: Service +metadata: + name: {{ include "gitops-reverser.receiverServiceName" . }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "gitops-reverser.labels" . | nindent 4 }} + app.kubernetes.io/component: receiver +spec: + type: ClusterIP + ports: + - name: http + port: {{ .Values.service.ports.receiver }} + targetPort: receiver + protocol: TCP + selector: + {{- include "gitops-reverser.selectorLabels" . | nindent 4 }} + control-plane: gitops-reverser +{{- end }} diff --git a/charts/gitops-reverser/values.schema.json b/charts/gitops-reverser/values.schema.json index 1ee6a758..a59a353b 100644 --- a/charts/gitops-reverser/values.schema.json +++ b/charts/gitops-reverser/values.schema.json @@ -131,6 +131,32 @@ } }, + "receiver": { + "type": "object", + "additionalProperties": false, + "properties": { + "bindAddress": { + "anyOf": [ + { "type": "string", "pattern": "^(|0|[^\\s]*:[1-9][0-9]{0,4})$" }, + { "const": 0 } + ], + "description": "host:port the push receiver listens on, over plain HTTP (--receiver-bind-address). Empty or \"0\" serves no receiver. When set, the chart also renders a -receiver Service; expose it with an Ingress or Gateway route that terminates TLS." + } + } + }, + + "applierAccess": { + "type": "object", + "additionalProperties": false, + "description": "Optional RBAC that lets GitOps Reverser ask an applier in this cluster to apply a commit. Off unless asked for, since most installs run one applier or none.", + "properties": { + "flux": { + "type": "boolean", + "description": "Grant get/list/watch/patch on Flux Kustomizations and GitRepositories, plus create on SelfSubjectAccessReviews." + } + } + }, + "certManager": { "type": "object", "additionalProperties": false, @@ -485,7 +511,8 @@ "additionalProperties": false, "properties": { "audit": { "$ref": "#/$defs/port" }, - "metrics": { "$ref": "#/$defs/port" } + "metrics": { "$ref": "#/$defs/port" }, + "receiver": { "$ref": "#/$defs/port" } } } } diff --git a/charts/gitops-reverser/values.yaml b/charts/gitops-reverser/values.yaml index 78526129..f546f1e8 100644 --- a/charts/gitops-reverser/values.yaml +++ b/charts/gitops-reverser/values.yaml @@ -112,6 +112,29 @@ git: # "every tick". gitopsreverser_git_fetches_total{reason="refresh"} is what it costs. refreshInterval: 10m +# The push receiver: the endpoint a Git host's push webhook points at, so GitOps Reverser learns in +# seconds, instead of at the next refresh, that somebody else pushed. A GitProvider opts in with +# spec.receiver and publishes its path as status.webhookPath (/git-push//). +receiver: + # host:port the receiver listens on (passed as --receiver-bind-address), e.g. ":9292". Empty (the + # default) serves no receiver: a foreign push is then still found, by the next rejected push or by + # git.refreshInterval. When set, the chart adds a containerPort and a -receiver Service + # (port service.ports.receiver). The listener is plain HTTP, because a Git host cannot present the + # client certificate the audit listener requires: expose the Service to your Git host with an + # Ingress or Gateway route of your own that terminates TLS, the way Flux's webhook-receiver is + # exposed. Every request must still carry a valid signature for the route it names. + bindAddress: "" + +# Optional RBAC that lets GitOps Reverser ask an applier in this cluster to apply a commit, after +# somebody else pushed to a branch a GitTarget writes. Off unless asked for, since most installs run +# one applier or none. +applierAccess: + # Grant get/list/watch/patch on kustomize.toolkit.fluxcd.io Kustomizations and + # source.toolkit.fluxcd.io GitRepositories, plus create on SelfSubjectAccessReviews (to check that + # access before relying on it). Cluster-wide, because a GitTarget's applier can live in any + # namespace. + flux: false + # cert-manager issuer shared by every certificate the chart mints. One self-signed CA # backs them all, so the issuer is genuinely cross-cutting and lives here; each server # opts into a cert via servers..tls.certManager. (Deliberately NOT duplicated per @@ -406,6 +429,9 @@ service: ports: audit: 9444 metrics: 8080 + # Port of the -receiver Service, rendered only when receiver.bindAddress is set. It + # forwards to the port in receiver.bindAddress. + receiver: 80 auditService: type: NodePort diff --git a/cmd/main.go b/cmd/main.go index 94de00ea..d4863e15 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -38,6 +38,7 @@ import ( ctrladmission "sigs.k8s.io/controller-runtime/pkg/webhook/admission" configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/applier" "github.com/ConfigButler/gitops-reverser/internal/attribution" "github.com/ConfigButler/gitops-reverser/internal/controller" "github.com/ConfigButler/gitops-reverser/internal/git" @@ -132,6 +133,7 @@ func main() { workerManager.SetSSHHostKeyConfig(cfg.sshHostKeys) workerManager.SetCredentialTransportPolicy(cfg.credentialPolicy) fatalIfErr(mgr.Add(workerManager), "unable to add worker manager to manager") + setupPushReceiver(mgr, cfg, workerManager) // Watch ingestion manager (placeholder, will get EventRouter set later) watchMgr := &watch.Manager{ @@ -179,6 +181,13 @@ func main() { // The read-only twin of the refusal reporter: the refresher re-reads an idle target's folder, // and a folder somebody else broke in Git produces no Kubernetes event to notice it by. workerManager.SetScanAcceptanceReporter(watchMgr.ReportGitPathScan) + // The branch workers decide when a GitTarget's applier may apply; the dispatcher delivers it, + // off their queues, to the Flux objects in each target's source cluster. + applyDispatcher := applier.NewDispatcher( + mgr.GetClient(), watchMgr.SourceDynamicClient, mgr.GetEventRecorderFor("gittarget"), + ctrl.Log.WithName("applier")) + fatalIfErr(mgr.Add(applyDispatcher), "unable to add apply request dispatcher to manager") + workerManager.SetApplyRequester(applyDispatcher) // WatchRule controller (with WatchManager reference for dynamic reconciliation) fatalIfErr((&controller.WatchRuleReconciler{ @@ -345,6 +354,8 @@ func main() { Recorder: mgr.GetEventRecorderFor("gittarget"), GitRefreshInterval: cfg.gitRefreshInterval, + ApplierClients: watchMgr.SourceDynamicClient, + ApplyRequests: applyDispatcher, }).SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "GitTarget") os.Exit(1) @@ -447,6 +458,7 @@ type appConfig struct { attributionCollectionWindow time.Duration attributionCollectionUIDCap int auditRouteAnnotationKey string + receiverBindAddress string // replicaCount is how many replicas of this Deployment run. The process cannot observe that // itself, so the chart templates it in from .Values.replicaCount. It exists for exactly one // check: the in-memory fact transport only works when the audit receiver and the resolver are @@ -661,6 +673,7 @@ func parseFlagsWithArgs(fs *flag.FlagSet, args []string) (appConfig, error) { "known_hosts at all. A present-but-unparseable known_hosts is always a hard error.") fs.BoolVar(&cfg.credentialPolicy.AllowInsecureGitHTTP, "insecure-allow-git-http", false, "INSECURE, dev/throwaway clusters only: permit credentials with http:// GitProvider URLs.") + bindReceiverFlags(fs, &cfg) cfg.zapOpts = zap.Options{ // Production mode defaults to JSON encoding, which is easier for log processors to parse. Development: false, @@ -690,6 +703,9 @@ func parseFlagsWithArgs(fs *flag.FlagSet, args []string) (appConfig, error) { if err := validateAdmissionWebhookConfig(cfg); err != nil { return appConfig{}, err } + if err := validateReceiverConfig(cfg); err != nil { + return appConfig{}, err + } // Rejected rather than read as "keep everything": a negative age has no meaning, and 0 already // says that. if cfg.commitRequestTTL < 0 { diff --git a/cmd/main_chart_receiver_test.go b/cmd/main_chart_receiver_test.go new file mode 100644 index 00000000..15e45e5c --- /dev/null +++ b/cmd/main_chart_receiver_test.go @@ -0,0 +1,198 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "flag" + "os" + "os/exec" + "strings" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + appsv1 "k8s.io/api/apps/v1" + corev1 "k8s.io/api/core/v1" + rbacv1 "k8s.io/api/rbac/v1" + "k8s.io/apimachinery/pkg/util/intstr" + "sigs.k8s.io/yaml" +) + +// receiverChartInputs are read in-process so `go test` treats them as cache inputs; helm reads +// the chart in a subprocess the test log cannot see. +var receiverChartInputs = []string{ + "../charts/gitops-reverser/values.yaml", + "../charts/gitops-reverser/values.schema.json", + "../charts/gitops-reverser/templates/_helpers.tpl", + "../charts/gitops-reverser/templates/deployment.yaml", + "../charts/gitops-reverser/templates/services.yaml", + "../charts/gitops-reverser/templates/applier-rbac.yaml", +} + +// helmTemplateDocs renders one chart template and splits it into its YAML documents. A template +// that renders nothing is an empty list, not an error: helm refuses --show-only for a template with +// no output, so that refusal is read as "nothing rendered". +func helmTemplateDocs(t *testing.T, template string, setValues ...string) []string { + t.Helper() + for _, path := range receiverChartInputs { + _, err := os.ReadFile(path) + require.NoError(t, err) + } + + args := []string{"template", "gitops-reverser", "../charts/gitops-reverser", "--show-only", template} + for _, sv := range setValues { + args = append(args, "--set", sv) + } + out, err := exec.Command("helm", args...).CombinedOutput() + if err != nil && strings.Contains(string(out), "could not find template") { + return nil + } + require.NoErrorf(t, err, "helm template failed: %s", out) + + var docs []string + for _, doc := range strings.Split(string(out), "\n---") { + if strings.Contains(doc, "kind:") { + docs = append(docs, doc) + } + } + return docs +} + +func renderManager(t *testing.T, setValues ...string) corev1.Container { + t.Helper() + docs := helmTemplateDocs(t, "templates/deployment.yaml", setValues...) + require.Len(t, docs, 1) + var deploy appsv1.Deployment + require.NoError(t, yaml.Unmarshal([]byte(docs[0]), &deploy)) + for _, c := range deploy.Spec.Template.Spec.Containers { + if c.Name == "manager" { + return c + } + } + t.Fatal("no manager container in rendered Deployment") + return corev1.Container{} +} + +func renderReceiverService(t *testing.T, setValues ...string) (corev1.Service, bool) { + t.Helper() + for _, doc := range helmTemplateDocs(t, "templates/services.yaml", setValues...) { + var svc corev1.Service + require.NoError(t, yaml.Unmarshal([]byte(doc), &svc)) + if strings.HasSuffix(svc.Name, "-receiver") { + return svc, true + } + } + return corev1.Service{}, false +} + +func containerPort(c corev1.Container, name string) (int32, bool) { + for _, p := range c.Ports { + if p.Name == name { + return p.ContainerPort, true + } + } + return 0, false +} + +// The receiver is off by default and the chart must say nothing about it: no flag (the binary's +// own default is off), no port, no Service. +func TestChartReceiver_OffByDefault(t *testing.T) { + for name, setValues := range map[string][]string{ + "default": nil, + "explicit zero": {"receiver.bindAddress=0"}, + "explicit empty": {"receiver.bindAddress="}, + } { + t.Run(name, func(t *testing.T) { + manager := renderManager(t, setValues...) + cfg, err := parseFlagsWithArgs(flag.NewFlagSet("manager", flag.ContinueOnError), manager.Args) + require.NoError(t, err) + assert.False(t, receiverEnabled(cfg)) + for _, arg := range manager.Args { + assert.NotContains(t, arg, "--receiver-bind-address") + } + _, hasPort := containerPort(manager, "receiver") + assert.False(t, hasPort) + _, hasService := renderReceiverService(t, setValues...) + assert.False(t, hasService) + }) + } +} + +// Set, the address reaches the binary unquoted, and the containerPort and the Service's target +// are the port the binary binds. +func TestChartReceiver_BindAddressReachesTheBinaryAndTheService(t *testing.T) { + for name, tc := range map[string]struct { + bindAddress string + wantPort int32 + }{ + "port only": {":9292", 9292}, + "host and port": {"0.0.0.0:9393", 9393}, + } { + t.Run(name, func(t *testing.T) { + setValues := []string{"receiver.bindAddress=" + tc.bindAddress} + manager := renderManager(t, setValues...) + + cfg, err := parseFlagsWithArgs(flag.NewFlagSet("manager", flag.ContinueOnError), manager.Args) + require.NoError(t, err, "the binary must accept the argv the chart renders") + assert.Equal(t, tc.bindAddress, cfg.receiverBindAddress, "a quoted render would bind nothing") + + port, ok := containerPort(manager, "receiver") + require.True(t, ok, "the receiver needs a named containerPort for the Service to target") + assert.Equal(t, tc.wantPort, port) + + svc, ok := renderReceiverService(t, setValues...) + require.True(t, ok) + assert.Equal(t, "gitops-reverser-receiver", svc.Name) + assert.Equal(t, corev1.ServiceTypeClusterIP, svc.Spec.Type) + require.Len(t, svc.Spec.Ports, 1) + assert.Equal(t, int32(80), svc.Spec.Ports[0].Port) + assert.Equal(t, intstr.FromString("receiver"), svc.Spec.Ports[0].TargetPort) + }) + } +} + +func TestChartReceiver_MalformedBindAddressFailsTheRender(t *testing.T) { + out, err := exec.Command("helm", "template", "gitops-reverser", "../charts/gitops-reverser", + "--show-only", "templates/deployment.yaml", "--set", "receiver.bindAddress=localhost").CombinedOutput() + require.Error(t, err) + assert.Contains(t, string(out), "receiver") +} + +// applierAccess.flux is off by default, and on it grants exactly what an apply request to Flux +// needs, to the operator's own ServiceAccount. +func TestChartApplierAccess_Flux(t *testing.T) { + assert.Empty(t, helmTemplateDocs(t, "templates/applier-rbac.yaml"), "off unless asked for") + assert.Empty(t, helmTemplateDocs(t, "templates/applier-rbac.yaml", + "applierAccess.flux=true", "rbac.create=false"), "rbac.create=false renders no RBAC at all") + + docs := helmTemplateDocs(t, "templates/applier-rbac.yaml", "applierAccess.flux=true") + require.Len(t, docs, 2) + + var role rbacv1.ClusterRole + require.NoError(t, yaml.Unmarshal([]byte(docs[0]), &role)) + assert.Equal(t, "ClusterRole", role.Kind) + assert.Equal(t, []rbacv1.PolicyRule{ + { + APIGroups: []string{"kustomize.toolkit.fluxcd.io"}, + Resources: []string{"kustomizations"}, + Verbs: []string{"get", "list", "watch", "patch"}, + }, + { + APIGroups: []string{"source.toolkit.fluxcd.io"}, + Resources: []string{"gitrepositories"}, + Verbs: []string{"get", "list", "watch", "patch"}, + }, + { + APIGroups: []string{"authorization.k8s.io"}, + Resources: []string{"selfsubjectaccessreviews"}, + Verbs: []string{"create"}, + }, + }, role.Rules) + + var binding rbacv1.ClusterRoleBinding + require.NoError(t, yaml.Unmarshal([]byte(docs[1]), &binding)) + assert.Equal(t, role.Name, binding.RoleRef.Name) + require.Len(t, binding.Subjects, 1) + assert.Equal(t, "ServiceAccount", binding.Subjects[0].Kind) + assert.Equal(t, "gitops-reverser", binding.Subjects[0].Name) +} diff --git a/cmd/receiver.go b/cmd/receiver.go new file mode 100644 index 00000000..b2c2f689 --- /dev/null +++ b/cmd/receiver.go @@ -0,0 +1,156 @@ +// SPDX-License-Identifier: Apache-2.0 + +// The push receiver's listener: a plain HTTP server for Git hosts' push webhooks. It is its own +// listener because the audit listener requires a client certificate signed by the audit CA, which +// no Git host can present. See docs/configuration.md, "Push receiver settings". + +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "net" + "net/http" + "strings" + "sync/atomic" + "time" + + ctrl "sigs.k8s.io/controller-runtime" + + webhookhandler "github.com/ConfigButler/gitops-reverser/internal/webhook" +) + +const ( + // receiverOff is the --receiver-bind-address value that serves no receiver, the same sentinel + // --metrics-bind-address uses. + receiverOff = "0" + + // The receiver's server timeouts. A delivery is one small POST answered without waiting on the + // remote (the branch worker reads the remote later, on its own loop), so these only have to + // outlast a slow client sending a body up to the 25 MiB cap, and keep an idle or trickling + // connection from holding a socket. They are not flags: nothing a Git host does needs them + // tuned, and every knob is one more thing to get wrong. + receiverReadHeaderTimeout = 10 * time.Second + receiverReadTimeout = 30 * time.Second + receiverWriteTimeout = 30 * time.Second + receiverIdleTimeout = 60 * time.Second + receiverShutdownTimeout = 10 * time.Second +) + +// bindReceiverFlags registers the push receiver's flags. +func bindReceiverFlags(fs *flag.FlagSet, cfg *appConfig) { + fs.StringVar(&cfg.receiverBindAddress, "receiver-bind-address", receiverOff, + "Address (host:port) the push receiver listens on, over plain HTTP: a Git host's push webhook "+ + "points here plus a GitProvider's status.webhookPath (/git-push//), so a "+ + "foreign push is noticed in seconds instead of at the next refresh. Expose it through an "+ + "Ingress or Gateway that terminates TLS. Default 0 serves no receiver; foreign pushes are "+ + "then still found by the next rejected push or --git-refresh-interval. Served only on the "+ + "elected replica, because a delivery drives a branch worker.") +} + +// receiverEnabled reports whether a push receiver listener is configured. +func receiverEnabled(cfg appConfig) bool { + addr := strings.TrimSpace(cfg.receiverBindAddress) + return addr != "" && addr != receiverOff +} + +// validateReceiverConfig rejects a malformed --receiver-bind-address before the manager starts, +// rather than letting the listener fail to bind after leader election. +func validateReceiverConfig(cfg appConfig) error { + if !receiverEnabled(cfg) { + return nil + } + if _, _, err := splitBindAddress(cfg.receiverBindAddress); err != nil { + return fmt.Errorf("invalid receiver-bind-address %q (use host:port, or 0 to serve no receiver): %w", + cfg.receiverBindAddress, err) + } + return nil +} + +// setupPushReceiver adds the push receiver listener to the manager when --receiver-bind-address +// is set. notifier is the branch workers' owner (the WorkerManager). +func setupPushReceiver(mgr ctrl.Manager, cfg appConfig, notifier webhookhandler.PushNotifier) { + if !receiverEnabled(cfg) { + setupLog.Info("push receiver off (--receiver-bind-address=0): foreign pushes are found by the " + + "next rejected push or the periodic refresh") + return + } + // The manager client reads GitProviders from its cache and Secrets straight from the API + // server (Secrets are excluded from the cache), so a rotated token is used on the next delivery. + handler, err := webhookhandler.NewReceiverHandler(webhookhandler.ReceiverHandlerConfig{ + Reader: mgr.GetClient(), + Notifier: notifier, + }) + fatalIfErr(err, "unable to build push receiver handler") + fatalIfErr(mgr.Add(newReceiverServerRunnable(cfg.receiverBindAddress, handler)), + "unable to add push receiver server runnable") + setupLog.Info("push receiver enabled", "address", cfg.receiverBindAddress, + "path", webhookhandler.ReceiverPathPrefix+"/") +} + +// buildReceiverServeMux routes only the receiver prefix to the handler. Everything else is the +// mux's own 404, so the listener answers nothing but /git-push/. +func buildReceiverServeMux(handler http.Handler) *http.ServeMux { + mux := http.NewServeMux() + mux.Handle(webhookhandler.ReceiverPathPrefix, handler) + return mux +} + +// receiverServerRunnable serves the push receiver as a manager Runnable. +type receiverServerRunnable struct { + server *http.Server + + // boundAddr is the listener's address once it is bound, nil before and after. It is what a + // test dials when the configured port is 0. + boundAddr atomic.Pointer[string] +} + +func newReceiverServerRunnable(addr string, handler http.Handler) *receiverServerRunnable { + return &receiverServerRunnable{server: &http.Server{ + Addr: strings.TrimSpace(addr), + Handler: buildReceiverServeMux(handler), + ReadHeaderTimeout: receiverReadHeaderTimeout, + ReadTimeout: receiverReadTimeout, + WriteTimeout: receiverWriteTimeout, + IdleTimeout: receiverIdleTimeout, + }} +} + +// NeedLeaderElection keeps the receiver on the elected replica. A delivery drives a branch worker, +// and only the leader runs branch workers (WorkerManager needs leader election too): served on a +// standby, every delivery would find no worker and answer matched=0 for a branch that is written. +func (r *receiverServerRunnable) NeedLeaderElection() bool { + return true +} + +// Start serves until ctx is cancelled, then shuts down gracefully. +func (r *receiverServerRunnable) Start(ctx context.Context) error { + listener, err := (&net.ListenConfig{}).Listen(ctx, "tcp", r.server.Addr) + if err != nil { + return fmt.Errorf("push receiver failed to bind %q: %w", r.server.Addr, err) + } + bound := listener.Addr().String() + r.boundAddr.Store(&bound) + defer r.boundAddr.Store(nil) + setupLog.Info("Starting push receiver", "address", bound) + + shutdownDone := make(chan struct{}) + go func() { + defer close(shutdownDone) + <-ctx.Done() + shutdownCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), receiverShutdownTimeout) + defer cancel() + if err := r.server.Shutdown(shutdownCtx); err != nil { + setupLog.Error(err, "Failed to shut down the push receiver") + } + }() + + err = r.server.Serve(listener) + <-shutdownDone + if errors.Is(err, http.ErrServerClosed) { + return nil + } + return fmt.Errorf("push receiver failed: %w", err) +} diff --git a/cmd/receiver_test.go b/cmd/receiver_test.go new file mode 100644 index 00000000..afc56153 --- /dev/null +++ b/cmd/receiver_test.go @@ -0,0 +1,97 @@ +// SPDX-License-Identifier: Apache-2.0 + +package main + +import ( + "context" + "flag" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func TestParseFlags_ReceiverIsOffByDefault(t *testing.T) { + cfg, err := parseFlagsWithArgs(flag.NewFlagSet("receiver-default", flag.ContinueOnError), nil) + require.NoError(t, err) + assert.Equal(t, "0", cfg.receiverBindAddress) + assert.False(t, receiverEnabled(cfg)) +} + +func TestParseFlags_ReceiverBindAddress(t *testing.T) { + tests := map[string]struct { + value string + wantEnabled bool + wantErr string + }{ + "port only": {value: ":9292", wantEnabled: true}, + "host and port": {value: "0.0.0.0:9292", wantEnabled: true}, + "explicit off": {value: "0"}, + "empty is off": {value: ""}, + "missing port": {value: "localhost", wantErr: "invalid receiver-bind-address"}, + "non-numeric port": {value: ":http", wantErr: "invalid receiver-bind-address"}, + "zero port refused": {value: ":0", wantErr: "invalid receiver-bind-address"}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + cfg, err := parseFlagsWithArgs(flag.NewFlagSet("receiver", flag.ContinueOnError), + []string{"--receiver-bind-address=" + tc.value}) + if tc.wantErr != "" { + require.Error(t, err) + assert.Contains(t, err.Error(), tc.wantErr) + return + } + require.NoError(t, err) + assert.Equal(t, tc.wantEnabled, receiverEnabled(cfg)) + }) + } +} + +// TestReceiverServerRunnable_ServesOnlyThePrefixAndShutsDown runs the listener for real: only +// /git-push/ reaches the handler, and cancelling the context is a clean stop. +func TestReceiverServerRunnable_ServesOnlyThePrefixAndShutsDown(t *testing.T) { + var gotPath string + handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + gotPath = r.URL.Path + w.WriteHeader(http.StatusAccepted) + }) + r := newReceiverServerRunnable("127.0.0.1:0", handler) + assert.True(t, r.NeedLeaderElection(), "a delivery drives a branch worker, which only the leader runs") + assert.Equal(t, receiverReadHeaderTimeout, r.server.ReadHeaderTimeout) + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- r.Start(ctx) }() + + require.Eventually(t, func() bool { return r.boundAddr.Load() != nil }, 2*time.Second, 5*time.Millisecond) + base := "http://" + *r.boundAddr.Load() + + resp, err := http.Post(base+"/git-push/team-a/repo", "application/json", strings.NewReader("{}")) + require.NoError(t, err) + _ = resp.Body.Close() + assert.Equal(t, http.StatusAccepted, resp.StatusCode) + assert.Equal(t, "/git-push/team-a/repo", gotPath) + + resp, err = http.Post(base+"/audit-webhook/x", "application/json", strings.NewReader("{}")) + require.NoError(t, err) + _ = resp.Body.Close() + assert.Equal(t, http.StatusNotFound, resp.StatusCode, "the receiver listener serves nothing but /git-push/") + + cancel() + require.NoError(t, <-done, "graceful shutdown returns nil") + assert.Nil(t, r.boundAddr.Load()) +} + +func TestReceiverServerRunnable_BindFailureIsAnError(t *testing.T) { + occupied := httptest.NewServer(http.NotFoundHandler()) + defer occupied.Close() + + r := newReceiverServerRunnable(strings.TrimPrefix(occupied.URL, "http://"), http.NotFoundHandler()) + err := r.Start(context.Background()) + require.Error(t, err) + assert.Contains(t, err.Error(), "push receiver failed to bind") +} diff --git a/config/crd/bases/configbutler.ai_gitproviders.yaml b/config/crd/bases/configbutler.ai_gitproviders.yaml index d946809b..4adbdd93 100644 --- a/config/crd/bases/configbutler.ai_gitproviders.yaml +++ b/config/crd/bases/configbutler.ai_gitproviders.yaml @@ -138,6 +138,41 @@ spec: required: - name type: object + receiver: + description: |- + Receiver lets the Git host tell GitOps Reverser that somebody pushed. Point the host's push + webhook at the receiver listener plus status.webhookPath, with the Secret's token as the + webhook secret. A delivery is only a hint: it makes GitOps Reverser read where the branch is + on the remote, and what the remote says decides whether a GitTarget's applier is asked to + apply. Without a receiver, a foreign push is still found, by the next rejected push or by + the periodic refresh. + properties: + secretRef: + description: SecretRef names a Secret in this namespace whose + "token" key is the webhook secret. + properties: + name: + description: Name of the referent. + type: string + required: + - name + type: object + type: + description: |- + Type is the payload and signature the Git host sends. "github" checks X-Hub-Signature-256, + "gitlab" checks X-Gitlab-Token, "gitea" checks X-Gitea-Signature, and "generic-hmac" checks + X-Signature: sha256= over a {"repository", "branch", "after"} body, + for a relay of your own. + enum: + - github + - gitlab + - gitea + - generic-hmac + type: string + required: + - secretRef + - type + type: object secretRef: description: |- SecretRef for authentication credentials (may be nil for public repos) @@ -296,6 +331,11 @@ spec: Register this as a signing key on your git platform. Only populated when commit.signing is configured and a signing key is available. type: string + webhookPath: + description: |- + WebhookPath is the path spec.receiver is served on, under the receiver listener. Empty + without spec.receiver. + type: string type: object required: - spec diff --git a/config/crd/bases/configbutler.ai_gittargets.yaml b/config/crd/bases/configbutler.ai_gittargets.yaml index d8558cc4..b933278a 100644 --- a/config/crd/bases/configbutler.ai_gittargets.yaml +++ b/config/crd/bases/configbutler.ai_gittargets.yaml @@ -52,6 +52,10 @@ spec: name: RenderMatchesLive priority: 1 type: string + - jsonPath: .status.conditions[?(@.type=="ApplierConfigured")].status + name: ApplierConfigured + priority: 1 + type: string - jsonPath: .status.conditions[?(@.type=="StreamsRunning")].status name: StreamsRunning priority: 1 @@ -108,6 +112,51 @@ spec: spec: description: spec defines the desired state of GitTarget properties: + applier: + description: |- + Applier names the object that applies this folder back to its source cluster. With it set, + GitOps Reverser decides when that object applies: it asks for an apply only when the branch + holds no open commit window and no pending writes, after a commit it did not push, after a + refused edit (spec.onRefusal: Reapply), and, once the branch has been quiet for quietPeriod, + to keep the applier current with its own pushes. Switch off the applier's own triggers (short + intervals, Git-host webhooks into it), or they go on applying without that check. The + reference is looked up in the source cluster, with the identity the ClusterProvider uses. + properties: + kustomizationRef: + description: |- + KustomizationRef is a Flux Kustomization whose spec.sourceRef is a GitRepository on this + target's repository and branch. Its namespace defaults to the GitTarget's namespace. GitOps + Reverser asks it to apply the way `flux reconcile kustomization --with-source` does: it + stamps reconcile.fluxcd.io/requestedAt on the GitRepository, and on the Kustomization too + when the GitRepository's artifact already holds the commit. + properties: + name: + description: Name of the referent. + type: string + namespace: + description: Namespace of the referent, when not specified + it acts as LocalObjectReference. + type: string + required: + - name + type: object + quietPeriod: + default: 30s + description: |- + QuietPeriod is how long the branch must hold no open commit window and no pending writes, + after GitOps Reverser's own pushes, before the applier is asked to catch up with them. Flux + applies on its own timers and on a controller restart from whatever artifact it holds, so + an artifact left behind those pushes would revert edits that are already in Git. Every new + write restarts the wait. A Go duration of at most "1h". Defaults to "30s". + pattern: ^([0-9]+(\.[0-9]+)?(ns|us|µs|μs|ms|s|m|h))+$ + type: string + x-kubernetes-validations: + - message: spec.applier.quietPeriod must be a Go duration of at + most 1h + rule: duration(self) <= duration('1h') + required: + - kustomizationRef + type: object branch: description: |- Branch to use for this target. @@ -309,44 +358,25 @@ spec: default: Ignore description: |- OnRefusal decides what happens when a live edit is refused because it has no legal - destination in the Git folder. A refusal produces no commit, so nothing normally reverts - the edit: with Argo CD selfHeal off it stays OutOfSync until a human acts, and under Flux - it survives until the next apply interval. + destination in the Git folder. A refusal produces no commit, so nothing reverts the edit + until something applies Git over it. "Ignore", the default, records the refusal in conditions and events and does nothing else. - "PushEmptyCommit" pushes a commit that changes no file, which moves the branch. Flux then - sees a new artifact revision and re-applies; Argo CD sees a revision it has not synced, so - its selfHeal=false skip does not apply and automated sync reverts the edit. - - It fires only when the refused write was an edit to a document the folder ALREADY holds and - is not removing. An object Git does not manage is left alone deliberately: re-applying would - do nothing to it (Flux prunes from its inventory and Argo CD from the resources it tracks, - and a live-created object is in neither), or, if it was managed and has since been removed - from Git, would prune it. That second one is the reconciler's decision to take on its own - schedule, not this operator's to hurry. Eligibility is per document, not per file: a write - that removes a document from a file other documents keep alive is still a removal, and a - flush that removes one anywhere is excluded whole. + "Reapply" asks this target's applier (spec.applier, which it requires) to apply the current + tip once the branch holds no open window and no pending writes, which reverts the refused + edit and nothing that is still waiting to be published. No other folder on the branch is + woken. - A standing refusal earns one commit, not one per recheck. The target re-reads its folder and - refuses again for as long as nobody corrects the edit, and an observation of the same objects - with the same issues is already covered by the commit that was made for it. A different - refused edit is a new commit, as is the same edit re-made after the target accepted a write - in between. - - It also fires only for a refusal where the folder itself is accepted and this one write had - nowhere to land. A folder-level refusal (unparseable YAML, a foreign file) is left alone: an - empty commit cannot repair a folder, and the reconciler may be mid-way through its own - corrections there. - - Two consequences remain to weigh. The commit wakes EVERYTHING watching the branch, not just - the refused object, so it can hurry along unrelated work including another target's pending - prune. And it can revert allowed live edits that are still waiting in the commit window. An - Argo CD Application carrying argocd.argoproj.io/manifest-generate-paths ignores the commit - entirely, because no file under its refresh paths changed. + It fires only for an edit to a document the folder ALREADY holds, in a write that removes no + document anywhere: re-applying would otherwise prune, or put back, an object somebody else is + deleting. A folder-level refusal (unparseable YAML, a foreign file) never fires it, because + re-applying Git cannot repair a folder. A standing refusal is requested once, not once per + recheck; a different refused edit, or the same edit re-made after a write of that type was + accepted, is a new request. enum: - Ignore - - PushEmptyCommit + - Reapply type: string parentBranch: description: |- @@ -541,9 +571,35 @@ spec: - message: spec.clusterProviderRef is immutable; delete and recreate the GitTarget to change the cluster it mirrors rule: self.clusterProviderRef == oldSelf.clusterProviderRef + - message: 'spec.onRefusal: Reapply needs spec.applier, the object asked + to reapply' + rule: '!has(self.onRefusal) || self.onRefusal != ''Reapply'' || has(self.applier)' status: description: status defines the observed state of GitTarget properties: + applier: + description: |- + Applier is the last apply request made to spec.applier: the commit, when, and why. Absent + until the first request. + properties: + reason: + description: Reason is why it was asked. + enum: + - ForeignCommit + - Refusal + - CatchUp + type: string + requestedAt: + description: |- + RequestedAt is when the request was made. It says the applier was asked, never that it + applied: read the applier's own status for that. + format: date-time + type: string + requestedCommit: + description: RequestedCommit is the commit the applier was asked + to apply. + type: string + type: object conditions: description: Conditions represent the latest available observations of an object's state diff --git a/config/deployment.yaml b/config/deployment.yaml index 6b34d435..393d912c 100644 --- a/config/deployment.yaml +++ b/config/deployment.yaml @@ -48,6 +48,9 @@ spec: # somebody moved from outside, without waiting out the 10m default. The effective # granularity is still the 5m control-plane reconcile. - --git-refresh-interval=30s + # The push receiver, so the e2e suite can deliver a Git host's push webhook through the + # gitops-reverser-receiver Service (receiver-service.yaml) instead of waiting for a refresh. + - --receiver-bind-address=:9292 # Dev/e2e only: the in-cluster Gitea SSH host key is not pinned, so permit SSH # when no known_hosts source exists. Never set this in production. - --insecure-allow-missing-known-hosts @@ -92,6 +95,9 @@ spec: - containerPort: 8443 name: metrics protocol: TCP + - containerPort: 9292 + name: receiver + protocol: TCP readinessProbe: httpGet: path: /readyz diff --git a/config/kustomization.yaml b/config/kustomization.yaml index cdde3c56..a4c7da4b 100644 --- a/config/kustomization.yaml +++ b/config/kustomization.yaml @@ -6,6 +6,7 @@ resources: - rbac - service.yaml - audit-service.yaml +- receiver-service.yaml - deployment.yaml - certs - webhook diff --git a/config/rbac/flux-applier-role.yaml b/config/rbac/flux-applier-role.yaml new file mode 100644 index 00000000..48051a1d --- /dev/null +++ b/config/rbac/flux-applier-role.yaml @@ -0,0 +1,36 @@ +# What GitOps Reverser needs to drive a Flux Kustomization named in GitTarget.spec.applier: read +# the Kustomization and the GitRepository it applies from, stamp reconcile.fluxcd.io/requestedAt on +# either (the request `flux reconcile kustomization --with-source` makes), and check that it may. +# +# It is kept OUT of role.yaml because no GitTarget needs it until one names an applier, and an +# install that does not use Flux should not hold patch rights on Flux objects. Drop this pair when +# no GitTarget sets spec.applier (Helm: `applierAccess.flux: false`, the default there). For a +# remote source cluster, grant the same to the identity of the ClusterProvider's kubeconfig there. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/managed-by: kustomize + app.kubernetes.io/name: gitops-reverser + name: gitops-reverser-flux-applier +rules: + - apiGroups: + - kustomize.toolkit.fluxcd.io + resources: + - kustomizations + verbs: + - get + - patch + - apiGroups: + - source.toolkit.fluxcd.io + resources: + - gitrepositories + verbs: + - get + - patch + - apiGroups: + - authorization.k8s.io + resources: + - selfsubjectaccessreviews + verbs: + - create diff --git a/config/rbac/flux-applier-rolebinding.yaml b/config/rbac/flux-applier-rolebinding.yaml new file mode 100644 index 00000000..3e36d690 --- /dev/null +++ b/config/rbac/flux-applier-rolebinding.yaml @@ -0,0 +1,15 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + app.kubernetes.io/managed-by: kustomize + app.kubernetes.io/name: gitops-reverser + name: gitops-reverser-flux-applier +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: gitops-reverser-flux-applier +subjects: +- kind: ServiceAccount + name: gitops-reverser + namespace: sut diff --git a/config/rbac/kustomization.yaml b/config/rbac/kustomization.yaml index 5907796f..10cdf148 100644 --- a/config/rbac/kustomization.yaml +++ b/config/rbac/kustomization.yaml @@ -7,6 +7,9 @@ resources: # See docs/rbac.md; the Helm chart renders the same pair from `rbac.watchTypes`. - watch-any-role.yaml - watch-any-rolebinding.yaml + # Patch rights on Flux Kustomizations and GitRepositories, for GitTarget.spec.applier. See the file. + - flux-applier-role.yaml + - flux-applier-rolebinding.yaml - cluster-role-jane-aggregated.yaml - cluster-rolebinding.yaml - cluster-rolebinding-jane.yaml # E2E-only helper: allows impersonated user jane@acme.com writes in tests. diff --git a/config/receiver-service.yaml b/config/receiver-service.yaml new file mode 100644 index 00000000..23263159 --- /dev/null +++ b/config/receiver-service.yaml @@ -0,0 +1,20 @@ +# The push receiver: plain HTTP for Git hosts' push webhooks (--receiver-bind-address in +# deployment.yaml). A delivery is POST /git-push//. +apiVersion: v1 +kind: Service +metadata: + labels: + app.kubernetes.io/managed-by: kustomize + app.kubernetes.io/name: gitops-reverser + name: gitops-reverser-receiver + namespace: sut +spec: + type: ClusterIP + ports: + - name: http + port: 80 + protocol: TCP + targetPort: receiver + selector: + app.kubernetes.io/name: gitops-reverser + control-plane: gitops-reverser diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index 041a945f..c8a3cdd4 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -7,6 +7,29 @@ guidance that the changelog's breaking-change entries link to. We are pre-1.0, so breaking changes bump the **minor** version (release-please is configured with `bump-minor-pre-major`) rather than the major. Read the relevant entry before upgrading across it. +## `onRefusal: PushEmptyCommit` is now `onRefusal: Reapply`, and needs `spec.applier` + +**Breaking.** `GitTarget.spec.onRefusal` accepts `Ignore` and `Reapply`. `PushEmptyCommit` is gone: +GitOps Reverser no longer moves the branch with an empty commit when an edit is refused. With +`Reapply` it asks the target's applier, the Flux `Kustomization` named in the new +`spec.applier.kustomizationRef`, to apply the current tip once the branch holds no open commit window +and no pending writes. Only that applier is asked, so other folders on the branch are no longer woken, +and an allowed edit still in a commit window is published before the apply instead of being reverted +with the refused one. The API rejects `Reapply` without `spec.applier`. + +A stored `GitTarget` that still says `PushEmptyCommit` reads as `Ignore`, and any update to it is +rejected until the field changes. Before upgrading, set `onRefusal: Ignore` on those targets, or +`onRefusal: Reapply` together with `spec.applier`. See +[reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal). + +`spec.applier` also changes what else applies the folder: GitOps Reverser asks the Kustomization to +apply when somebody else pushes to the branch, and to catch up with its own pushes once the branch +has been quiet for `spec.applier.quietPeriod`. Lengthen the `GitRepository` and `Kustomization` +intervals and remove any Git-host webhook into a Flux `Receiver` for the repository, as the +[bi-directional guide](bi-directional.md#configure-flux) describes, or Flux keeps applying on its own +schedule as well. The operator needs `get` and `patch` on `kustomizations` and `gitrepositories` in +the source cluster: Helm `applierAccess.flux: true` grants it for the operator's own cluster. + ## `git_queue_drops_total` is now `git_queue_refusals_total` `gitopsreverser_git_queue_drops_total` is renamed `gitopsreverser_git_queue_refusals_total`, with the diff --git a/docs/architecture.md b/docs/architecture.md index 394bd51e..41e6f726 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -116,16 +116,15 @@ cluster snapshot: the cluster still holds the values from before the push, so a publish them over the incoming change. **GitOps Reverser decides when Git is applied to the cluster.** An apply is safe only when nothing -live is missing from the commit being applied, and only the branch worker can see the edits that -are live and not yet pushed. So a `GitTarget` names its **applier** (a Flux `Kustomization`, an -Argo CD `Application`, or a webhook), the applier's own triggers are switched off, and the branch -worker sends **apply requests** instead. A request is sent only on a **quiet branch**, with no open -commit window and no pending writes: for a commit somebody else pushed (after closing open windows -and replaying the pending writes onto it), for a refused edit the target opted to revert, and, for -Flux, to keep its artifact current after GitOps Reverser's own pushes. Argo CD is never asked to -apply GitOps Reverser's own commits, because the cluster already holds those values. A request asks -and never reports; it runs off the publication path and cannot fail a publication. See -[the apply trigger design](design/push-notification-and-reconcile-trigger.md). +live is missing from the commit being applied, and only the branch worker can see the edits that are +live and not yet pushed. So a `GitTarget` names its **applier** (a Flux `Kustomization`), the +applier's own short triggers are switched off, and the branch worker sends **apply requests** +instead. A request is sent only on a **quiet branch**, with no open commit window and no pending +writes: for a commit somebody else pushed (after closing open windows and replaying the pending +writes onto it), for a refused edit the target opted to revert, and to keep Flux's artifact current +after GitOps Reverser's own pushes, because Flux's timers and restarts apply whatever artifact it +holds. A request asks and never reports; it runs off the publication path and cannot fail a +publication. See [the apply trigger design](design/push-notification-and-reconcile-trigger.md). **Redis/Valkey is optional but advised.** The default configured-author mode runs without it: a plain `helm install` comes up healthy and watches initialize from current state on startup. When an endpoint @@ -264,7 +263,7 @@ graph LR subgraph SOURCE["Source cluster selected by ClusterProvider"] API["Kubernetes API\ndiscovery · Namespace policy · watches"] - APPLIER["Applier\nFlux Kustomization or Argo CD Application"] + APPLIER["Applier\nFlux Kustomization"] end HOST[("Git host")] @@ -290,8 +289,8 @@ source context: client, discovery catalog, followability registry, source-namesp reachability state. Its `GitTarget`s open their source watches through that context. The contexts are keyed by provider name and are released when no `GitTarget` references them. -The applier is not one of the six kinds. It belongs to Flux or Argo CD, lives in the source cluster -it applies to, and is reached through the same source context and identity the watches use. +The applier is not one of the six kinds. It belongs to Flux, lives in the source cluster it applies +to, and is reached through the same source context and identity the watches use. | CRD | Scope | One line role | |---|---|---| @@ -391,9 +390,8 @@ Key fields: - `spec.commit.window`: when this target's commit windows close: `idleTimeout` (default `5s`) of silence, or `maxDuration` (default `1m`) after the window opened. - `spec.commit.message`: `liveTemplate` / `reconcileTemplate` Go templates. -- `spec.applier`: optional; exactly one of `kustomizationRef`, `applicationRef` or `webhook`, plus - `quietPeriod` (default `30s`). The object that applies this folder, and that GitOps Reverser asks - to apply. +- `spec.applier`: optional; `kustomizationRef` (the Flux `Kustomization` that applies this folder, + and that GitOps Reverser asks to apply) and `quietPeriod` (default `30s`). - `spec.onRefusal`: `Ignore` (default) or `Reapply`, which asks the applier to reapply the tip when an edit to an existing document is refused. `Reapply` requires `spec.applier`. @@ -660,11 +658,12 @@ before `PushAtomic` returns is still recognized as its own. A rollback onto one commits is foreign, because the advertisement names a tip that is not the last one it pushed. The same quiet-branch rule sends two other requests: `Refusal`, to the refusing target's applier -when `spec.onRefusal: Reapply` is set, and `CatchUp`, to Flux and webhook appliers once the branch -has been quiet for `spec.applier.quietPeriod` after the worker's own pushes. Delivering them is -the job of the applier client in `internal/applier`, off the worker's queue: a Kubernetes patch -for Flux and Argo CD, a signed POST for a webhook, retried with backoff, recorded as an -`ApplyRequested` Event and on `status.applier`. A failed request never reaches back into the publication that +when `spec.onRefusal: Reapply` is set, and `CatchUp`, to every applier on the branch once it has +been quiet for `spec.applier.quietPeriod` after the worker's own pushes. Delivering them is the job +of the dispatcher in `internal/applier`, off the worker's queue: it stamps +`reconcile.fluxcd.io/requestedAt` on the `GitRepository` or the `Kustomization`, retries with +backoff, lets a newer request for the same `Kustomization` supersede one not yet sent, and records an +`ApplyRequested` Event and `status.applier`. A failed request never reaches back into the publication that caused it. *** @@ -1640,13 +1639,15 @@ as one funnel and its loss paths as one selector: [Interpreting metrics](interpreting-metrics.md#reading-git_fetches_total). **Apply requests.** `apply_requests_total{applier_kind,reason,outcome}` counts every -request the worker decided to send (`applier_kind` is `kustomization`, `application` or -`webhook`; `reason` is `ForeignCommit`, `Refusal` or `CatchUp`), with `outcome` separating +request the worker decided to send (`applier_kind` is `kustomization`; `reason` is +`ForeignCommit`, `Refusal` or `CatchUp`), with `outcome` separating `sent` from `failed` and `superseded` (a newer request for the same applier replaced it before it -went out). `receiver_deliveries_total{type,outcome}` counts receiver deliveries: -`foreign`, `own`, `ignored` (another branch, or a tag), and `rejected` (bad signature). A healthy -bi-directional branch shows `CatchUp` and `ForeignCommit` and never a request on a branch with -pending writes, which is what the decision rule promises. +went out). `receiver_deliveries_total{type,outcome}` counts receiver deliveries: `notified`, +`unmatched`, `ignored` (a ping, a tag, another event type), `rejected` (bad signature) and `invalid`. +The receiver cannot tell a foreign push from an own one; the worker decides that from the +advertisement, which is why a `notified` delivery on an otherwise quiet branch is followed by a +`CatchUp` and only a genuinely foreign one by a `ForeignCommit`. See +[Interpreting metrics](interpreting-metrics.md#apply-requests-and-the-push-receiver). **Authorship**, when `--author-attribution` is on: @@ -1719,8 +1720,9 @@ Current limitations: triggers are off, so while the operator is down a Git-side push reaches the cluster only through a manual sync, `flux reconcile`, or Flux's long backstop interval. - **The applier must live in the cluster it applies to.** It is looked up in the source cluster - with the `ClusterProvider`'s identity. A hub Argo CD whose `Application` lives in a management - cluster that is not the source is not supported. + with the `ClusterProvider`'s identity. +- **Only Flux can be an applier.** An Argo CD `Application` keeps its own triggers until it can be + named in `spec.applier`. - **An apply request cannot reserve time.** An edit made while a requested apply runs is reverted and mirrored as the applier's commit. The rule confines that to the seconds after a foreign push, a refusal with `Reapply`, or a quiet period on a Flux branch. @@ -1741,7 +1743,7 @@ Current limitations: | Package | Role | |---|---| | [api/v1alpha3/](../api/v1alpha3/) | CRD types | -| `internal/applier/` | apply requests: the Flux, Argo CD and webhook clients, and the `ApplierConfigured` check | +| [internal/applier/](../internal/applier/) | apply requests: the Flux client, the dispatcher that delivers them, and the `ApplierConfigured` check | | [cmd/](../cmd/) | operator entry point and server setup | | [internal/auditutil/](../internal/auditutil/) | audit identity, objectRef, and subresource helpers feeding attribution facts | | [internal/controller/](../internal/controller/) | Kubernetes reconcilers | diff --git a/docs/bi-directional.md b/docs/bi-directional.md index 88a2969d..e161c5d2 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -4,19 +4,20 @@ GitOps Reverser publishes live cluster state to Git. Flux or Argo CD applies Git cluster. Run both and a resource can be changed from either side. This guide covers who decides what, how to configure each actor, and which parts are proven by tests. -**The short version.** GitOps Reverser decides when the cluster is applied from Git. Flux and -Argo CD keep doing what they are good at (fetching, rendering, applying, pruning, reporting -health), but they apply when GitOps Reverser asks, not on their own schedule. You configure that -in three places: - -1. Name each folder's **applier** on its `GitTarget`: the Flux `Kustomization` or the Argo CD - `Application` that applies it. See [declaring the applier](#declaring-the-applier). -2. Switch off the applier's own triggers: Argo CD's automated sync, Flux's short intervals and any - Git-host webhook into Flux. See [configure Argo CD](#configure-argo-cd) and - [configure Flux](#configure-flux). -3. Point your Git host's push webhook at GitOps Reverser, not at the applier. See +**The short version.** GitOps Reverser decides when the cluster is applied from Git. Flux keeps +doing what it is good at (fetching, rendering, applying, pruning, reporting health), but it applies +when GitOps Reverser asks, not on its own schedule. You configure that in three places: + +1. Name each folder's **applier** on its `GitTarget`: the Flux `Kustomization` that applies it. See + [declaring the applier](#declaring-the-applier). +2. Switch off Flux's own triggers: short intervals and any Git-host webhook into a Flux `Receiver`. + See [configure Flux](#configure-flux). +3. Point your Git host's push webhook at GitOps Reverser instead. See [telling GitOps Reverser that somebody else pushed](#telling-gitops-reverser-that-somebody-else-pushed). +Argo CD cannot be driven this way yet; [configure Argo CD](#configure-argo-cd) says what that costs +in the meantime. + There is no merge and no lock. What remains exposed is small and named in [what is still exposed](#what-is-still-exposed). @@ -25,7 +26,7 @@ flowchart LR subgraph CLUSTER["Kubernetes cluster"] API["Kubernetes API"] REV["GitOps Reverser"] - APP["Applier
Flux Kustomization or
Argo CD Application"] + APP["Applier
Flux Kustomization"] end GIT[("Git branch")] HUMAN(["Somebody pushes to Git"]) @@ -48,7 +49,7 @@ flowchart LR | Kubernetes API | Takes edits from people and controllers | | | GitOps Reverser | Mirrors every edit into Git, and decides when the applier may apply | Apply anything to the cluster itself | | Git host | Stores the branch, and tells GitOps Reverser when somebody else pushed | Notify the applier | -| Applier (Flux or Argo CD) | Fetches, renders, applies and prunes, when asked | Apply on a timer or a webhook of its own | +| Applier (a Flux `Kustomization`) | Fetches, renders, applies and prunes, when asked | Apply on a short timer or a webhook of its own | | A human | Can still run `flux reconcile` or press Sync | | ## Why GitOps Reverser owns the trigger @@ -85,10 +86,10 @@ branch worker. That is the moment nothing live is missing from the tip. | --- | --- | --- | | `ForeignCommit` | The branch gained a commit GitOps Reverser did not push | Every applier declared on the branch | | `Refusal` | An edit was refused and the target has `onRefusal: Reapply` | The refusing target's applier | -| `CatchUp` | The branch has been quiet for `spec.applier.quietPeriod` after GitOps Reverser's own pushes | Flux and webhook appliers only | +| `CatchUp` | The branch has been quiet for `spec.applier.quietPeriod` after GitOps Reverser's own pushes | Every applier on the branch | -GitOps Reverser never requests an apply of its own commits from an Argo CD `Application`: the -cluster already holds those values, because they were read from it. Flux is the exception, and +The cluster already holds the values of GitOps Reverser's own commits, because they were read from +it, so applying them changes nothing. Flux is asked to catch up with them anyway, and [configure Flux](#configure-flux) says why. A foreign commit that arrives while edits are pending does not wait for their windows to close. @@ -137,7 +138,7 @@ spec: branch: main path: apps/checkout applier: - kustomizationRef: # or applicationRef, or webhook + kustomizationRef: namespace: flux-system name: checkout onRefusal: Reapply @@ -150,49 +151,40 @@ folder's repository and branch, or the target reports `ApplierConfigured=False`. [configuration reference](configuration.md#who-applies-this-folder-specapplier) has every field and reason. -Several `GitTarget`s may name the same applier, for example an `Application` whose path covers -several folders. Requests to one applier object are sent once. - -`webhook` is for any other tool: GitOps Reverser sends a signed POST naming the repository, the -branch, the commit and the reason, and the receiver decides what to do with it. +Several `GitTarget`s may name the same `Kustomization`, for example one whose path covers several +folders. Requests to one `Kustomization` are sent once. ## Configure Argo CD -Switch automated sync off and keep its settings. Argo CD 3.1 and later has a field for exactly -that: +GitOps Reverser cannot drive an Argo CD `Application` yet; `spec.applier` names a Flux +`Kustomization` only, and Argo CD is next. Until then an `Application` keeps its own triggers, with +the exposure [why GitOps Reverser owns the trigger](#why-gitops-reverser-owns-the-trigger) +describes: the timed refresh syncs each of GitOps Reverser's pushes two to three minutes later, over +whatever is pending at that moment. + +The configuration that keeps the most of a live edit in the meantime: ```yaml spec: syncPolicy: automated: - enabled: false # GitOps Reverser makes the sync requests - prune: true # still read: GitOps Reverser's syncs prune when this is true - syncOptions: - - ServerSideApply=true + selfHeal: false + prune: true ``` -GitOps Reverser's request is the sync operation Argo CD's own automated sync would have made, -pinned to one commit: `prune` from `automated.prune`, `syncOptions` and `retry` from the -`Application`, and `initiatedBy` naming GitOps Reverser. It waits for a running operation to -finish rather than replacing it. Because the commit is pinned, the sync applies exactly the commit -GitOps Reverser checked, not whatever the tip has become by the time it runs. - -With automated sync off, a refresh only compares: the timed refresh, a Git-host webhook into -Argo CD, and the UI's Refresh button all report `Synced` or `OutOfSync` and apply nothing. So -`timeout.reconciliation` can stay at its default, and a Git-host webhook into Argo CD is optional; -it only makes the comparison status fresher. `selfHeal` has nothing left to act on. - -On an Argo CD older than 3.1, omit `automated` altogether. GitOps Reverser's syncs then do not -prune. - -Every sync runs the `Application`'s resource hooks, which is the other reason GitOps Reverser never -syncs an `Application` for its own commits. +With `selfHeal: true` Argo CD reverts a live edit in about a second, usually before GitOps Reverser +can publish it, and Git records a two-commit flap. `selfHeal: false` only suppresses repeating a +sync of a commit already synced, so a new commit or a manual sync still applies Git over an +unpublished API edit. Use `prune: true` only when a deletion in Git should delete the live object, +and point the Git host's webhook at `https:///api/webhook` so a Git-side push is +synced in seconds rather than on the timer. ### The split-ownership alternative -`ignoreDifferences` on a field, with `RespectIgnoreDifferences=true` in `syncOptions`, hands that -one field to the API. Its differences alone do not make the `Application` `OutOfSync`, and the -option copies the live value into the apply target, so GitOps Reverser's syncs preserve it. +`ignoreDifferences` on a field, with `RespectIgnoreDifferences=true` in `syncOptions`, keeps +`selfHeal: true` everywhere else and hands that one field to the API. Its differences alone do not +make the `Application` `OutOfSync`, and the option copies the live value into the apply target, so an +ordinary sync preserves it. The cost is that the field stops being Git-driven on an existing object: Git seeds it at creation, but later Git edits to it do not reach the object while the ignore is active. It protects no other @@ -221,7 +213,7 @@ the one `flux reconcile kustomization --with-source` makes: it stamps `reconcile.fluxcd.io/requestedAt` on the `GitRepository`, and stamps the `Kustomization` too when the artifact already holds the requested commit. -Flux needs one request Argo CD does not. Its timers and a controller restart apply whatever artifact +Flux needs one more request than the cluster's own state suggests. Its timers and a controller restart apply whatever artifact the `GitRepository` holds, and with a long interval that artifact falls behind GitOps Reverser's own pushes. Applying a stale artifact would revert published edits, not only pending ones. So after its own pushes, once the branch has been quiet for `spec.applier.quietPeriod` (`30s` by default), @@ -470,20 +462,21 @@ concurrent writers. | Covered | Where | | --- | --- | -| Argo CD bookkeeping never reaches Git, asserted as zero commits during a sync | [Argo CD](../test/e2e/argocd_bi_directional_e2e_test.go) | -| Two API edits to one field seconds apart end at the later value, with no commit authored by the applier | Argo CD and [Flux](../test/e2e/flux_bi_directional_e2e_test.go) | -| With a short `timeout.reconciliation`, Argo CD applies nothing GitOps Reverser did not request | Argo CD | -| A Git-side push while an API edit is pending lands both, and the applier applies the replayed commit | Argo CD and Flux | -| After a quiet period the `GitRepository` artifact holds the tip, and a forced `Kustomization` reconcile changes nothing | Flux | -| A refused edit with `onRefusal: Reapply` is reverted, and only its own applier is asked | Argo CD and Flux | -| With no receiver configured, a foreign push is still applied after the refresh finds it | Flux | +| Two API edits to one field seconds apart end at the later value with no further commit, and Flux catches up with both by itself | [Flux](../test/e2e/flux_bi_directional_e2e_test.go) | +| A Git-side push with an API edit on top lands both, and Flux applies the replayed commit without being reconciled by hand | Flux | +| A new `GitRepository` revision is enough for the `Kustomization` to apply and revert drift | Flux | +| A refused edit with `onRefusal: Reapply` is reverted by its `Kustomization`, with no commit, and settles instead of repeating | [Refusal reapply](../test/e2e/refusal_reapply_e2e_test.go) | +| A Git to cluster to Git round trip settles without a commit loop | Flux | | A SOPS-encrypted Secret round-trips without a reverse commit | Flux | | A Git revert removes the live object and the committed file | Flux | +| Argo CD bookkeeping never reaches Git, asserted as zero commits during a sync | [Argo CD](../test/e2e/argocd_bi_directional_e2e_test.go) | +| `selfHeal: true` loses the API edit, and the flap is exactly two commits; `selfHeal: false` keeps it | Argo CD | **No spec tests simultaneous writes to one field from both sides**, because the outcome depends on arrival order the corner does not control. The replay mechanism is covered; a conflict policy is not, because there is not one. -Run `task test-e2e-bi-directional` for the corner, which is also a CI job. `task argocd-ui` opens -the installed Argo CD UI. The [corner specification](spec/e2e-bi-directional-corner.md) describes +Run `task test-e2e-bi-directional` for the corner, which is also a CI job. `task test-e2e` excludes +the corner and includes the refusal spec. `task argocd-ui` opens the installed Argo CD UI. The +[corner specification](spec/e2e-bi-directional-corner.md) describes the setup. diff --git a/docs/configuration.md b/docs/configuration.md index 8b5b15ca..8faeedbd 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -32,7 +32,7 @@ The usual flow is: 3. Create one or more `WatchRule` or `ClusterWatchRule` objects that point at that target. 4. Create a `CommitRequest` only when you want a save: your writes in one commit, on the request's timers rather than the target's, and optionally with your message. -5. When the folder is also applied back to the cluster by Flux or Argo CD, name that applier on the +5. When Flux also applies the folder back to the cluster, name its `Kustomization` on the `GitTarget` and give the `GitProvider` a push receiver, so GitOps Reverser decides when Git is applied. See [who applies this folder](#who-applies-this-folder-specapplier) and the [bi-directional guide](bi-directional.md). @@ -65,6 +65,7 @@ fields that need more than a line. | `knownHostsRef` | the Secret's own `known_hosts` | Optional ConfigMap or Secret sharing SSH host keys across providers. Host-key verification fails closed either way | | `commit.committer` | the operator's built-in identity | Name and email written as the commit committer. See [committer identity](#committer-identity) | | `commit.signing` | off | SSH commit signing. See [commit signing](#commit-signing) | +| `receiver` | off | Push webhook endpoint, so the Git host can say somebody pushed. See [receiving push notifications](#receiving-push-notifications-specreceiver) | ### `ClusterProvider` (cluster-scoped): the source cluster @@ -96,6 +97,7 @@ The destination fields are immutable: to move a target, delete it and create a n | `placement.useKustomize` | off | Keep the folder a kustomize folder. See [keeping the folder a kustomize folder](#keeping-the-folder-a-kustomize-folder-specplacementusekustomize) | | `serializeNamespace` | inferred per document | Whether a committed document carries its own `metadata.namespace`. An explicit `false` admits exactly one source namespace. See [whether documents carry their namespace](#whether-documents-carry-their-namespace-specserializenamespace) | | `prune.mode` | `OnEvent` | Which deletions reach Git. See [deletion policy](#deletion-policy-specprunemode) | +| `applier` | Flux applies on its own schedule | The Flux `Kustomization` that applies this folder (`kustomizationRef`), which GitOps Reverser then asks to apply, and how long the branch stays quiet before it catches up (`quietPeriod`, `30s`). See [who applies this folder](#who-applies-this-folder-specapplier) | | `onRefusal` | `Ignore` | What happens to an edit with no legal destination. See [reverting a refused edit](#reverting-a-refused-edit-speconrefusal) | | `encryption` | off | SOPS + age encryption for sensitive resources | | `suspend` | `false` | Stop writing without deleting the target. See [stopping a target from writing](#stopping-a-target-from-writing-specsuspend) | @@ -621,8 +623,8 @@ The important fields are: omit it and each document's namespace is inferred from the folder - `spec.prune`: which deletion paths may remove documents from this target's folder (see [Deletion policy](#deletion-policy-specprunemode)); omit it for the safe default -- `spec.applier`: the Flux `Kustomization` or Argo CD `Application` that applies this folder, so - GitOps Reverser decides when it applies (see +- `spec.applier`: the Flux `Kustomization` that applies this folder, so GitOps Reverser decides when + it applies (see [Who applies this folder](#who-applies-this-folder-specapplier)); omit it when nothing applies the folder back - `spec.onRefusal`: whether a refused edit asks the applier to reapply Git (see @@ -1088,81 +1090,58 @@ Git-host configuration. Seconds-fresh is the job of the ### Who applies this folder (`spec.applier`) -When Flux or Argo CD applies this folder back to the cluster, `spec.applier` names the object that -does it, and GitOps Reverser becomes the one that decides when it applies. The applier keeps doing -the fetching, rendering, applying and pruning. The reason it has to be this way, and how to -configure each tool, is the [bi-directional guide](bi-directional.md). - -Exactly one of three: +When Flux applies this folder back to the cluster, `spec.applier` names the `Kustomization` that does +it, and GitOps Reverser becomes the one that decides when it applies. Flux keeps doing the fetching, +rendering, applying and pruning. The reason it has to be this way, and how to configure Flux for it, +is the [bi-directional guide](bi-directional.md). ```yaml spec: applier: - kustomizationRef: # a Flux Kustomization - namespace: flux-system + kustomizationRef: + namespace: flux-system # default: this GitTarget's namespace name: checkout -``` - -```yaml -spec: - applier: - applicationRef: # an Argo CD Application - namespace: argocd - name: checkout -``` - -```yaml -spec: - applier: - webhook: # anything else - url: https://deployer.example.com/apply - secretRef: - name: deployer-hmac # key: token - quietPeriod: 30s # kustomizationRef and webhook only + quietPeriod: 30s ``` | Field | Default | What it does | | --- | --- | --- | -| `kustomizationRef` | | A Flux `Kustomization` whose `sourceRef` is a `GitRepository` on this repository and branch | -| `applicationRef` | | An Argo CD `Application` whose source is this repository and branch | -| `webhook.url` | | Where to POST each apply request | -| `webhook.secretRef` | | Secret whose `token` key signs the request | -| `quietPeriod` | `30s` | How long the branch must stay quiet after GitOps Reverser's own pushes before a `CatchUp` request. Ignored for `applicationRef` | - -**Where the reference is looked up.** In the source cluster, the cluster the applier applies to, -with the identity the [`ClusterProvider`](#clusterprovider) already uses. For the operator's own -cluster the chart grants `get` and `patch` on `kustomizations` and `gitrepositories` -(`kustomize.toolkit.fluxcd.io`, `source.toolkit.fluxcd.io`) and on `applications` -(`argoproj.io`). For a remote source cluster, grant the same to the kubeconfig's identity. An -applier that lives in a cluster other than the one it applies to, such as a hub Argo CD managing a -remote cluster, is not supported. +| `kustomizationRef` | **required** | A Flux `Kustomization` whose `sourceRef` is a `GitRepository` on this target's repository and branch | +| `quietPeriod` | `30s` | How long the branch must stay quiet after GitOps Reverser's own pushes before a `CatchUp` request. At most `1h` | + +Argo CD (`applicationRef`) and a webhook for any other tool are the next appliers; until they +exist, an Argo CD `Application` keeps its own triggers. + +**Where the reference is looked up.** In the source cluster, the cluster the `Kustomization` applies +to, with the identity the [`ClusterProvider`](#clusterprovider) already uses. That identity needs `get` +and `patch` on `kustomizations` (`kustomize.toolkit.fluxcd.io`) and `gitrepositories` +(`source.toolkit.fluxcd.io`), and `create` on `selfsubjectaccessreviews` to check it. For the +operator's own cluster the chart grants them with `applierAccess.flux: true`; for a remote source +cluster, grant the same to the kubeconfig's identity. A `Kustomization` that lives in a cluster other +than the one it applies to is not supported. **When a request is made.** Only on a quiet branch: no open commit window and no pending writes on the branch worker, which is the moment nothing live is missing from the tip. | `reason` | When | Sent to | | --- | --- | --- | -| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Open windows are closed and pending writes replayed and pushed first, so the request names that pushed commit | Every applier declared on the branch, once per applier object | +| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Open windows are closed and pending writes replayed and pushed first, so the request names that pushed commit | Every applier declared on the branch, once per `Kustomization` | | `Refusal` | An eligible refused edit, with `spec.onRefusal: Reapply` | This target's applier | -| `CatchUp` | The branch has been quiet for `quietPeriod` since GitOps Reverser's own pushes | `kustomizationRef` and `webhook` appliers on the branch | +| `CatchUp` | The branch has been quiet for `quietPeriod` since GitOps Reverser's own pushes, or since the operator started | Every applier declared on the branch | -An `applicationRef` applier is never asked to apply GitOps Reverser's own commits: with automated -sync off, Argo CD applies nothing on its own, so there is no stale state to catch up, and every sync -runs the `Application`'s hooks. Flux applies on its timers and on a controller restart, so its -`GitRepository` artifact is kept current instead. +Flux applies on its own timers and on a controller restart, from whatever artifact the +`GitRepository` holds. `CatchUp` keeps that artifact at the tip, so such an apply applies values the +cluster already holds; an artifact left behind would revert edits that are already in Git. +`quietPeriod` keeps the catch-up out of an editing session: every new write restarts it. -**What a request does.** - -| Applier | Request | -| --- | --- | -| `kustomizationRef` | What `flux reconcile kustomization --with-source` does: stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository`, then on the `Kustomization` when the artifact already held the commit. A `CatchUp` stamps only the `GitRepository` | -| `applicationRef` | Set `operation.sync` exactly as Argo CD's automated sync would: `revision` pinned to the commit, `prune` from `syncPolicy.automated.prune`, `syncOptions` and `retry` from the `Application`, `initiatedBy.username: gitops-reverser`. A running operation is waited for, never replaced | -| `webhook` | POST `{"repository", "branch", "commit", "reason"}` with `X-Signature: sha256=`, the header Flux's `generic-hmac` `Receiver` checks | +**What a request does.** What `flux reconcile kustomization --with-source` does: stamp +`reconcile.fluxcd.io/requestedAt` on the `GitRepository`, and on the `Kustomization` instead when +the artifact already holds the requested commit. A new artifact wakes the `Kustomization` by itself. +A `CatchUp` whose artifact already holds the commit stamps nothing. A request asks; it never claims the apply happened. It runs off the publication path, is retried with backoff on its own, and a failed request never fails a publication that succeeded. Read the -applier's own status (`lastAppliedRevision` on a `Kustomization`, `status.sync` on an -`Application`) for what it applied. +`Kustomization`'s `status.lastAppliedRevision` for what it applied. **What it reports.** @@ -1171,8 +1150,8 @@ status: conditions: - type: ApplierConfigured status: "False" - reason: AutomatedSyncEnabled - message: 'Application argocd/checkout has automated sync enabled, so it also applies on its own' + reason: SourceMismatch + message: 'Kustomization flux-system/checkout applies GitRepository flux-system/infra, whose branch is "release", not "main"' applier: requestedCommit: 4f2c1ab9... requestedAt: "2026-10-20T10:14:03Z" @@ -1183,13 +1162,12 @@ status: | Reason | Means | | --- | --- | -| `ApplierNotFound` | No such object in the source cluster, or its kind is not installed there | -| `SourceMismatch` | The applier applies a different repository or branch | -| `AutomatedSyncEnabled` | The `Application` still syncs on its own. Requests are still sent; the guarantee is not there | -| `Forbidden` | The source-cluster identity may not read or patch the applier | +| `ApplierNotFound` | No such `Kustomization` in the source cluster, or Flux is not installed there | +| `SourceMismatch` | The `Kustomization` applies a different repository or branch, or a source that is not a `GitRepository` | +| `Forbidden` | The source-cluster identity may not patch the `Kustomization` or its `GitRepository` | -It is checked on the target's reconcile tick rather than watched, so a cluster without the Flux or -Argo CD CRDs installed costs nothing. Each request also records an `ApplyRequested` Event on the +It is checked on the target's reconcile tick rather than watched, so a cluster without the Flux CRDs +installed costs nothing. Each delivered request also records an `ApplyRequested` Event on the `GitTarget`, and a request that keeps failing records `ApplyRequestFailed`. ### Reverting a refused edit (`spec.onRefusal`) @@ -2484,12 +2462,14 @@ present the client certificate the audit listener requires, so the two cannot sh | --- | --- | --- | --- | | `--receiver-bind-address` | `receiver.bindAddress` | `0` (off) | `host:port` the receiver listens on. `0` serves no receiver | -The chart creates a `gitops-reverser-receiver` Service when the listener is on. Expose it to your +The chart creates a `-receiver` Service when the listener is on. Expose it to your Git host with an Ingress or a Gateway route of your own that terminates TLS, the way Flux's `webhook-receiver` Service is exposed. Every request still has to carry a valid signature for the route it names; a route whose `GitProvider` has no `spec.receiver` does not serve. -Like everything that drives a branch worker, a delivery is handled on the elected replica. +Like everything that drives a branch worker, the listener runs on the elected replica only; a +standby refuses the connection. With the chart's default single replica that is the only pod. The +Service targets port `receiver` and is exposed on `service.ports.receiver` (`80` by default). ## Quickstart vs hand-managed resources diff --git a/docs/design/branch-worker-event-model.md b/docs/design/branch-worker-event-model.md index 58c2f497..6d55bd8f 100644 --- a/docs/design/branch-worker-event-model.md +++ b/docs/design/branch-worker-event-model.md @@ -479,7 +479,7 @@ message does not preserve the race guarantee by itself. The [write gates](../../internal/git/write_gate.go) apply suspension and render fidelity when a commit is decided. A commit already created locally can still publish after a gate closes. -Delayed [refusal actions](../../internal/git/refusal_touch.go) recheck consent and suspension. +Delayed [refusal actions](../../internal/git/refusal_reapply.go) recheck consent and suspension. Preserve those distinct effective points and record their decisions. Applying current gates while reconstructing a previously completed decision would rewrite its history. diff --git a/docs/design/collection-terminology-rename.md b/docs/design/collection-terminology-rename.md index de384a9c..781ac47d 100644 --- a/docs/design/collection-terminology-rename.md +++ b/docs/design/collection-terminology-rename.md @@ -270,7 +270,7 @@ A type observed in three namespaces still counts as one type in public status. ### Git writes, resyncs, and refusals Rename provenance in `Event`, `ResyncRequest`, write aggregation, queue logging, and -[`refusal_touch.go`](../../internal/git/refusal_touch.go). Preserve these distinct cases: +[`refusal_reapply.go`](../../internal/git/refusal_reapply.go). Preserve these distinct cases: - A single producer supplies its collection; mixed-producer or whole-target work remains unclaimed. - `ResyncRequest.refusalCollection()` continues deriving its key from the resync scope. It must @@ -497,8 +497,8 @@ identifier occurrences; `comments` and `strings` count matched words/components | [`internal/git/git_path_refusal.go`](../../internal/git/git_path_refusal.go) | 7 | 2 | 1 | 10 | | [`internal/git/git_path_refusal_test.go`](../../internal/git/git_path_refusal_test.go) | 16 | 0 | 0 | 16 | | [`internal/git/refusal_observation_test.go`](../../internal/git/refusal_observation_test.go) | 9 | 9 | 3 | 21 | -| [`internal/git/refusal_touch.go`](../../internal/git/refusal_touch.go) | 23 | 14 | 1 | 38 | -| [`internal/git/refusal_touch_test.go`](../../internal/git/refusal_touch_test.go) | 2 | 2 | 0 | 4 | +| [`internal/git/refusal_reapply.go`](../../internal/git/refusal_reapply.go) | 23 | 14 | 1 | 38 | +| [`internal/git/refusal_reapply_test.go`](../../internal/git/refusal_reapply_test.go) | 2 | 2 | 0 | 4 | | [`internal/git/render_fidelity_gate.go`](../../internal/git/render_fidelity_gate.go) | 13 | 6 | 0 | 19 | | [`internal/git/render_fidelity_gate_test.go`](../../internal/git/render_fidelity_gate_test.go) | 16 | 7 | 8 | 31 | | [`internal/git/resync_flush.go`](../../internal/git/resync_flush.go) | 2 | 3 | 0 | 5 | @@ -542,7 +542,7 @@ identifier occurrences; `comments` and `strings` count matched words/components | [`internal/watch/watch_plane_state.go`](../../internal/watch/watch_plane_state.go) | 5 | 5 | 0 | 10 | | [`internal/watch/watched_type_resolver.go`](../../internal/watch/watched_type_resolver.go) | 1 | 5 | 0 | 6 | | [`test/e2e/audit_route_attribution_e2e_test.go`](../../test/e2e/audit_route_attribution_e2e_test.go) | 0 | 2 | 0 | 2 | -| [`test/e2e/refusal_empty_commit_e2e_test.go`](../../test/e2e/refusal_empty_commit_e2e_test.go) | 0 | 1 | 0 | 1 | +| [`test/e2e/refusal_reapply_e2e_test.go`](../../test/e2e/refusal_reapply_e2e_test.go) | 0 | 1 | 0 | 1 | ### Current documents diff --git a/docs/design/label-selection-follow-ups.md b/docs/design/label-selection-follow-ups.md index 3458dc1e..ef33deb7 100644 --- a/docs/design/label-selection-follow-ups.md +++ b/docs/design/label-selection-follow-ups.md @@ -17,7 +17,7 @@ For context only. None of these is open. |---|---| | The recovery hint for `unowned-delete-patch` said "rename or remove". For an object a patch hides, the refusal's path is the hidden base document, so the hint read as "delete your manifest". | The hint names the patch and its kustomization reference. | | A batch holding two events for one inherited object read the pre-batch render for the second one. A re-entry followed by an upsert refused the batch as hidden by an unowned patch. A re-entry followed by a delete left the object rendered. | The batch records what it did to each inherited object's presence and consults that record before the pre-batch store. See `rendersInBatch` in [`plan_flush.go`](../../internal/git/plan_flush.go). | -| Refusal memory and queued `onRefusal` commits were keyed by the selector-bearing collection. After a selector change, the old key survived recovery. A first attempt fixed only the resync side and broke live refusals: those still used the selector-bearing key, so an accepted snapshot never cleared them, even with an unchanged selector. | Every refusal key is built by `newRefusalKey`, which drops the selector. The full collection is still used for watch identity and status. See [`refusal_touch.go`](../../internal/git/refusal_touch.go). | +| Refusal memory and queued `onRefusal` commits were keyed by the selector-bearing collection. After a selector change, the old key survived recovery. A first attempt fixed only the resync side and broke live refusals: those still used the selector-bearing key, so an accepted snapshot never cleared them, even with an unchanged selector. | Every refusal key is built by `newRefusalKey`, which drops the selector. The full collection is still used for watch identity and status. See [`refusal_reapply.go`](../../internal/git/refusal_reapply.go). | | `targetWatchSet.stop` retired the producer gate before cancelling the stream. An in-flight snapshot enqueue blocked in `Client.Get` on the stream context could hold the owner loop. | Cancel first, then retire. The gate still admits nothing afterwards. | About the second row: a commit window keeps only the last event per path, and every pending write diff --git a/docs/design/push-notification-and-reconcile-trigger.md b/docs/design/push-notification-and-reconcile-trigger.md index 868630fa..d100a45e 100644 --- a/docs/design/push-notification-and-reconcile-trigger.md +++ b/docs/design/push-notification-and-reconcile-trigger.md @@ -1,6 +1,7 @@ # Push notification and the apply trigger -> **design**: decided 2026-10-16, not built. Index: [`../INDEX.md`](../INDEX.md) +> **design**: decided 2026-10-16; the Flux applier, the receiver and `Reapply` are built, Argo CD +> and the webhook applier are not. Index: [`../INDEX.md`](../INDEX.md) > User-facing model: [`../bi-directional.md`](../bi-directional.md). Fields: > [`../configuration.md`](../configuration.md#who-applies-this-folder-specapplier). The request that > led here: [`bi-directional-right.md`](bi-directional-right.md). @@ -131,15 +132,16 @@ accepted. Bi-directional editing already stops in that state. ### 3.1 The applier -`GitTarget.spec.applier` names exactly one applier, enforced with a CEL rule: +`GitTarget.spec.applier` names the applier: -| Field | Applier | -| --- | --- | -| `kustomizationRef` | a Flux `Kustomization` whose `sourceRef` is a `GitRepository` | -| `applicationRef` | an Argo CD `Application` | -| `webhook` (`url`, `secretRef`) | anything else, through a signed POST | +| Field | Applier | Built | +| --- | --- | --- | +| `kustomizationRef` | a Flux `Kustomization` whose `sourceRef` is a `GitRepository` | yes | +| `applicationRef` | an Argo CD `Application` | next | +| `webhook` (`url`, `secretRef`) | anything else, through a signed POST | after that | -plus `quietPeriod` (default `30s`, a Go duration string), used for `CatchUp` (§3.5). +plus `quietPeriod` (default `30s`, a Go duration string), used for `CatchUp` (§3.5). Once a second +kind exists, a CEL rule requires exactly one. The reference is a declaration, not an inference. The earlier objection to naming an orchestrator object was to **deriving** which object applies a folder: neither Flux nor Argo CD derives that @@ -230,7 +232,7 @@ applier could not fetch from either. | --- | --- | --- | --- | | `ForeignCommit` | §3.3 | The tip once the branch is quiet, after any replay | Every applier on the branch | | `Refusal` | An eligible refusal (§3.7) on a target with `onRefusal: Reapply` | The tip once the branch is quiet | That target's applier | -| `CatchUp` | An accepted push of the worker's own | The tip once the branch has been quiet for `quietPeriod` | `kustomizationRef` and `webhook` appliers | +| `CatchUp` | An accepted push of the worker's own, or its first observation of the branch | The tip once the branch has been quiet for `quietPeriod` | `kustomizationRef` and `webhook` appliers | **Why Flux gets `CatchUp` and Argo CD does not.** With automated sync disabled Argo CD applies nothing on its own, so an `Application` that has not seen GitOps Reverser's own commits has nothing @@ -241,6 +243,11 @@ Applying a stale artifact reverts **published** edits, not only pending ones, wh `CatchUp` keeps the artifact at the tip, so any apply Flux starts on its own applies values the cluster already holds. +A worker's first observation is a baseline, never a `ForeignCommit`: nothing says what the applier +last applied, and asking for the tip on a restart, before the watches have captured the edits made +while the operator was down, would revert them. It owes a `CatchUp`, which waits for a quiet branch +like any other. + `quietPeriod` exists because a quiet branch is a moment, not a guarantee. An editor who saves twice a few seconds apart produces a quiet instant between the first push and the second write; a catch-up sent then is exactly the race of Part 2. Waiting `quietPeriod`, restarted by every write, keeps the @@ -351,7 +358,8 @@ URL normalization is the receiver's (§4.1): case-insensitive host and path, no ### 4.1 The receiver `GitProvider.spec.receiver` (`type`, `secretRef`), named after Flux's `Receiver`, served on its own -listener (`--receiver-bind-address`, Helm `receiver.bindAddress`, off by default). It cannot share +listener (`--receiver-bind-address`, Helm `receiver.bindAddress`, off by default), on the elected +replica only. It cannot share the audit listener, which requires a client certificate signed by the audit CA that no Git host can present. @@ -381,7 +389,8 @@ which branches are tracked. A route whose `GitProvider` has no `spec.receiver` d | `400` | Verified but not parseable, or a required field missing | | `401` | Signature absent or wrong | | `404` | No such route | -| `503` | Not the elected replica, or not ready | +| `413` | The body is over 25 MiB, GitHub's own limit | +| `503` | The route's token Secret, or its `token` key, cannot be read | A verified delivery that matches nothing is a success, because it is not the host's fault and an error teaches a host to disable the hook. A `generic-hmac` body naming a repository other than @@ -417,10 +426,10 @@ status: | Family | Labels | Reads as | | --- | --- | --- | | `gitopsreverser_apply_requests_total` | `applier_kind`, `reason`, `outcome` | Requests decided. `outcome`: `sent`, `failed`, `superseded` | -| `gitopsreverser_receiver_deliveries_total` | `type`, `outcome` | Deliveries. `outcome`: `foreign`, `own`, `ignored`, `rejected` | +| `gitopsreverser_receiver_deliveries_total` | `type`, `outcome` | Deliveries. `outcome`: `notified`, `unmatched`, `ignored`, `rejected`, `invalid`. Foreign or own is the worker's call, not the receiver's | | `gitopsreverser_git_fetches_total` | `reason` | Unchanged. A delivery that finds a foreign tip on an idle branch fetches under `refresh` | -These are recorded in [interpreting-metrics.md](../interpreting-metrics.md) when they ship. +Both are in [interpreting-metrics.md](../interpreting-metrics.md#apply-requests-and-the-push-receiver). ### 4.4 Chart @@ -457,9 +466,9 @@ superseding, and that no request leaves a branch with pending writes. 1. **An applier in another cluster.** Hub Argo CD keeps `Application`s in a management cluster. Supporting it means a second identity, the control cluster's or a kubeconfig, on `spec.applier`. Deferred until someone runs that topology. -2. **High availability.** The decision lives on the replica that owns the branch worker, so a - delivery reaching another replica answers `503` today. With multiple replicas, deliveries need - routing to the owner, or a shared queue in Redis. +2. **High availability.** The decision lives on the replica that owns the branch worker, so the + listener runs on the elected replica only and a standby refuses the connection. With multiple + replicas, deliveries need routing to the owner, or a shared queue in Redis. 3. **Path scoping for `ForeignCommit`.** Every applier on the branch is asked, whether or not the foreign commit touched its folder. That is safe, since the branch is quiet, but it runs every `Application`'s hooks. Narrowing it needs the applier's own source paths, which are not ours to @@ -467,9 +476,8 @@ superseding, and that no request leaves a branch with pending writes. ### 4.7 Build order -1. Worker state, the quiet-branch rule and `status.applier`, with the Flux client. This alone fixes - the original race for Flux users. -2. The receiver, on its own listener. -3. The Argo CD client, and the guide's Argo CD configuration. -4. `onRefusal: Reapply` in place of `PushEmptyCommit`, with the upgrade note. +1. Worker state, the quiet-branch rule and `status.applier`, with the Flux client. Built. +2. The receiver, on its own listener. Built. +3. `onRefusal: Reapply` in place of `PushEmptyCommit`, with the upgrade note. Built. +4. The Argo CD client, and the guide's Argo CD configuration. Next. 5. The webhook applier. diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index e7f36846..8407fdfd 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -201,10 +201,10 @@ boundary, the commit, the push. Background: | `git_pushes_total` | counter | `provider_namespace`, `provider_name`, `branch`, `outcome` | One per push cycle: `pushed` or `failed`. | | `git_push_retries_total` | counter | `provider_namespace`, `provider_name`, `branch`, `reason` | Replay rounds inside a cycle. `reason` is `remote_moved`. | | `git_push_duration_seconds` | histogram | `provider_namespace`, `provider_name`, `branch` | One cycle end to end, retries included. | -| `git_fetches_total` | counter | `provider_namespace`, `provider_name`, `branch`, `reason` | Every call that reads the remote through a `SmartFetch`. `reason` is `bootstrap` (repository preparation; **no production caller reaches it today**, so the series stays at zero) / `publication` (the head of a publication cycle) / `recovery` (re-establishing a base that could not be trusted, including a worktree a failed write left dirty that could not be undone locally) / `contention` (the reset onto the new tip after a push was rejected because somebody else moved the branch) / `push_failure_probe` (a push that failed without the remote saying anything, looking up where the branch is) / `forced_recheck` (a full re-read outside the publication cycle: a forced recheck, or the snapshot a resync judges against) / `refresh` (the periodic top-up of an idle branch, and only on the intervals where the branch had actually moved). See the note below. | +| `git_fetches_total` | counter | `provider_namespace`, `provider_name`, `branch`, `reason` | Every call that reads the remote through a `SmartFetch`. `reason` is `bootstrap` (repository preparation; **no production caller reaches it today**, so the series stays at zero) / `publication` (the head of a publication cycle) / `recovery` (re-establishing a base that could not be trusted, including a worktree a failed write left dirty that could not be undone locally) / `contention` (the reset onto the new tip after a push was rejected because somebody else moved the branch) / `push_failure_probe` (a push that failed without the remote saying anything, looking up where the branch is) / `forced_recheck` (a full re-read outside the publication cycle: a forced recheck, or the snapshot a resync judges against) / `refresh` (the periodic top-up of an idle branch, or a push receiver delivery on one, and only when the branch had actually moved). See the note below. | | `git_queue_refusals_total` | counter | `provider_namespace`, `provider_name`, `branch`, `kind` | Work refused at enqueue: the queue was full, intake was paused through an outage, or the worker was stopping. `kind` is `write` / `attach` / `resync` / `refresh`. The producer keeps a refused item and offers it again: the watch delivers a write again from its cursor, the controller re-sends a save, a resync is gathered again, and the next reconcile asks for a refresh. What it can cost, when its watch cursor expires before the write comes back, is the write's own commit and, under `prune.mode: OnEvent`, a deleted object's removal from Git. | | `git_commit_failures_total` | counter | `provider_namespace`, `provider_name`, `branch`, `kind`, `reason` | A window or request that died between routing and pushing. `kind` is `window` / `atomic`; `reason` is `refused` (a Git path a human must fix) / `error`. Every increment is a window's events lost until the next resync. An unreachable remote does not count: a decided window waits for the publication retry instead. | -| `git_commit_windows_total` | counter | `gittarget_namespace`, `gittarget_name`, `close_reason`, `timer_source` | One per closed commit window, whether or not it then produced a commit. `close_reason` is `idle_timeout` / `max_duration` / `attach_next` / `identity_change` / `buffer_limit` / `resync_before_apply` / `atomic_before_apply` / `shutdown`; `timer_source` is `target` or `commit_request`. See [tuning commit windows](#tuning-commit-windows). | +| `git_commit_windows_total` | counter | `gittarget_namespace`, `gittarget_name`, `close_reason`, `timer_source` | One per closed commit window, whether or not it then produced a commit. `close_reason` is `idle_timeout` / `max_duration` / `attach_next` / `identity_change` / `buffer_limit` / `resync_before_apply` / `atomic_before_apply` / `foreign_commit` (the branch moved, or a Git host said it may have, so the window is published now) / `shutdown`; `timer_source` is `target` or `commit_request`. See [tuning commit windows](#tuning-commit-windows). | | `git_commit_window_duration_seconds` | histogram | `gittarget_namespace`, `gittarget_name` | How long each window collected: from the write that opened it to the start of its finalize. Its count equals `git_commit_windows_total` summed over `close_reason` and `timer_source`. | | `git_queue_depth` | gauge | `provider_namespace`, `provider_name`, `branch` | Pending + in-flight + committed-but-unpushed. Read at scrape time. | | `git_retained_bytes` | gauge | `provider_namespace`, `provider_name`, `branch` | What the worker holds until a push lands, in the units of the retained-byte budget: the open window, the retained writes, deferred snapshots and waiting saves, each with a 1 KiB per-item charge. Serialized bytes, not heap. | @@ -814,6 +814,29 @@ are unbounded, and both are in the log line at the write site. --- +## Apply requests and the push receiver + +With `GitTarget.spec.applier` set, the branch worker decides when the Flux `Kustomization` applies, +and the push receiver tells it that somebody else pushed. See +[who applies this folder](configuration.md#who-applies-this-folder-specapplier). + +| Metric | Type | Labels | Meaning | +| --- | --- | --- | --- | +| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per applier per request the branch worker decided to send. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (attempts exhausted, or the `Kustomization` cannot take a request) or `superseded` (a newer request for the same `Kustomization` replaced it before it went out) | +| `receiver_deliveries_total` | counter | `type`, `outcome` | One per verified-or-refused delivery to `/git-push/`. `type` is the `GitProvider.spec.receiver.type`; `outcome` is `notified` (handed to the branch worker), `unmatched` (no worker for that branch), `ignored` (a ping, a tag, another branch's event type, or a `generic-hmac` body naming another repository), `rejected` (bad signature or an oversized body) or `invalid` (verified but unparseable) | + +A delivery counted `notified` is only a hint: the worker reads the advertisement, and fetches under +`git_fetches_total{reason="refresh"}` only when the branch actually moved. A `ForeignCommit` request +follows when it did. So on a branch where only GitOps Reverser pushes, `notified` deliveries are +followed by `CatchUp` requests and never by `ForeignCommit`. + +```promql +# Requests that could not be delivered, by reason. +sum by (reason) (rate(gitopsreverser_apply_requests_total{outcome="failed"}[15m])) +# A receiver a Git host is calling with the wrong secret. +sum by (type) (rate(gitopsreverser_receiver_deliveries_total{outcome="rejected"}[15m])) > 0 +``` + ## Audit attribution (optional) Audit runs when `--author-attribution` is on. The kube-apiserver POSTs audit `EventList` payloads to diff --git a/hack/crdfields/fields.yaml b/hack/crdfields/fields.yaml index 7aaa7344..0a53d785 100644 --- a/hack/crdfields/fields.yaml +++ b/hack/crdfields/fields.yaml @@ -50,6 +50,11 @@ kinds: - path: commit.signing default: "off" doc: SSH commit signing. See [commit signing](#commit-signing) + - path: receiver + default: "off" + doc: >- + Push webhook endpoint, so the Git host can say somebody pushed. See + [receiving push notifications](#receiving-push-notifications-specreceiver) - kind: ClusterProvider title: "`ClusterProvider` (cluster-scoped): the source cluster" @@ -132,6 +137,12 @@ kinds: doc: >- Which deletions reach Git. See [deletion policy](#deletion-policy-specprunemode) + - path: applier + default: Flux applies on its own schedule + doc: >- + The Flux `Kustomization` that applies this folder (`kustomizationRef`), which GitOps + Reverser then asks to apply, and how long the branch stays quiet before it catches up + (`quietPeriod`, `30s`). See [who applies this folder](#who-applies-this-folder-specapplier) - path: onRefusal doc: >- What happens to an edit with no legal destination. See diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go new file mode 100644 index 00000000..dd859867 --- /dev/null +++ b/internal/applier/dispatcher.go @@ -0,0 +1,546 @@ +// SPDX-License-Identifier: Apache-2.0 + +package applier + +import ( + "context" + "fmt" + "slices" + "sort" + "sync" + "time" + + "github.com/go-logr/logr" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/tools/record" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/manager" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/git" + "github.com/ConfigButler/gitops-reverser/internal/telemetry" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// Event reasons recorded on a GitTarget. +const ( + // EventReasonApplyRequested is a request that reached the applier. + EventReasonApplyRequested = "ApplyRequested" + // EventReasonApplyRequestFailed is a request given up on after its last attempt. + EventReasonApplyRequestFailed = "ApplyRequestFailed" +) + +// apply_requests_total label values. +const ( + applierKindKustomization = "kustomization" + + outcomeSent = "sent" + outcomeFailed = "failed" + outcomeSuperseded = "superseded" +) + +// maxInbox bounds the requests waiting to be resolved. The loop drains the inbox the moment it is +// woken and never waits on an applier while doing so (delivery runs on its own goroutines), so the +// inbox only fills when the loop is wedged, which is a bug; the bound keeps that bug from also +// being a leak. +const maxInbox = 1024 + +// shortCommitLength is how much of a commit an Event names. +const shortCommitLength = 8 + +// SourceClients resolves a source cluster's dynamic client by ClusterProvider name +// (GitTarget.SourceCluster()). In production it is (*watch.Manager).SourceDynamicClient. +type SourceClients func(ctx context.Context, clusterID string) (dynamic.Interface, error) + +// Backoff is the retry schedule for one apply request: Initial after the first failure, doubling, +// capped at Max, and given up on after Attempts attempts in all. +type Backoff struct { + Initial time.Duration + Max time.Duration + Attempts int +} + +// DefaultBackoff gives an applier about four minutes to answer before a request is given up on. +// That is long enough to ride out a source-cluster API server restart, and short enough that a +// misconfigured applier says so (ApplyRequestFailed) while somebody is still looking. Nothing is +// lost by giving up: the next request for the applier carries a commit at least as new. +func DefaultBackoff() Backoff { + return Backoff{Initial: defaultBackoffInitial, Max: time.Minute, Attempts: defaultBackoffAttempts} +} + +const ( + defaultBackoffInitial = 2 * time.Second + defaultBackoffAttempts = 8 +) + +// delay is the wait after the given number of consecutive failures (at least one). +func (b Backoff) delay(failures int) time.Duration { + d := b.Initial + for i := 1; i < failures && d < b.Max; i++ { + d *= 2 + } + return min(d, b.Max) +} + +// DefaultAttemptTimeout bounds one attempt: a source cluster that does not answer fails the +// attempt, and the backoff decides when to try again. +const DefaultAttemptTimeout = 30 * time.Second + +// Dispatcher delivers the branch workers' apply requests to the appliers they name. It implements +// git.ApplyRequester, and runs as a leader-elected manager.Runnable, the same replica the branch +// workers run on. +// +// A request names GitTargets; the dispatcher resolves each to its applier and asks every applier +// once, however many targets share it. Each applier is delivered to independently, on its own +// goroutine and with its own retries, so one that does not answer never holds another. +// +// A newer request for an applier supersedes one not yet sent, because the newer one names a commit +// at least as new and was decided on a quiet branch too: it is always safe to apply in place of the +// older. A CatchUp never weakens what it supersedes, though. A ForeignCommit or Refusal stamps the +// Kustomization when the artifact already holds the commit, and a CatchUp does not, so a CatchUp +// arriving behind a Refusal keeps the Refusal's reason with the newer commit; otherwise the revert +// the Refusal asked for would be lost whenever the artifact had already caught up. +type Dispatcher struct { + reader client.Reader + sourceClients SourceClients + recorder record.EventRecorder + log logr.Logger + + // Backoff is the retry schedule. Set before Start. + Backoff Backoff + // AttemptTimeout bounds one delivery attempt. Set before Start. + AttemptTimeout time.Duration + // Now is the clock requestedAt stamps and the ledger read. Retries are scheduled on real time. + Now func() time.Time + + inboxMu sync.Mutex + inbox []git.ApplyRequest + wake chan struct{} + + ledgerMu sync.RWMutex + ledger map[string]v1alpha3.GitTargetApplierStatus + + // Owned by the Start loop. + pending map[applierKey]*job + inflight map[applierKey]*job + results chan result +} + +var ( + _ git.ApplyRequester = (*Dispatcher)(nil) + _ manager.Runnable = (*Dispatcher)(nil) + _ manager.LeaderElectionRunnable = (*Dispatcher)(nil) +) + +// NewDispatcher returns a Dispatcher that reads GitTargets through reader (the control cluster), +// reaches appliers through sourceClients, and records Events through recorder (nil records none). +func NewDispatcher( + reader client.Reader, + sourceClients SourceClients, + recorder record.EventRecorder, + log logr.Logger, +) *Dispatcher { + return &Dispatcher{ + reader: reader, + sourceClients: sourceClients, + recorder: recorder, + log: log, + Backoff: DefaultBackoff(), + AttemptTimeout: DefaultAttemptTimeout, + Now: time.Now, + wake: make(chan struct{}, 1), + ledger: map[string]v1alpha3.GitTargetApplierStatus{}, + pending: map[applierKey]*job{}, + inflight: map[applierKey]*job{}, + results: make(chan result), + } +} + +// NeedLeaderElection is true: the branch workers that decide requests run only on the leader. +func (d *Dispatcher) NeedLeaderElection() bool { return true } + +// RequestApply queues a request and returns at once; it is called on a branch worker's loop. +// +// A queued request for exactly the same branch, reason and targets is replaced in place rather +// than queued twice: the newer one names the newer commit, and both would resolve to the same +// appliers. Past maxInbox the oldest request is dropped, and said so in the log. +func (d *Dispatcher) RequestApply(req git.ApplyRequest) { + if len(req.Targets) == 0 { + return + } + req.Targets = slices.Clone(req.Targets) + + d.inboxMu.Lock() + replaced := false + for i := range d.inbox { + if sameScope(d.inbox[i], req) { + d.inbox[i] = req + replaced = true + break + } + } + var dropped *git.ApplyRequest + if !replaced { + if len(d.inbox) >= maxInbox { + oldest := d.inbox[0] + dropped = &oldest + d.inbox = d.inbox[1:] + } + d.inbox = append(d.inbox, req) + } + d.inboxMu.Unlock() + + if dropped != nil { + d.log.Error(nil, "apply request inbox full; dropped the oldest request", + "branch", dropped.Branch.String(), "reason", dropped.Reason, "commit", dropped.Commit) + } + select { + case d.wake <- struct{}{}: + default: + } +} + +// sameScope reports whether two requests are for the same branch, reason and set of targets. +func sameScope(a, b git.ApplyRequest) bool { + if a.Branch != b.Branch || a.Reason != b.Reason || len(a.Targets) != len(b.Targets) { + return false + } + keys := func(refs []itypes.ResourceReference) []string { + out := make([]string, len(refs)) + for i, ref := range refs { + out[i] = ref.Key() + } + sort.Strings(out) + return out + } + return slices.Equal(keys(a.Targets), keys(b.Targets)) +} + +// LastRequest is the last apply request delivered for a GitTarget's applier, for status.applier. +// It is false until one was delivered. A CatchUp that found the applier already current is not a +// request made, and leaves it unchanged. +func (d *Dispatcher) LastRequest(ref itypes.ResourceReference) (v1alpha3.GitTargetApplierStatus, bool) { + d.ledgerMu.RLock() + defer d.ledgerMu.RUnlock() + status, ok := d.ledger[ref.Key()] + if !ok { + return v1alpha3.GitTargetApplierStatus{}, false + } + return *status.DeepCopy(), true +} + +// applierKey identifies one applier for one branch. The branch is part of it so that two branches +// naming one Kustomization (a misconfiguration ApplierConfigured reports as SourceMismatch) never +// have one branch's commit sent in place of the other's. +type applierKey struct { + branch git.BranchKey + cluster string + namespace string + name string +} + +func (k applierKey) ref() types.NamespacedName { + return types.NamespacedName{Namespace: k.namespace, Name: k.name} +} + +// job is one applier's share of one or more requests. +type job struct { + key applierKey + commit string + reason v1alpha3.ApplyReason + targets map[string]*v1alpha3.GitTarget + failures int + due time.Time +} + +// absorb takes over an older job this one supersedes: its targets, and its reason when this one +// would weaken it. +func (j *job) absorb(older *job) { + j.reason = supersedingReason(older.reason, j.reason) + for key, target := range older.targets { + if _, ok := j.targets[key]; !ok { + j.targets[key] = target + } + } +} + +// supersedingReason is the reason a newer request carries when it replaces an older one: its own, +// except that a CatchUp never replaces a reason that asks for more (see Dispatcher). +func supersedingReason(older, newer v1alpha3.ApplyReason) v1alpha3.ApplyReason { + if newer == v1alpha3.ApplyReasonCatchUp && older != v1alpha3.ApplyReasonCatchUp { + return older + } + return newer +} + +// sortedTargets returns the job's GitTargets in a stable order. +func (j *job) sortedTargets() []*v1alpha3.GitTarget { + keys := make([]string, 0, len(j.targets)) + for key := range j.targets { + keys = append(keys, key) + } + sort.Strings(keys) + out := make([]*v1alpha3.GitTarget, 0, len(keys)) + for _, key := range keys { + out = append(out, j.targets[key]) + } + return out +} + +type result struct { + job *job + outcome Outcome + err error +} + +// Start runs the delivery loop until ctx ends. Attempts still running then are cancelled with it. +func (d *Dispatcher) Start(ctx context.Context) error { + d.log.Info("apply request dispatcher started") + var attempts sync.WaitGroup + defer attempts.Wait() + + for { + d.takeInbox(ctx) + d.startDue(ctx, &attempts) + + var timer *time.Timer + var fire <-chan time.Time + if next, ok := d.nextDue(); ok { + timer = time.NewTimer(time.Until(next)) + fire = timer.C + } + select { + case <-ctx.Done(): + if timer != nil { + timer.Stop() + } + return nil + case <-d.wake: + case res := <-d.results: + d.settle(ctx, res) + case <-fire: + } + if timer != nil { + timer.Stop() + } + } +} + +// takeInbox resolves every queued request into per-applier jobs and queues them. +func (d *Dispatcher) takeInbox(ctx context.Context) { + d.inboxMu.Lock() + requests := d.inbox + d.inbox = nil + d.inboxMu.Unlock() + + for _, req := range requests { + for _, j := range d.resolve(ctx, req) { + d.enqueue(ctx, j) + } + } +} + +// resolve turns a request into one job per applier its targets name. A target that is gone, or +// declares no applier, is skipped: the worker decided on the targets it knew, and either can have +// changed since. The read is the manager's cache, so it costs the loop no round trip. +func (d *Dispatcher) resolve(ctx context.Context, req git.ApplyRequest) []*job { + jobs := map[applierKey]*job{} + var order []applierKey + for _, ref := range req.Targets { + target := &v1alpha3.GitTarget{} + if err := d.reader.Get(ctx, client.ObjectKey{Namespace: ref.Namespace, Name: ref.Name}, target); err != nil { + if !apierrors.IsNotFound(err) { + d.log.Error(err, "cannot read GitTarget for an apply request; skipping it", + "gitTarget", ref.String(), "reason", req.Reason, "commit", req.Commit) + } + continue + } + if target.Spec.Applier == nil { + continue + } + kustomization := target.Spec.Applier.KustomizationRef + namespace := kustomization.Namespace + if namespace == "" { + namespace = target.Namespace + } + key := applierKey{ + branch: req.Branch, + cluster: target.SourceCluster(), + namespace: namespace, + name: kustomization.Name, + } + j := jobs[key] + if j == nil { + j = &job{key: key, commit: req.Commit, reason: req.Reason, targets: map[string]*v1alpha3.GitTarget{}} + jobs[key] = j + order = append(order, key) + } + j.targets[itypes.NewResourceReference(target.Name, target.Namespace).Key()] = target + } + out := make([]*job, 0, len(order)) + for _, key := range order { + out = append(out, jobs[key]) + } + return out +} + +// enqueue queues a job, superseding the applier's job that is waiting to be sent, if any. A job +// already being attempted is not superseded: it was sent. The new job waits behind it. +func (d *Dispatcher) enqueue(ctx context.Context, j *job) { + if older, ok := d.pending[j.key]; ok { + d.count(ctx, older.reason, outcomeSuperseded) + j.absorb(older) + } + d.pending[j.key] = j +} + +// startDue starts an attempt for every job that is due and whose applier has none running. +func (d *Dispatcher) startDue(ctx context.Context, attempts *sync.WaitGroup) { + now := time.Now() + for key, j := range d.pending { + if _, busy := d.inflight[key]; busy || j.due.After(now) { + continue + } + delete(d.pending, key) + d.inflight[key] = j + attempts.Add(1) + go func() { + defer attempts.Done() + outcome, err := d.attempt(ctx, j) + select { + case d.results <- result{job: j, outcome: outcome, err: err}: + case <-ctx.Done(): + } + }() + } +} + +// nextDue is the earliest time a waiting job becomes due, among those that can start then. +func (d *Dispatcher) nextDue() (time.Time, bool) { + var next time.Time + found := false + for key, j := range d.pending { + if _, busy := d.inflight[key]; busy { + continue + } + if !found || j.due.Before(next) { + next, found = j.due, true + } + } + return next, found +} + +// attempt makes one delivery attempt. +func (d *Dispatcher) attempt(ctx context.Context, j *job) (Outcome, error) { + ctx, cancel := context.WithTimeout(ctx, d.AttemptTimeout) + defer cancel() + sourceClient, err := d.sourceClients(ctx, j.key.cluster) + if err != nil { + return "", fmt.Errorf("source cluster %q: %w", j.key.cluster, err) + } + flux := NewFlux(sourceClient) + flux.Now = d.Now + return flux.Request(ctx, j.key.ref(), j.commit, j.reason) +} + +// settle records the result of an attempt: delivered, superseded by a request that arrived while +// it ran, retried later, or given up on. +func (d *Dispatcher) settle(ctx context.Context, res result) { + j := res.job + delete(d.inflight, j.key) + log := d.log.WithValues("applier", j.key.ref().String(), "cluster", j.key.cluster, + "branch", j.key.branch.String(), "reason", j.reason, "commit", j.commit) + + if res.err == nil { + d.count(ctx, j.reason, outcomeSent) + if res.outcome == OutcomeCurrent { + log.V(1).Info("applier already holds the commit; nothing to request") + return + } + log.Info("apply requested", "outcome", res.outcome) + d.delivered(j) + return + } + + j.failures++ + if newer, ok := d.pending[j.key]; ok { + // A newer request is waiting, so this one is not retried: the newer one is what to send. + d.count(ctx, j.reason, outcomeSuperseded) + newer.absorb(j) + log.Info("apply request failed and is superseded by a newer one", "error", res.err.Error()) + return + } + if isPermanent(res.err) || j.failures >= d.Backoff.Attempts { + d.count(ctx, j.reason, outcomeFailed) + log.Error(res.err, "apply request failed; giving up", "attempts", j.failures) + d.failed(j, res.err) + return + } + delay := d.Backoff.delay(j.failures) + j.due = time.Now().Add(delay) + d.pending[j.key] = j + log.Info("apply request failed; retrying", "attempt", j.failures, "retryIn", delay, "error", res.err.Error()) +} + +// delivered records a delivered request in the ledger and as an ApplyRequested Event on every +// GitTarget it was for. +func (d *Dispatcher) delivered(j *job) { + at := metav1.NewTime(d.Now()) + status := v1alpha3.GitTargetApplierStatus{RequestedCommit: j.commit, RequestedAt: &at, Reason: j.reason} + targets := j.sortedTargets() + + d.ledgerMu.Lock() + for _, target := range targets { + d.ledger[itypes.NewResourceReference(target.Name, target.Namespace).Key()] = *status.DeepCopy() + } + d.ledgerMu.Unlock() + + if d.recorder == nil { + return + } + for _, target := range targets { + d.recorder.Eventf(target, corev1.EventTypeNormal, EventReasonApplyRequested, + "asked %s to apply %s (%s)", j.key.ref(), describeCommit(j.commit), j.reason) + } +} + +// failed records a request given up on as an ApplyRequestFailed Event on every GitTarget it was for. +func (d *Dispatcher) failed(j *job, err error) { + if d.recorder == nil { + return + } + for _, target := range j.sortedTargets() { + d.recorder.Eventf(target, corev1.EventTypeWarning, EventReasonApplyRequestFailed, + "could not ask %s to apply %s (%s) after %d attempts: %v", + j.key.ref(), describeCommit(j.commit), j.reason, j.failures, err) + } +} + +// describeCommit names a commit in an Event: its short hash, or what an empty one means. +func describeCommit(commit string) string { + if commit == "" { + return "the branch, which the remote does not carry" + } + if len(commit) > shortCommitLength { + return commit[:shortCommitLength] + } + return commit +} + +// count records one apply_requests_total decision. +func (d *Dispatcher) count(ctx context.Context, reason v1alpha3.ApplyReason, outcome string) { + if telemetry.ApplyRequestsTotal == nil { + return + } + telemetry.ApplyRequestsTotal.Add(ctx, 1, metric.WithAttributes( + attribute.String("applier_kind", applierKindKustomization), + attribute.String("reason", string(reason)), + attribute.String("outcome", outcome), + )) +} diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go new file mode 100644 index 00000000..417b7999 --- /dev/null +++ b/internal/applier/dispatcher_test.go @@ -0,0 +1,397 @@ +// SPDX-License-Identifier: Apache-2.0 + +package applier + +import ( + "context" + "fmt" + "strings" + "sync" + "testing" + "time" + + fluxmeta "github.com/fluxcd/pkg/apis/meta" + "github.com/go-logr/logr" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/dynamic" + dynamicfake "k8s.io/client-go/dynamic/fake" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/tools/record" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/git" + "github.com/ConfigButler/gitops-reverser/internal/telemetry" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +const ( + applyRequestsMetric = "gitopsreverser_apply_requests_total" + targetNS = "team-a" +) + +var testBranchKey = git.BranchKey{RepoNamespace: targetNS, RepoName: "platform", Branch: testBranch} + +// gitTarget builds a GitTarget; a nil kustomizationRef declares no applier. +func gitTarget(name string, kustomizationRef *fluxmeta.NamespacedObjectReference, cluster string) *v1alpha3.GitTarget { + target := &v1alpha3.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Namespace: targetNS, Name: name}, + Spec: v1alpha3.GitTargetSpec{Branch: testBranch, Path: name}, + } + if kustomizationRef != nil { + target.Spec.Applier = &v1alpha3.GitTargetApplier{KustomizationRef: *kustomizationRef} + } + if cluster != "" { + target.Spec.ClusterProviderRef = &fluxmeta.LocalObjectReference{Name: cluster} + } + return target +} + +func ksRef(namespace, name string) *fluxmeta.NamespacedObjectReference { + return &fluxmeta.NamespacedObjectReference{Namespace: namespace, Name: name} +} + +func ref(name string) itypes.ResourceReference { return itypes.NewResourceReference(name, targetNS) } + +// sourceStub hands out one fake source cluster's client, failing the first failFirst calls (or +// every call, with failFirst < 0), and remembers which cluster ids it was asked for. +type sourceStub struct { + client *dynamicfake.FakeDynamicClient + mu sync.Mutex + failFirst int + calls int + clusters []string +} + +func (s *sourceStub) get(_ context.Context, clusterID string) (dynamic.Interface, error) { + s.mu.Lock() + defer s.mu.Unlock() + s.calls++ + s.clusters = append(s.clusters, clusterID) + if s.failFirst < 0 || s.calls <= s.failFirst { + return nil, fmt.Errorf("source cluster unreachable (call %d)", s.calls) + } + return s.client, nil +} + +func (s *sourceStub) callCount() int { + s.mu.Lock() + defer s.mu.Unlock() + return s.calls +} + +type dispatcherFixture struct { + dispatcher *Dispatcher + source *sourceStub + recorder *record.FakeRecorder + reader *sdkmetric.ManualReader +} + +func newDispatcherFixture(t *testing.T, targets []client.Object, sourceObjs ...runtime.Object) *dispatcherFixture { + t.Helper() + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + scheme := runtime.NewScheme() + require.NoError(t, clientgoscheme.AddToScheme(scheme)) + require.NoError(t, v1alpha3.AddToScheme(scheme)) + controlClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(targets...).Build() + + source := &sourceStub{client: newSourceClient(sourceObjs...)} + recorder := record.NewFakeRecorder(64) + d := NewDispatcher(controlClient, source.get, recorder, logr.Discard()) + d.Backoff = Backoff{Initial: time.Millisecond, Max: 4 * time.Millisecond, Attempts: 4} + d.Now = func() time.Time { return stampTime } + return &dispatcherFixture{dispatcher: d, source: source, recorder: recorder, reader: reader} +} + +// start runs the dispatcher until the test ends. +func (f *dispatcherFixture) start(t *testing.T) { + t.Helper() + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { + defer close(done) + _ = f.dispatcher.Start(ctx) + }() + t.Cleanup(func() { + cancel() + <-done + }) +} + +func (f *dispatcherFixture) count(reason v1alpha3.ApplyReason, outcome string) int64 { + value, _ := telemetry.CollectInt64Sum(f.reader, applyRequestsMetric, map[string]string{ + "applier_kind": applierKindKustomization, "reason": string(reason), "outcome": outcome, + }) + return value +} + +// waitForCount waits until the counter reaches want. +func (f *dispatcherFixture) waitForCount(t *testing.T, reason v1alpha3.ApplyReason, outcome string, want int64) { + t.Helper() + require.Eventually(t, func() bool { return f.count(reason, outcome) == want }, + 5*time.Second, time.Millisecond, "apply_requests_total{reason=%s,outcome=%s} never reached %d (at %d)", + reason, outcome, want, f.count(reason, outcome)) +} + +// events drains the recorded Events. +func (f *dispatcherFixture) events() []string { + var out []string + for { + select { + case event := <-f.recorder.Events: + out = append(out, event) + default: + return out + } + } +} + +func (f *dispatcherFixture) patches() []string { + var out []string + for _, action := range f.source.client.Actions() { + if action.GetVerb() == "patch" { + out = append(out, action.GetResource().Resource+" "+action.GetNamespace()) + } + } + return out +} + +// TestDispatcher_AsksEachApplierOnce: targets sharing an applier ask it once; a target with no +// applier, or one that is gone, asks nothing; an omitted kustomizationRef namespace is the target's. +func TestDispatcher_AsksEachApplierOnce(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{ + gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""), + gitTarget("b", ksRef(fluxSystemNS, "checkout"), ""), + gitTarget("c", ksRef("", "own"), "prod-eu-1"), + gitTarget("no-applier", nil, ""), + }, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit), + kustomizationObj(targetNS, "own", gitRepoSourceRef("own")), + gitRepositoryObj(targetNS, "own", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit), + ) + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a"), ref("b"), ref("c"), ref("no-applier"), ref("deleted")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeSent, 2) + + assert.ElementsMatch(t, []string{"gitrepositories " + fluxSystemNS, "gitrepositories " + targetNS}, f.patches(), + "one source stamp per applier, however many targets share it") + assert.ElementsMatch(t, []string{v1alpha3.DefaultClusterProviderName, "prod-eu-1"}, f.source.clusters, + "each applier is looked up in its target's source cluster") + + want := v1alpha3.GitTargetApplierStatus{ + RequestedCommit: testCommit, + RequestedAt: &metav1.Time{Time: stampTime}, + Reason: v1alpha3.ApplyReasonForeignCommit, + } + for _, name := range []string{"a", "b", "c"} { + got, ok := f.dispatcher.LastRequest(ref(name)) + require.True(t, ok, name) + assert.True(t, got.RequestedAt.Equal(want.RequestedAt), name) + assert.Equal(t, want.RequestedCommit, got.RequestedCommit, name) + assert.Equal(t, want.Reason, got.Reason, name) + } + _, ok := f.dispatcher.LastRequest(ref("no-applier")) + assert.False(t, ok, "a target without an applier records no request") + + assert.ElementsMatch(t, []string{ + "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)", + "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)", + "Normal ApplyRequested asked team-a/own to apply 4f2c1ab9 (ForeignCommit)", + }, f.events()) +} + +// TestDispatcher_RetriesThenDelivers: a failed attempt is retried on the backoff, and only the +// delivery counts. +func TestDispatcher_RetriesThenDelivers(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + ) + f.source.failFirst = 2 + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonRefusal, outcomeSent, 1) + + assert.Equal(t, 3, f.source.callCount(), "two failures, then the attempt that landed") + assert.Equal(t, int64(0), f.count(v1alpha3.ApplyReasonRefusal, outcomeFailed)) + assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches(), + "a refusal at the artifact's commit stamps the kustomization") + got, ok := f.dispatcher.LastRequest(ref("a")) + require.True(t, ok) + assert.Equal(t, v1alpha3.ApplyReasonRefusal, got.Reason) +} + +// TestDispatcher_GivesUp: a request that keeps failing is given up on after its last attempt, with a +// Warning Event, and is not recorded as made. +func TestDispatcher_GivesUp(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}) + f.source.failFirst = -1 + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) + + assert.Equal(t, f.dispatcher.Backoff.Attempts, f.source.callCount()) + _, ok := f.dispatcher.LastRequest(ref("a")) + assert.False(t, ok) + events := f.events() + require.Len(t, events, 1) + const want = "Warning ApplyRequestFailed could not ask flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)" + + " after 4 attempts" + assert.True(t, strings.HasPrefix(events[0], want), events[0]) +} + +// TestDispatcher_PermanentFailureIsNotRetried: a Kustomization a request cannot work with fails +// once rather than retrying to the limit. +func TestDispatcher_PermanentFailureIsNotRetried(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + kustomizationObj(fluxSystemNS, "checkout", map[string]any{"kind": "Bucket", "name": "platform"}), + ) + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) + assert.Equal(t, 1, f.source.callCount()) +} + +// TestDispatcher_NewerRequestSupersedes: a request waiting on its retry is replaced by a newer one +// for the same applier, which is sent at once with the newer commit. The newer one is a CatchUp +// and the older a Refusal, so it keeps the Refusal's reason: the artifact already holds the newer +// commit, and a CatchUp would then stamp nothing and drop the revert. +func TestDispatcher_NewerRequestSupersedes(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + ) + f.dispatcher.Backoff = Backoff{Initial: time.Hour, Max: time.Hour, Attempts: 4} + f.source.failFirst = 1 + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: olderCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + require.Eventually(t, func() bool { return f.source.callCount() == 1 }, 5*time.Second, time.Millisecond) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonCatchUp, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonRefusal, outcomeSent, 1) + + assert.Equal(t, int64(1), f.count(v1alpha3.ApplyReasonRefusal, outcomeSuperseded), + "the failed Refusal at the older commit was superseded, not retried") + assert.Equal(t, int64(0), f.count(v1alpha3.ApplyReasonCatchUp, outcomeSent)) + assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches()) + got, ok := f.dispatcher.LastRequest(ref("a")) + require.True(t, ok) + assert.Equal(t, testCommit, got.RequestedCommit, "the newer commit is the one sent") + assert.Equal(t, v1alpha3.ApplyReasonRefusal, got.Reason) +} + +// TestDispatcher_CatchUpAlreadyCurrent: a CatchUp the applier already satisfies counts as sent and +// records nothing, since nothing was asked of it. +func TestDispatcher_CatchUpAlreadyCurrent(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + ) + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonCatchUp, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonCatchUp, outcomeSent, 1) + + assert.Empty(t, f.patches()) + assert.Empty(t, f.events()) + _, ok := f.dispatcher.LastRequest(ref("a")) + assert.False(t, ok) +} + +// TestDispatcher_RequestApplyNeverBlocks: with no loop running, requests queue without blocking; an +// identical scope is replaced in place, and past the bound the oldest is dropped. +func TestDispatcher_RequestApplyNeverBlocks(t *testing.T) { + f := newDispatcherFixture(t, nil) + d := f.dispatcher + + same := func(commit string) git.ApplyRequest { + return git.ApplyRequest{Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: commit, + Targets: []itypes.ResourceReference{ref("a"), ref("b")}} + } + d.RequestApply(same(olderCommit)) + reordered := same(testCommit) + reordered.Targets = []itypes.ResourceReference{ref("b"), ref("a")} + d.RequestApply(reordered) + d.RequestApply(git.ApplyRequest{Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit}) + + d.inboxMu.Lock() + require.Len(t, d.inbox, 1, "an identical scope is replaced, and an empty request is ignored") + assert.Equal(t, testCommit, d.inbox[0].Commit) + d.inboxMu.Unlock() + + for i := range maxInbox + 10 { + d.RequestApply(git.ApplyRequest{Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, + Commit: testCommit, Targets: []itypes.ResourceReference{ref(fmt.Sprintf("t%d", i))}}) + } + d.inboxMu.Lock() + defer d.inboxMu.Unlock() + assert.Len(t, d.inbox, maxInbox) + assert.Equal(t, "t10", d.inbox[0].Targets[0].Name, "the oldest requests were dropped") +} + +func TestBackoff_Delay(t *testing.T) { + b := Backoff{Initial: 2 * time.Second, Max: time.Minute, Attempts: 8} + for failures, want := range map[int]time.Duration{ + 1: 2 * time.Second, 2: 4 * time.Second, 3: 8 * time.Second, 5: 32 * time.Second, 6: time.Minute, 20: time.Minute, + } { + assert.Equal(t, want, b.delay(failures), "after %d failures", failures) + } +} + +func TestSupersedingReason(t *testing.T) { + tests := []struct{ older, newer, want v1alpha3.ApplyReason }{ + {v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonRefusal}, + {v1alpha3.ApplyReasonForeignCommit, v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonForeignCommit}, + {v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonCatchUp}, + {v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonRefusal}, + {v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonForeignCommit, v1alpha3.ApplyReasonForeignCommit}, + } + for _, tt := range tests { + assert.Equal(t, tt.want, supersedingReason(tt.older, tt.newer), "%s then %s", tt.older, tt.newer) + } +} + +func TestDescribeCommit(t *testing.T) { + assert.Equal(t, "4f2c1ab9", describeCommit(testCommit)) + assert.Equal(t, "abc", describeCommit("abc")) + assert.Equal(t, "the branch, which the remote does not carry", describeCommit("")) +} diff --git a/internal/applier/flux.go b/internal/applier/flux.go new file mode 100644 index 00000000..c1e7cb91 --- /dev/null +++ b/internal/applier/flux.go @@ -0,0 +1,433 @@ +// SPDX-License-Identifier: Apache-2.0 + +/* +Package applier delivers apply requests: what GitOps Reverser sends the object that applies a +GitTarget's folder back to its source cluster (its applier) when the branch worker has decided a +commit is safe to apply. The worker decides WHEN; this package decides HOW, off the worker's queue, +so a slow or unreachable applier never holds a branch. + +Only Flux is served: spec.applier.kustomizationRef, a Kustomization whose sourceRef is a +GitRepository. See docs/design/push-notification-and-reconcile-trigger.md §3.6 and §3.8. + +A request asks and never reports. Nothing here claims an apply happened; the applier's own status +(lastAppliedRevision on a Kustomization) is the record of that. +*/ +package applier + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "strings" + "time" + + fluxmeta "github.com/fluxcd/pkg/apis/meta" + authorizationv1 "k8s.io/api/authorization/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/repourl" +) + +// The Flux kinds are read through the dynamic client at their served v1 versions, rather than +// through Flux's typed APIs: importing source-controller and kustomize-controller for two GETs and +// a PATCH would tie this module's dependency graph to theirs, and a cluster without Flux installed +// must cost nothing (§3.8), which a typed scheme registration would not guarantee. + +// kustomizationGVR is Flux's Kustomization. +func kustomizationGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: "kustomize.toolkit.fluxcd.io", Version: "v1", Resource: "kustomizations"} +} + +// gitRepositoryGVR is Flux's GitRepository. +func gitRepositoryGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: "source.toolkit.fluxcd.io", Version: "v1", Resource: "gitrepositories"} +} + +// selfSubjectAccessReviewGVR is the access check Validate creates. +func selfSubjectAccessReviewGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{ + Group: "authorization.k8s.io", Version: "v1", Resource: "selfsubjectaccessreviews", + } +} + +const ( + kindKustomization = "Kustomization" + kindGitRepository = "GitRepository" +) + +// Outcome is what one Flux request did. +type Outcome string + +const ( + // OutcomeStampedSource stamped the GitRepository: its artifact did not hold the commit, and the + // new artifact it produces wakes the Kustomization by itself. + OutcomeStampedSource Outcome = "StampedSource" + // OutcomeStampedKustomization stamped the Kustomization: the artifact already held the commit, + // so a source stamp would produce no new artifact and wake nothing. + OutcomeStampedKustomization Outcome = "StampedKustomization" + // OutcomeCurrent stamped nothing: a CatchUp found the artifact already at the commit, which is + // all a CatchUp asks for. + OutcomeCurrent Outcome = "Current" +) + +// ApplierConfigured reasons (docs/configuration.md, "Who applies this folder"). Succeeded and +// Failed are the generic reasons, aliased from Flux's meta package as the conditions guide asks. +const ( + // ReasonSucceeded is ApplierConfigured=True: the applier exists, applies this repository and + // branch, and may be asked to apply. + ReasonSucceeded = fluxmeta.SucceededReason + // ReasonApplierNotFound is no such object in the source cluster, or its kind is not served there. + ReasonApplierNotFound = "ApplierNotFound" + // ReasonSourceMismatch is an applier that applies a different repository or branch. + ReasonSourceMismatch = "SourceMismatch" + // ReasonForbidden is a source-cluster identity that may not read or patch the applier. + ReasonForbidden = "Forbidden" + // ReasonCheckFailed is a check that could not complete: the source cluster did not answer. + // Verdict.Err carries the cause; it says nothing about the applier itself. + ReasonCheckFailed = fluxmeta.FailedReason +) + +// Verdict is the answer Validate gives for the ApplierConfigured condition. +type Verdict struct { + // OK is ApplierConfigured=True. + OK bool + // Reason is the condition reason. + Reason string + // Message names the object and what is wrong with it. + Message string + // Err is set only with ReasonCheckFailed: the read that failed. A caller may prefer to keep the + // verdict it already published over replacing a known answer with "could not tell". + Err error +} + +// AccessCheck reports whether the source-cluster identity may perform attrs. +type AccessCheck func(ctx context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) + +// Flux asks a Flux Kustomization to apply, in one source cluster. +type Flux struct { + client dynamic.Interface + + // Now is the clock the requestedAt stamp is taken from. + Now func() time.Time + // CanI is the access check Validate runs for patch. NewFlux sets it to a + // SelfSubjectAccessReview through the same client, so it is answered for the identity that + // will make the request. + CanI AccessCheck +} + +// NewFlux returns a Flux client over a source cluster's dynamic client. +func NewFlux(client dynamic.Interface) *Flux { + return &Flux{client: client, Now: time.Now, CanI: SelfSubjectAccessCheck(client)} +} + +// permanentError marks a request no retry can make succeed: the Kustomization is configured in a +// way a request cannot work with. It fails loudly once rather than retrying to the limit; a newer +// request still tries again, because the configuration may have been fixed in between. +type permanentError struct{ err error } + +func (e permanentError) Error() string { return e.err.Error() } +func (e permanentError) Unwrap() error { return e.err } + +// isPermanent reports whether err is a permanentError. +func isPermanent(err error) bool { + var p permanentError + return errors.As(err, &p) +} + +// Request asks the Kustomization ref to apply commit, the way `flux reconcile kustomization +// --with-source` does, minus the waiting: a request asks and never waits to see the apply. +// +// When the GitRepository's artifact does not hold commit, it stamps the GitRepository: the fetch +// produces a new artifact and the new artifact wakes the Kustomization by itself, so a second stamp +// would only be a second apply. When the artifact already holds it, a source stamp would wake +// nothing, so it stamps the Kustomization instead, which is what a Refusal at an unchanged commit +// needs; a CatchUp needs nothing then, because a CatchUp only keeps the artifact current. An empty +// commit (the remote does not carry the branch) stamps the GitRepository, so Flux finds out itself. +func (f *Flux) Request( + ctx context.Context, + ref types.NamespacedName, + commit string, + reason v1alpha3.ApplyReason, +) (Outcome, error) { + ks, err := f.client.Resource(kustomizationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + if err != nil { + return "", fmt.Errorf("get %s %s: %w", kindKustomization, ref, err) + } + sourceRef, err := gitRepositoryRef(ks) + if err != nil { + return "", permanentError{err} + } + source, err := f.client.Resource(gitRepositoryGVR()).Namespace(sourceRef.Namespace). + Get(ctx, sourceRef.Name, metav1.GetOptions{}) + if err != nil { + return "", fmt.Errorf("get %s %s: %w", kindGitRepository, sourceRef, err) + } + + if commit != "" && artifactHolds(source, commit) { + if reason == v1alpha3.ApplyReasonCatchUp { + return OutcomeCurrent, nil + } + if err := f.stamp(ctx, kustomizationGVR(), ref); err != nil { + return "", fmt.Errorf("stamp %s %s: %w", kindKustomization, ref, err) + } + return OutcomeStampedKustomization, nil + } + if err := f.stamp(ctx, gitRepositoryGVR(), sourceRef); err != nil { + return "", fmt.Errorf("stamp %s %s: %w", kindGitRepository, sourceRef, err) + } + return OutcomeStampedSource, nil +} + +// stamp sets reconcile.fluxcd.io/requestedAt to now, which is the whole of what `flux reconcile` +// does to ask: requestReconciliation in external-sources/flux/flux2/cmd/flux/reconcile.go +// (@1ed703a3) merge-patches that annotation, and only that, with time.Now().Format(time.RFC3339Nano). +// Flux reconciles when the value differs from the status.lastHandledReconcileAt it last answered, +// so any new value asks; no read-modify-write is needed and no conflict is possible. +func (f *Flux) stamp(ctx context.Context, gvr schema.GroupVersionResource, ref types.NamespacedName) error { + patch, err := json.Marshal(map[string]any{ + "metadata": map[string]any{ + "annotations": map[string]string{ + fluxmeta.ReconcileRequestAnnotation: f.Now().Format(time.RFC3339Nano), + }, + }, + }) + if err != nil { + return err + } + _, err = f.client.Resource(gvr).Namespace(ref.Namespace). + Patch(ctx, ref.Name, types.MergePatchType, patch, metav1.PatchOptions{}) + return err +} + +// gitRepositoryRef reads a Kustomization's spec.sourceRef, which must name a GitRepository. Its +// namespace defaults to the Kustomization's own, as Flux defaults it. +func gitRepositoryRef(ks *unstructured.Unstructured) (types.NamespacedName, error) { + kind, _, _ := unstructured.NestedString(ks.Object, "spec", "sourceRef", "kind") + name, _, _ := unstructured.NestedString(ks.Object, "spec", "sourceRef", "name") + namespace, _, _ := unstructured.NestedString(ks.Object, "spec", "sourceRef", "namespace") + self := types.NamespacedName{Namespace: ks.GetNamespace(), Name: ks.GetName()} + switch { + case kind == "" || name == "": + return types.NamespacedName{}, fmt.Errorf("%s %s names no spec.sourceRef", kindKustomization, self) + case kind != kindGitRepository: + return types.NamespacedName{}, fmt.Errorf("%s %s applies from a %s, not a %s", + kindKustomization, self, kind, kindGitRepository) + } + if namespace == "" { + namespace = ks.GetNamespace() + } + return types.NamespacedName{Namespace: namespace, Name: name}, nil +} + +// artifactHolds reports whether the GitRepository's current artifact was built from commit. +func artifactHolds(source *unstructured.Unstructured, commit string) bool { + revision, _, _ := unstructured.NestedString(source.Object, "status", "artifact", "revision") + _, hash := splitRevision(revision) + return hash != "" && strings.EqualFold(hash, commit) +} + +// splitRevision splits a Flux artifact revision into its named pointer (the branch, or "" when +// the revision names none) and its commit hash. It accepts every format source-controller has +// written, mirroring SplitRevision in external-sources/flux/pkg/git/utils.go (@c8dd701d), rather +// than importing fluxcd/pkg/git and its go-git for one string split: +// +// - main@sha1: and feature/branch@sha1:, the current form +// - sha1:, a revision with no named pointer +// - main/ and feature/branch/, the form before source-controller v1 +// - HEAD/ and a bare +// +// The algorithm prefix is dropped: a commit is compared by its hash, and a SHA-256 repository's +// "sha256:" splits the same way. +func splitRevision(revision string) (string, string) { + if revision == "" { + return "", "" + } + var named string + if i := strings.LastIndex(revision, "@"); i != -1 { + named = revision[:i] + } else if i := strings.LastIndex(revision, "/"); i != -1 && revision[:i] != "HEAD" { + named = revision[:i] + } + if i := strings.LastIndex(revision, ":"); i != -1 { + return named, revision[i+1:] + } + if i := strings.LastIndex(revision, "/"); i != -1 { + return named, revision[i+1:] + } + return named, revision +} + +// Validate answers ApplierConfigured for a Kustomization: that it exists in the source cluster and +// its kind is served there, that it applies a GitRepository fetching branch of repositoryURL (the +// GitProvider's spec.url, compared after normalization), and that this identity may patch both +// objects, since a request patches one or the other. +// +// It reads; it never watches. A cluster without Flux installed answers ApplierNotFound from one +// failed GET and keeps no informer for kinds it does not serve. +func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, repositoryURL, branch string) Verdict { + ks, err := f.client.Resource(kustomizationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + if err != nil { + return readFailure(kindKustomization, kustomizationGVR(), ref, err) + } + sourceRef, err := gitRepositoryRef(ks) + if err != nil { + return Verdict{Reason: ReasonSourceMismatch, Message: err.Error()} + } + source, err := f.client.Resource(gitRepositoryGVR()).Namespace(sourceRef.Namespace). + Get(ctx, sourceRef.Name, metav1.GetOptions{}) + if err != nil { + verdict := readFailure(kindGitRepository, gitRepositoryGVR(), sourceRef, err) + verdict.Message = fmt.Sprintf("%s (the sourceRef of %s %s)", verdict.Message, kindKustomization, ref) + return verdict + } + if msg := sourceMismatch(source, sourceRef, repositoryURL, branch); msg != "" { + return Verdict{Reason: ReasonSourceMismatch, Message: msg} + } + + for _, obj := range []struct { + kind string + gvr schema.GroupVersionResource + ref types.NamespacedName + }{ + {kindGitRepository, gitRepositoryGVR(), sourceRef}, + {kindKustomization, kustomizationGVR(), ref}, + } { + allowed, err := f.CanI(ctx, authorizationv1.ResourceAttributes{ + Namespace: obj.ref.Namespace, + Verb: "patch", + Group: obj.gvr.Group, + Version: obj.gvr.Version, + Resource: obj.gvr.Resource, + Name: obj.ref.Name, + }) + if err != nil { + return Verdict{ + Reason: ReasonCheckFailed, + Message: fmt.Sprintf("could not check whether %s %s may be patched: %v", obj.kind, obj.ref, err), + Err: err, + } + } + if !allowed { + return Verdict{ + Reason: ReasonForbidden, + Message: fmt.Sprintf("the source-cluster identity may not patch %s %s (%s)", + obj.kind, obj.ref, obj.gvr.GroupResource()), + } + } + } + + return Verdict{ + OK: true, + Reason: ReasonSucceeded, + Message: fmt.Sprintf("%s %s applies %s %s, which fetches branch %s", + kindKustomization, ref, kindGitRepository, sourceRef, branch), + } +} + +// readFailure classifies a failed GET for Validate. +// +// A kind that is not served is ApplierNotFound like a missing object, because the fix is the same +// (install Flux, or name the right object) and the condition is a gate either way. The two are +// told apart only for the message: a client with a RESTMapper says so with a NoKindMatch, and the +// API server answers the dynamic client's GET of an unserved resource with a 404 whose details name +// no object, where a missing object's 404 names it. +func readFailure(kind string, gvr schema.GroupVersionResource, ref types.NamespacedName, err error) Verdict { + switch { + case apimeta.IsNoMatchError(err) || (apierrors.IsNotFound(err) && !namesObject(err, ref.Name)): + return Verdict{ + Reason: ReasonApplierNotFound, + Message: fmt.Sprintf("the source cluster does not serve %s (%s); is Flux installed there?", + kind, gvr.GroupVersion()), + } + case apierrors.IsNotFound(err): + return Verdict{ + Reason: ReasonApplierNotFound, + Message: fmt.Sprintf("%s %s not found in the source cluster", kind, ref), + } + case apierrors.IsForbidden(err): + return Verdict{ + Reason: ReasonForbidden, + Message: fmt.Sprintf("the source-cluster identity may not get %s %s: %v", kind, ref, err), + } + default: + return Verdict{ + Reason: ReasonCheckFailed, + Message: fmt.Sprintf("could not read %s %s: %v", kind, ref, err), + Err: err, + } + } +} + +// namesObject reports whether a NotFound error is about the named object, rather than the resource. +func namesObject(err error, name string) bool { + var status apierrors.APIStatus + if !errors.As(err, &status) { + return false + } + details := status.Status().Details + return details != nil && details.Name == name +} + +// sourceMismatch returns why a GitRepository does not fetch branch of repositoryURL, or "" when it +// does. A ref that pins anything stronger than a branch is a mismatch even when it also names the +// branch: Flux resolves commit, then name, then semver, then tag, before branch, so the pin wins and +// the artifact never follows the branch GitOps Reverser writes. +func sourceMismatch(source *unstructured.Unstructured, ref types.NamespacedName, repositoryURL, branch string) string { + url, _, _ := unstructured.NestedString(source.Object, "spec", "url") + if !repourl.Same(url, repositoryURL) { + return fmt.Sprintf("%s %s fetches %s, not %s", kindGitRepository, ref, url, repositoryURL) + } + gitRef, found, _ := unstructured.NestedStringMap(source.Object, "spec", "ref") + if !found || len(gitRef) == 0 { + return fmt.Sprintf("%s %s names no spec.ref.branch; GitOps Reverser writes branch %s", + kindGitRepository, ref, branch) + } + for _, pin := range []string{"commit", "name", "semver", "tag"} { + if value := gitRef[pin]; value != "" { + return fmt.Sprintf("%s %s pins spec.ref.%s %q, so it never follows branch %s", + kindGitRepository, ref, pin, value, branch) + } + } + if gitRef["branch"] != branch { + return fmt.Sprintf("%s %s fetches branch %q, not %s", kindGitRepository, ref, gitRef["branch"], branch) + } + return "" +} + +// SelfSubjectAccessCheck answers an AccessCheck with a SelfSubjectAccessReview created through +// client, so it is answered for the identity client authenticates as. It goes through the dynamic +// client rather than a typed clientset because that client is the only one a source cluster has. +func SelfSubjectAccessCheck(client dynamic.Interface) AccessCheck { + return func(ctx context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) { + review := &authorizationv1.SelfSubjectAccessReview{ + TypeMeta: metav1.TypeMeta{ + APIVersion: authorizationv1.SchemeGroupVersion.String(), + Kind: "SelfSubjectAccessReview", + }, + Spec: authorizationv1.SelfSubjectAccessReviewSpec{ResourceAttributes: &attrs}, + } + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(review) + if err != nil { + return false, err + } + created, err := client.Resource(selfSubjectAccessReviewGVR()). + Create(ctx, &unstructured.Unstructured{Object: content}, metav1.CreateOptions{}) + if err != nil { + return false, fmt.Errorf("create SelfSubjectAccessReview: %w", err) + } + allowed, _, err := unstructured.NestedBool(created.Object, "status", "allowed") + if err != nil { + return false, fmt.Errorf("read SelfSubjectAccessReview status: %w", err) + } + return allowed, nil + } +} diff --git a/internal/applier/flux_test.go b/internal/applier/flux_test.go new file mode 100644 index 00000000..55bc8771 --- /dev/null +++ b/internal/applier/flux_test.go @@ -0,0 +1,494 @@ +// SPDX-License-Identifier: Apache-2.0 + +package applier + +import ( + "context" + "errors" + "testing" + "time" + + fluxmeta "github.com/fluxcd/pkg/apis/meta" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + authorizationv1 "k8s.io/api/authorization/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + dynamicfake "k8s.io/client-go/dynamic/fake" + k8stesting "k8s.io/client-go/testing" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" +) + +const ( + testCommit = "4f2c1ab9d0e1f2a3b4c5d6e7f8091a2b3c4d5e6f" + olderCommit = "0000000000000000000000000000000000000001" + testRepoURL = "https://github.com/acme/platform" + testBranch = "main" + fluxSystemNS = "flux-system" +) + +var stampTime = time.Date(2026, 10, 20, 10, 14, 3, 123456789, time.UTC) + +func newSourceClient(objs ...runtime.Object) *dynamicfake.FakeDynamicClient { + return dynamicfake.NewSimpleDynamicClientWithCustomListKinds(runtime.NewScheme(), + map[schema.GroupVersionResource]string{ + kustomizationGVR(): "KustomizationList", + gitRepositoryGVR(): "GitRepositoryList", + }, objs...) +} + +// kustomizationObj builds a Kustomization; a nil sourceRef omits spec.sourceRef. +func kustomizationObj(namespace, name string, sourceRef map[string]any) *unstructured.Unstructured { + obj := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "kustomize.toolkit.fluxcd.io/v1", + "kind": kindKustomization, + "metadata": map[string]any{"namespace": namespace, "name": name}, + "spec": map[string]any{"path": "./apps"}, + }} + if sourceRef != nil { + _ = unstructured.SetNestedMap(obj.Object, sourceRef, "spec", "sourceRef") + } + return obj +} + +func gitRepoSourceRef(name string) map[string]any { + return map[string]any{"kind": kindGitRepository, "name": name} +} + +// gitRepositoryObj builds a GitRepository; a nil ref omits spec.ref, an empty revision omits the +// artifact. +func gitRepositoryObj(namespace, name, url string, ref map[string]any, revision string) *unstructured.Unstructured { + obj := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "source.toolkit.fluxcd.io/v1", + "kind": kindGitRepository, + "metadata": map[string]any{"namespace": namespace, "name": name}, + "spec": map[string]any{"url": url}, + }} + if ref != nil { + _ = unstructured.SetNestedMap(obj.Object, ref, "spec", "ref") + } + if revision != "" { + _ = unstructured.SetNestedField(obj.Object, revision, "status", "artifact", "revision") + } + return obj +} + +func branchRef(branch string) map[string]any { return map[string]any{"branch": branch} } + +// requestedAt returns an object's reconcile.fluxcd.io/requestedAt, or "". +func requestedAt(t *testing.T, client *dynamicfake.FakeDynamicClient, gvr schema.GroupVersionResource, + namespace, name string) string { + t.Helper() + obj, err := client.Resource(gvr).Namespace(namespace).Get(context.Background(), name, metav1.GetOptions{}) + require.NoError(t, err) + return obj.GetAnnotations()[fluxmeta.ReconcileRequestAnnotation] +} + +func pinnedFlux(client *dynamicfake.FakeDynamicClient) *Flux { + flux := NewFlux(client) + flux.Now = func() time.Time { return stampTime } + return flux +} + +func TestSplitRevision_Formats(t *testing.T) { + tests := []struct { + revision string + wantNamed string + wantHash string + }{ + {"main@sha1:" + testCommit, "main", testCommit}, + {"feature/branch@sha1:" + testCommit, "feature/branch", testCommit}, + {"refs/heads/main@sha1:" + testCommit, "refs/heads/main", testCommit}, + {"sha1:" + testCommit, "", testCommit}, + {"main@sha256:abc123", "main", "abc123"}, + {"main/" + testCommit, "main", testCommit}, + {"feature/branch/" + testCommit, "feature/branch", testCommit}, + {"HEAD/" + testCommit, "", testCommit}, + {testCommit, "", testCommit}, + {"", "", ""}, + } + for _, tt := range tests { + t.Run(tt.revision, func(t *testing.T) { + named, hash := splitRevision(tt.revision) + assert.Equal(t, tt.wantNamed, named) + assert.Equal(t, tt.wantHash, hash) + }) + } +} + +// TestFluxRequest_Outcomes pins which object a request stamps, by whether the artifact already +// holds the commit and by reason. +func TestFluxRequest_Outcomes(t *testing.T) { + tests := []struct { + name string + revision string + commit string + reason v1alpha3.ApplyReason + want Outcome + wantSourceStamp bool + wantKustomStamps bool + }{ + { + name: "foreign commit not yet fetched stamps the source", revision: "main@sha1:" + olderCommit, + commit: testCommit, reason: v1alpha3.ApplyReasonForeignCommit, + want: OutcomeStampedSource, wantSourceStamp: true, + }, + { + name: "catch-up behind stamps the source", revision: "main@sha1:" + olderCommit, + commit: testCommit, reason: v1alpha3.ApplyReasonCatchUp, + want: OutcomeStampedSource, wantSourceStamp: true, + }, + { + name: "no artifact yet stamps the source", revision: "", + commit: testCommit, reason: v1alpha3.ApplyReasonForeignCommit, + want: OutcomeStampedSource, wantSourceStamp: true, + }, + { + name: "branch absent on the remote stamps the source", revision: "main@sha1:" + testCommit, + commit: "", reason: v1alpha3.ApplyReasonForeignCommit, + want: OutcomeStampedSource, wantSourceStamp: true, + }, + { + name: "refusal at the artifact's commit stamps the kustomization", revision: "main@sha1:" + testCommit, + commit: testCommit, reason: v1alpha3.ApplyReasonRefusal, + want: OutcomeStampedKustomization, wantKustomStamps: true, + }, + { + name: "foreign commit already fetched stamps the kustomization", revision: "main/" + testCommit, + commit: testCommit, reason: v1alpha3.ApplyReasonForeignCommit, + want: OutcomeStampedKustomization, wantKustomStamps: true, + }, + { + name: "catch-up already current stamps nothing", revision: "sha1:" + testCommit, + commit: testCommit, reason: v1alpha3.ApplyReasonCatchUp, + want: OutcomeCurrent, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := newSourceClient( + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), tt.revision), + ) + got, err := pinnedFlux(client).Request(context.Background(), + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, tt.commit, tt.reason) + require.NoError(t, err) + assert.Equal(t, tt.want, got) + + wantStamp := func(want bool) string { + if want { + return stampTime.Format(time.RFC3339Nano) + } + return "" + } + assert.Equal(t, wantStamp(tt.wantSourceStamp), + requestedAt(t, client, gitRepositoryGVR(), fluxSystemNS, "platform")) + assert.Equal(t, wantStamp(tt.wantKustomStamps), + requestedAt(t, client, kustomizationGVR(), fluxSystemNS, "checkout")) + }) + } +} + +// TestFluxRequest_SourceRefNamespace: an omitted sourceRef namespace is the Kustomization's own; a +// set one is followed across namespaces. +func TestFluxRequest_SourceRefNamespace(t *testing.T) { + tests := []struct { + name string + sourceRef map[string]any + wantNamespace string + }{ + {"defaults to the kustomization's", gitRepoSourceRef("platform"), "apps"}, + { + "follows an explicit one", + map[string]any{"kind": kindGitRepository, "name": "platform", "namespace": fluxSystemNS}, + fluxSystemNS, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := newSourceClient( + kustomizationObj("apps", "checkout", tt.sourceRef), + gitRepositoryObj(tt.wantNamespace, "platform", testRepoURL, branchRef(testBranch), ""), + ) + kustomization := types.NamespacedName{Namespace: "apps", Name: "checkout"} + got, err := pinnedFlux(client).Request(context.Background(), kustomization, testCommit, + v1alpha3.ApplyReasonForeignCommit) + require.NoError(t, err) + assert.Equal(t, OutcomeStampedSource, got) + assert.NotEmpty(t, requestedAt(t, client, gitRepositoryGVR(), tt.wantNamespace, "platform")) + }) + } +} + +func TestFluxRequest_Failures(t *testing.T) { + tests := []struct { + name string + objs []runtime.Object + wantPermanent bool + }{ + {name: "kustomization not found"}, + { + name: "git repository not found", + objs: []runtime.Object{kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform"))}, + }, + { + name: "source is not a git repository", + objs: []runtime.Object{kustomizationObj(fluxSystemNS, "checkout", + map[string]any{"kind": "Bucket", "name": "platform"})}, + wantPermanent: true, + }, + { + name: "no source ref", + objs: []runtime.Object{kustomizationObj(fluxSystemNS, "checkout", nil)}, + wantPermanent: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + _, err := pinnedFlux(newSourceClient(tt.objs...)).Request(context.Background(), + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testCommit, + v1alpha3.ApplyReasonForeignCommit) + require.Error(t, err) + assert.Equal(t, tt.wantPermanent, isPermanent(err)) + }) + } +} + +// allowAll and denyOn are AccessCheck fakes. +func allowAll(context.Context, authorizationv1.ResourceAttributes) (bool, error) { return true, nil } + +func denyOn(resource string) AccessCheck { + return func(_ context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) { + return attrs.Resource != resource, nil + } +} + +// TestFluxValidate_Reasons covers every ApplierConfigured answer. +func TestFluxValidate_Reasons(t *testing.T) { + healthyKustomization := kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")) + repo := func(url string, ref map[string]any) runtime.Object { + return gitRepositoryObj(fluxSystemNS, "platform", url, ref, "") + } + tests := []struct { + name string + objs []runtime.Object + reactor k8stesting.ReactionFunc + canI AccessCheck + wantReason string + wantMessage string + wantErr bool + }{ + { + name: "configured", + objs: []runtime.Object{healthyKustomization, repo(testRepoURL, branchRef(testBranch))}, + wantReason: ReasonSucceeded, + wantMessage: "Kustomization flux-system/checkout applies GitRepository flux-system/platform", + }, + { + name: "kustomization missing", + wantReason: ReasonApplierNotFound, + wantMessage: "Kustomization flux-system/checkout not found", + }, + { + name: "kind not served, as a RESTMapper says it", + reactor: func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, &apimeta.NoKindMatchError{ + GroupKind: schema.GroupKind{Group: kustomizationGVR().Group, Kind: kindKustomization}, + } + }, + wantReason: ReasonApplierNotFound, + wantMessage: "does not serve Kustomization (kustomize.toolkit.fluxcd.io/v1)", + }, + { + name: "kind not served, as the API server's bare 404 says it", + reactor: func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewNotFound(kustomizationGVR().GroupResource(), "") + }, + wantReason: ReasonApplierNotFound, + wantMessage: "is Flux installed there?", + }, + { + name: "git repository missing", + objs: []runtime.Object{healthyKustomization}, + wantReason: ReasonApplierNotFound, + wantMessage: "GitRepository flux-system/platform not found in the source cluster (the sourceRef of", + }, + { + name: "source is a bucket", + objs: []runtime.Object{kustomizationObj(fluxSystemNS, "checkout", + map[string]any{"kind": "Bucket", "name": "platform"})}, + wantReason: ReasonSourceMismatch, + wantMessage: "applies from a Bucket, not a GitRepository", + }, + { + name: "another repository", + objs: []runtime.Object{ + healthyKustomization, + repo("https://github.com/acme/other", branchRef(testBranch)), + }, + wantReason: ReasonSourceMismatch, + wantMessage: "fetches https://github.com/acme/other, not https://github.com/acme/platform", + }, + { + name: "another branch", + objs: []runtime.Object{healthyKustomization, repo(testRepoURL, branchRef("release"))}, + wantReason: ReasonSourceMismatch, + wantMessage: `fetches branch "release", not main`, + }, + { + name: "no ref", + objs: []runtime.Object{healthyKustomization, repo(testRepoURL, nil)}, + wantReason: ReasonSourceMismatch, + wantMessage: "names no spec.ref.branch", + }, + { + name: "pinned tag", + objs: []runtime.Object{healthyKustomization, repo(testRepoURL, map[string]any{"tag": "v1.0.0"})}, + wantReason: ReasonSourceMismatch, + wantMessage: `pins spec.ref.tag "v1.0.0"`, + }, + { + name: "pinned commit beside the branch", + objs: []runtime.Object{healthyKustomization, + repo(testRepoURL, map[string]any{"branch": testBranch, "commit": testCommit})}, + wantReason: ReasonSourceMismatch, + wantMessage: "pins spec.ref.commit", + }, + { + name: "may not patch the git repository", + objs: []runtime.Object{healthyKustomization, repo(testRepoURL, branchRef(testBranch))}, + canI: denyOn("gitrepositories"), + wantReason: ReasonForbidden, + wantMessage: "may not patch GitRepository flux-system/platform", + }, + { + name: "may not patch the kustomization", + objs: []runtime.Object{healthyKustomization, repo(testRepoURL, branchRef(testBranch))}, + canI: denyOn("kustomizations"), + wantReason: ReasonForbidden, + wantMessage: "may not patch Kustomization flux-system/checkout", + }, + { + name: "may not get the kustomization", + reactor: func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewForbidden(kustomizationGVR().GroupResource(), "checkout", + errors.New("denied")) + }, + wantReason: ReasonForbidden, + wantMessage: "may not get Kustomization flux-system/checkout", + }, + { + name: "source cluster did not answer", + reactor: func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewServiceUnavailable("etcd is sulking") + }, + wantReason: ReasonCheckFailed, + wantErr: true, + }, + { + name: "access check did not answer", + objs: []runtime.Object{healthyKustomization, repo(testRepoURL, branchRef(testBranch))}, + canI: func(context.Context, authorizationv1.ResourceAttributes) (bool, error) { + return false, errors.New("timeout") + }, + wantReason: ReasonCheckFailed, + wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := newSourceClient(tt.objs...) + if tt.reactor != nil { + client.PrependReactor("get", "kustomizations", tt.reactor) + } + flux := pinnedFlux(client) + flux.CanI = allowAll + if tt.canI != nil { + flux.CanI = tt.canI + } + + got := flux.Validate(context.Background(), + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testRepoURL, testBranch) + assert.Equal(t, tt.wantReason, got.Reason, got.Message) + assert.Equal(t, tt.wantReason == ReasonSucceeded, got.OK) + assert.Contains(t, got.Message, tt.wantMessage) + assert.Equal(t, tt.wantErr, got.Err != nil) + }) + } +} + +// TestFluxValidate_URLEquivalences: the GitRepository's url and the GitProvider's are compared as +// repositories, not as strings. +func TestFluxValidate_URLEquivalences(t *testing.T) { + tests := []struct { + name string + sourceURL string + provider string + wantOK bool + }{ + {"identical", testRepoURL, testRepoURL, true}, + {"trailing .git", testRepoURL + ".git", testRepoURL, true}, + {"scp-style ssh", "git@github.com:acme/platform.git", testRepoURL, true}, + {"ssh url", "ssh://git@github.com/acme/platform", "git@github.com:acme/platform.git", true}, + {"case and trailing slash", "https://GitHub.com/Acme/Platform/", testRepoURL, true}, + {"default port", "https://github.com:443/acme/platform", testRepoURL, true}, + {"another org", "https://github.com/other/platform", testRepoURL, false}, + {"another host", "https://gitlab.com/acme/platform", testRepoURL, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := newSourceClient( + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", tt.sourceURL, branchRef(testBranch), ""), + ) + flux := pinnedFlux(client) + flux.CanI = allowAll + got := flux.Validate(context.Background(), + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, tt.provider, testBranch) + assert.Equal(t, tt.wantOK, got.OK, got.Message) + }) + } +} + +// TestSelfSubjectAccessCheck_AsksTheSourceCluster: the default access check creates a +// SelfSubjectAccessReview with the attributes asked about and answers from its status. +func TestSelfSubjectAccessCheck_AsksTheSourceCluster(t *testing.T) { + client := newSourceClient() + var asked []string + client.PrependReactor("create", "selfsubjectaccessreviews", + func(action k8stesting.Action) (bool, runtime.Object, error) { + review := action.(k8stesting.CreateAction).GetObject().(*unstructured.Unstructured) + resource, _, _ := unstructured.NestedString(review.Object, "spec", "resourceAttributes", "resource") + verb, _, _ := unstructured.NestedString(review.Object, "spec", "resourceAttributes", "verb") + asked = append(asked, verb+" "+resource) + _ = unstructured.SetNestedField(review.Object, resource == "gitrepositories", "status", "allowed") + return true, review, nil + }) + check := SelfSubjectAccessCheck(client) + + allowed, err := check(context.Background(), authorizationv1.ResourceAttributes{ + Verb: "patch", Group: gitRepositoryGVR().Group, Resource: "gitrepositories", Namespace: fluxSystemNS, + }) + require.NoError(t, err) + assert.True(t, allowed) + + allowed, err = check(context.Background(), authorizationv1.ResourceAttributes{ + Verb: "patch", Group: kustomizationGVR().Group, Resource: "kustomizations", Namespace: fluxSystemNS, + }) + require.NoError(t, err) + assert.False(t, allowed) + assert.Equal(t, []string{"patch gitrepositories", "patch kustomizations"}, asked) + + client.PrependReactor("create", "selfsubjectaccessreviews", + func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewServiceUnavailable("down") + }) + _, err = check(context.Background(), authorizationv1.ResourceAttributes{Verb: "patch"}) + require.Error(t, err) +} diff --git a/internal/controller/gitprovider_controller.go b/internal/controller/gitprovider_controller.go index 00b5aa11..34587e21 100644 --- a/internal/controller/gitprovider_controller.go +++ b/internal/controller/gitprovider_controller.go @@ -121,6 +121,9 @@ func (r *GitProviderReconciler) reconcileGitProvider( log.Error(err, "Could not list the GitTargets for the branch inventory; the last one stands", "name", gitProvider.Name, "namespace", gitProvider.Namespace) } + // Ahead of the gates for the same reason: the path is a fact about the spec, and an operator + // wiring a Git host's webhook needs it while the provider is still red. + publishWebhookPath(gitProvider) if err := r.validateCommitConfiguration(gitProvider); err != nil { rd.stalled(ReasonCommitConfigInvalid, err.Error()) @@ -191,6 +194,17 @@ func (r *GitProviderReconciler) commitProvider( return ctrl.Result{RequeueAfter: st.requeueAfter(RequeueSteadyInterval)}, nil } +// publishWebhookPath sets status.webhookPath to the path spec.receiver is served on, and clears it +// when there is no receiver. It says where the route is, not that anything serves it: the listener +// is a flag (--receiver-bind-address) this object cannot see, and the path is the same either way. +func publishWebhookPath(gitProvider *configbutleraiv1alpha3.GitProvider) { + if gitProvider.Spec.Receiver == nil { + gitProvider.Status.WebhookPath = "" + return + } + gitProvider.Status.WebhookPath = configbutleraiv1alpha3.ReceiverPath(gitProvider.Namespace, gitProvider.Name) +} + // signingKeyFailureReason classifies an ensureSigningKey failure for the Stalled reason. func signingKeyFailureReason(err error) string { switch { diff --git a/internal/controller/gitprovider_controller_test.go b/internal/controller/gitprovider_controller_test.go index 15cf86ed..de9dcee6 100644 --- a/internal/controller/gitprovider_controller_test.go +++ b/internal/controller/gitprovider_controller_test.go @@ -364,6 +364,44 @@ var _ = Describe("GitProvider Controller", func() { }), "a branch nothing references any more is not what this repository is for") }) + It("publishes status.webhookPath while spec.receiver is set, and clears it when removed", func() { + // Published ahead of every gate like the inventory: the remote is unreachable here, and + // the path is still what an operator needs to wire the Git host's webhook. + gitProvider = &configbutleraiv1alpha3.GitProvider{ + ObjectMeta: metav1.ObjectMeta{Name: "test-provider-receiver", Namespace: "default"}, + Spec: configbutleraiv1alpha3.GitProviderSpec{ + URL: "https://example.invalid/unreachable.git", + AllowedBranches: []string{"main"}, + Receiver: &configbutleraiv1alpha3.GitProviderReceiver{ + Type: configbutleraiv1alpha3.ReceiverTypeGitHub, + SecretRef: meta.LocalObjectReference{Name: "receiver-token"}, + }, + }, + } + Expect(k8sClient.Create(ctx, gitProvider)).To(Succeed()) + + key := types.NamespacedName{Name: gitProvider.Name, Namespace: gitProvider.Namespace} + webhookPath := func(g Gomega) string { + _, err := reconciler.Reconcile(ctx, reconcile.Request{NamespacedName: key}) + g.Expect(err).NotTo(HaveOccurred()) + published := &configbutleraiv1alpha3.GitProvider{} + g.Expect(k8sClient.Get(ctx, key, published)).To(Succeed()) + return published.Status.WebhookPath + } + + Eventually(webhookPath, "30s", "500ms").Should(Equal("/git-push/default/test-provider-receiver")) + + Eventually(func(g Gomega) { + current := &configbutleraiv1alpha3.GitProvider{} + g.Expect(k8sClient.Get(ctx, key, current)).To(Succeed()) + current.Spec.Receiver = nil + g.Expect(k8sClient.Update(ctx, current)).To(Succeed()) + }, "10s", "200ms").Should(Succeed()) + + Eventually(webhookPath, "30s", "500ms").Should(BeEmpty(), + "a stale path would point a Git host at a route that no longer serves") + }) + It("should fail when secret is not found", func() { gitProvider = &configbutleraiv1alpha3.GitProvider{ ObjectMeta: metav1.ObjectMeta{ diff --git a/internal/controller/gitprovider_controller_unit_test.go b/internal/controller/gitprovider_controller_unit_test.go index 2938dc9d..e52486d6 100644 --- a/internal/controller/gitprovider_controller_unit_test.go +++ b/internal/controller/gitprovider_controller_unit_test.go @@ -150,3 +150,25 @@ func newGitProviderTestClient(t *testing.T, objects ...runtime.Object) ctrlclien WithRuntimeObjects(objects...). Build() } + +// TestPublishWebhookPath_FollowsSpecReceiver pins status.webhookPath to spec.receiver: set while a +// receiver is configured, cleared the moment it is removed, so a stale path never points a Git +// host at a route that has stopped serving. +func TestPublishWebhookPath_FollowsSpecReceiver(t *testing.T) { + provider := &configbutleraiv1alpha3.GitProvider{ + ObjectMeta: metav1.ObjectMeta{Name: "platform-repo", Namespace: "team-a"}, + Spec: configbutleraiv1alpha3.GitProviderSpec{ + Receiver: &configbutleraiv1alpha3.GitProviderReceiver{ + Type: configbutleraiv1alpha3.ReceiverTypeGitHub, + SecretRef: meta.LocalObjectReference{Name: "receiver-token"}, + }, + }, + } + + publishWebhookPath(provider) + assert.Equal(t, "/git-push/team-a/platform-repo", provider.Status.WebhookPath) + + provider.Spec.Receiver = nil + publishWebhookPath(provider) + assert.Empty(t, provider.Status.WebhookPath) +} diff --git a/internal/controller/gittarget_applier.go b/internal/controller/gittarget_applier.go new file mode 100644 index 00000000..c362abfc --- /dev/null +++ b/internal/controller/gittarget_applier.go @@ -0,0 +1,130 @@ +// SPDX-License-Identifier: Apache-2.0 + +package controller + +import ( + "context" + "fmt" + + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + k8stypes "k8s.io/apimachinery/pkg/types" + + configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/applier" + "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// GitTargetConditionApplierConfigured reports whether spec.applier names an object GitOps Reverser +// can ask to apply: it exists in the source cluster, applies this target's repository and branch, +// and may be patched. Present only while spec.applier is set. +const GitTargetConditionApplierConfigured = "ApplierConfigured" + +// ApplyRequestLedger is what the GitTarget controller reads to publish status.applier: the last +// apply request delivered for a GitTarget. The applier dispatcher implements it. +type ApplyRequestLedger interface { + LastRequest(ref types.ResourceReference) (configbutleraiv1alpha3.GitTargetApplierStatus, bool) +} + +// observeApplier publishes status.applier and the ApplierConfigured condition, and returns the +// condition for the readiness trio. present is false when the target names no applier, in which +// case both are removed. +// +// The check reads the applier on this reconcile rather than watching it. Flux may not be installed +// in a source cluster at all, and an informer on a kind that is not served would be a failure to +// report rather than a check that answers "not found"; the cadence is the target's own. +func (r *GitTargetReconciler) observeApplier( + ctx context.Context, + st *reconcileStatus, + target *configbutleraiv1alpha3.GitTarget, + gitProvider *configbutleraiv1alpha3.GitProvider, +) (conditionValue, bool) { + if target.Spec.Applier == nil { + st.remove(GitTargetConditionApplierConfigured) + target.Status.Applier = nil + return conditionValue{}, false + } + + if r.ApplyRequests != nil { + if last, ok := r.ApplyRequests.LastRequest(types.NewResourceReference(target.Name, target.Namespace)); ok { + target.Status.Applier = &last + } + } + + value := r.applierVerdict(ctx, target, gitProvider) + st.setValue(GitTargetConditionApplierConfigured, value) + return value, true +} + +// applierVerdict checks the applier in the source cluster. A check that could not be made keeps the +// verdict already published, since a blip reading the source cluster says nothing about the +// applier; with nothing published it is Unknown. +func (r *GitTargetReconciler) applierVerdict( + ctx context.Context, + target *configbutleraiv1alpha3.GitTarget, + gitProvider *configbutleraiv1alpha3.GitProvider, +) conditionValue { + ref := applierReference(target) + if r.ApplierClients == nil || gitProvider == nil { + return conditionValue{ + Status: metav1.ConditionUnknown, + Reason: GitTargetReasonNotChecked, + Message: fmt.Sprintf("Kustomization %s has not been checked: no source-cluster client is wired", ref), + } + } + client, err := r.ApplierClients(ctx, target.SourceCluster()) + if err != nil { + return keptOrUnknown(target, + fmt.Sprintf("Cannot reach the source cluster to check Kustomization %s: %v", ref, err)) + } + newFlux := r.newFlux + if newFlux == nil { + newFlux = applier.NewFlux + } + verdict := newFlux(client).Validate(ctx, ref, gitProvider.Spec.URL, target.Spec.Branch) + switch { + case verdict.OK: + return conditionValue{ + Status: metav1.ConditionTrue, + Reason: ReasonSucceeded, + Message: fmt.Sprintf("Kustomization %s applies this repository and branch, and can be asked to apply", ref), + } + case verdict.Err != nil: + return keptOrUnknown(target, verdict.Message) + default: + return conditionValue{Status: metav1.ConditionFalse, Reason: verdict.Reason, Message: verdict.Message} + } +} + +// keptOrUnknown is the ApplierConfigured value a check that could not be made publishes. +func keptOrUnknown(target *configbutleraiv1alpha3.GitTarget, message string) conditionValue { + for _, cond := range target.Status.Conditions { + if cond.Type == GitTargetConditionApplierConfigured { + return conditionValue{Status: cond.Status, Reason: cond.Reason, Message: cond.Message} + } + } + return conditionValue{Status: metav1.ConditionUnknown, Reason: ReasonProgressing, Message: message} +} + +// applierReference is spec.applier.kustomizationRef, its namespace defaulting to the GitTarget's. +func applierReference(target *configbutleraiv1alpha3.GitTarget) k8stypes.NamespacedName { + ref := target.Spec.Applier.KustomizationRef + namespace := ref.Namespace + if namespace == "" { + namespace = target.Namespace + } + return k8stypes.NamespacedName{Namespace: namespace, Name: ref.Name} +} + +// applierReadiness contributes ApplierConfigured to the trio. A Kustomization that applies another +// repository, or that the operator may not patch, needs a human; one that is missing may be on its +// way (Flux objects are often created after the GitTarget), and an unknown verdict is still +// settling. +func applierReadiness(rd *readiness, value conditionValue, present bool) { + if !present { + return + } + rd.stalledIf(value.Status == metav1.ConditionFalse && value.Reason != applier.ReasonApplierNotFound, + value.Reason, value.Message) + rd.progressingIf(value.Status == metav1.ConditionFalse, metav1.ConditionFalse, value.Reason, value.Message) + rd.progressingIf(value.Status == metav1.ConditionUnknown, metav1.ConditionUnknown, value.Reason, value.Message) +} diff --git a/internal/controller/gittarget_applier_test.go b/internal/controller/gittarget_applier_test.go new file mode 100644 index 00000000..4b2ce10c --- /dev/null +++ b/internal/controller/gittarget_applier_test.go @@ -0,0 +1,204 @@ +// SPDX-License-Identifier: Apache-2.0 + +package controller + +import ( + "context" + "errors" + "testing" + "time" + + meta "github.com/fluxcd/pkg/apis/meta" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + authorizationv1 "k8s.io/api/authorization/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/dynamic" + dynamicfake "k8s.io/client-go/dynamic/fake" + + configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/applier" + "github.com/ConfigButler/gitops-reverser/internal/types" +) + +const applierTestRepo = "https://github.com/acme/platform.git" + +// applierFluxObjects is a Kustomization flux-system/checkout applying a GitRepository on branch. +func applierFluxObjects(branch string) []runtime.Object { + ks := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "kustomize.toolkit.fluxcd.io/v1", + "kind": "Kustomization", + "metadata": map[string]any{"namespace": "flux-system", "name": "checkout"}, + "spec": map[string]any{ + "path": "./apps", + "sourceRef": map[string]any{"kind": "GitRepository", "name": "platform"}, + }, + }} + repo := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "source.toolkit.fluxcd.io/v1", + "kind": "GitRepository", + "metadata": map[string]any{"namespace": "flux-system", "name": "platform"}, + "spec": map[string]any{ + "url": "https://github.com/acme/platform", + "ref": map[string]any{"branch": branch}, + }, + }} + return []runtime.Object{ks, repo} +} + +func applierFakeSource(objs ...runtime.Object) *dynamicfake.FakeDynamicClient { + return dynamicfake.NewSimpleDynamicClientWithCustomListKinds(runtime.NewScheme(), + map[schema.GroupVersionResource]string{ + {Group: "kustomize.toolkit.fluxcd.io", Version: "v1", Resource: "kustomizations"}: "KustomizationList", + {Group: "source.toolkit.fluxcd.io", Version: "v1", Resource: "gitrepositories"}: "GitRepositoryList", + }, objs...) +} + +type fixedLedger map[types.ResourceReference]configbutleraiv1alpha3.GitTargetApplierStatus + +func (l fixedLedger) LastRequest(ref types.ResourceReference) (configbutleraiv1alpha3.GitTargetApplierStatus, bool) { + status, ok := l[ref] + return status, ok +} + +func applierTarget(applierSet bool) *configbutleraiv1alpha3.GitTarget { + target := &configbutleraiv1alpha3.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Name: "checkout", Namespace: "team-a"}, + Spec: configbutleraiv1alpha3.GitTargetSpec{Branch: "main", Path: "apps"}, + } + if applierSet { + target.Spec.Applier = &configbutleraiv1alpha3.GitTargetApplier{ + KustomizationRef: meta.NamespacedObjectReference{Namespace: "flux-system", Name: "checkout"}, + } + } + return target +} + +// applierReconciler checks appliers against a fake source cluster whose access review answers allowed. +func applierReconciler(source dynamic.Interface, sourceErr error, allowed bool) *GitTargetReconciler { + return &GitTargetReconciler{ + ApplierClients: func(context.Context, string) (dynamic.Interface, error) { return source, sourceErr }, + newFlux: func(client dynamic.Interface) *applier.Flux { + flux := applier.NewFlux(client) + flux.CanI = func(context.Context, authorizationv1.ResourceAttributes) (bool, error) { return allowed, nil } + return flux + }, + } +} + +func applierProvider() *configbutleraiv1alpha3.GitProvider { + return &configbutleraiv1alpha3.GitProvider{Spec: configbutleraiv1alpha3.GitProviderSpec{URL: applierTestRepo}} +} + +func TestObserveApplier_NoApplierRemovesTheCondition(t *testing.T) { + target := applierTarget(false) + target.Status.Conditions = []metav1.Condition{{Type: GitTargetConditionApplierConfigured, Status: "True"}} + target.Status.Applier = &configbutleraiv1alpha3.GitTargetApplierStatus{RequestedCommit: "old"} + st := beginStatus(nil, nil, target) + + _, present := applierReconciler(nil, nil, true).observeApplier(context.Background(), st, target, applierProvider()) + + assert.False(t, present) + assert.Nil(t, findCondition(target.Status.Conditions, GitTargetConditionApplierConfigured)) + assert.Nil(t, target.Status.Applier) +} + +func TestObserveApplier_Verdicts(t *testing.T) { + cases := []struct { + name string + objects []runtime.Object + allowed bool + wantStatus metav1.ConditionStatus + wantReason string + }{ + {name: "a Kustomization applying this repository and branch", objects: applierFluxObjects("main"), + allowed: true, wantStatus: metav1.ConditionTrue, wantReason: ReasonSucceeded}, + {name: "a Kustomization applying another branch", objects: applierFluxObjects("release"), + allowed: true, wantStatus: metav1.ConditionFalse, wantReason: applier.ReasonSourceMismatch}, + {name: "no such Kustomization", allowed: true, + wantStatus: metav1.ConditionFalse, wantReason: applier.ReasonApplierNotFound}, + {name: "an identity that may not patch it", objects: applierFluxObjects("main"), + wantStatus: metav1.ConditionFalse, wantReason: applier.ReasonForbidden}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + target := applierTarget(true) + st := beginStatus(nil, nil, target) + + value, present := applierReconciler(applierFakeSource(tc.objects...), nil, tc.allowed). + observeApplier(context.Background(), st, target, applierProvider()) + + require.True(t, present) + assert.Equal(t, tc.wantStatus, value.Status, value.Message) + assert.Equal(t, tc.wantReason, value.Reason) + published := findCondition(target.Status.Conditions, GitTargetConditionApplierConfigured) + require.NotNil(t, published) + assert.Equal(t, tc.wantReason, published.Reason) + }) + } +} + +// TestObserveApplier_AnUnreachableSourceKeepsThePublishedVerdict: a blip reading the source cluster +// says nothing about the applier, so it must not flip a True to Unknown and back on every hiccup. +func TestObserveApplier_AnUnreachableSourceKeepsThePublishedVerdict(t *testing.T) { + r := applierReconciler(nil, errors.New("connection refused"), true) + + target := applierTarget(true) + value, _ := r.observeApplier(context.Background(), beginStatus(nil, nil, target), target, applierProvider()) + assert.Equal(t, metav1.ConditionUnknown, value.Status, "nothing published yet") + + target = applierTarget(true) + target.Status.Conditions = []metav1.Condition{{ + Type: GitTargetConditionApplierConfigured, Status: metav1.ConditionTrue, Reason: ReasonSucceeded, + }} + value, _ = r.observeApplier(context.Background(), beginStatus(nil, nil, target), target, applierProvider()) + assert.Equal(t, metav1.ConditionTrue, value.Status, "the published verdict stands") +} + +func TestObserveApplier_PublishesTheLastRequest(t *testing.T) { + target := applierTarget(true) + at := metav1.NewTime(time.Date(2026, 10, 20, 10, 14, 3, 0, time.UTC)) + r := applierReconciler(applierFakeSource(applierFluxObjects("main")...), nil, true) + r.ApplyRequests = fixedLedger{types.NewResourceReference("checkout", "team-a"): { + RequestedCommit: "4f2c1ab9", RequestedAt: &at, Reason: configbutleraiv1alpha3.ApplyReasonForeignCommit, + }} + + r.observeApplier(context.Background(), beginStatus(nil, nil, target), target, applierProvider()) + + require.NotNil(t, target.Status.Applier) + assert.Equal(t, "4f2c1ab9", target.Status.Applier.RequestedCommit) + assert.Equal(t, configbutleraiv1alpha3.ApplyReasonForeignCommit, target.Status.Applier.Reason) +} + +func TestApplierReadiness_StallsOnlyWhatNeedsAHuman(t *testing.T) { + cases := []struct { + name string + value conditionValue + present bool + wantConverged bool + wantStalled bool + }{ + {name: "no applier", present: false, wantConverged: true}, + {name: "configured", present: true, wantConverged: true, + value: conditionValue{Status: metav1.ConditionTrue, Reason: ReasonSucceeded}}, + {name: "not created yet", present: true, + value: conditionValue{Status: metav1.ConditionFalse, Reason: applier.ReasonApplierNotFound}}, + {name: "another branch", present: true, wantStalled: true, + value: conditionValue{Status: metav1.ConditionFalse, Reason: applier.ReasonSourceMismatch}}, + {name: "forbidden", present: true, wantStalled: true, + value: conditionValue{Status: metav1.ConditionFalse, Reason: applier.ReasonForbidden}}, + {name: "not checked yet", present: true, + value: conditionValue{Status: metav1.ConditionUnknown, Reason: ReasonProgressing}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + rd := newGitTargetReadiness() + applierReadiness(rd, tc.value, tc.present) + assert.Equal(t, tc.wantConverged, rd.converged()) + assert.Equal(t, tc.wantStalled, !rd.converged() && !rd.converging()) + }) + } +} diff --git a/internal/controller/gittarget_controller.go b/internal/controller/gittarget_controller.go index 17c382e7..91cd6642 100644 --- a/internal/controller/gittarget_controller.go +++ b/internal/controller/gittarget_controller.go @@ -18,6 +18,7 @@ import ( metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" k8stypes "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" "k8s.io/client-go/tools/record" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/builder" @@ -30,6 +31,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/source" configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/applier" "github.com/ConfigButler/gitops-reverser/internal/git" "github.com/ConfigButler/gitops-reverser/internal/reconcile" "github.com/ConfigButler/gitops-reverser/internal/telemetry" @@ -151,6 +153,15 @@ type GitTargetReconciler struct { // clock is the source of "now" for the publication ledger, so a test can hold time still // across a reconcile that is measured in wall clock. Nil means time.Now. clock func() time.Time + + // ApplierClients resolves the dynamic client of a GitTarget's source cluster, where its + // spec.applier lives, and ApplyRequests is the ledger of requests delivered to it. Nil leaves + // ApplierConfigured Unknown and status.applier unpublished (tests, and the CLI). + ApplierClients applier.SourceClients + ApplyRequests ApplyRequestLedger + // newFlux builds the client that checks a Kustomization; nil is applier.NewFlux. A seam for + // tests, whose fake source cluster cannot answer an access review. + newFlux func(dynamic.Interface) *applier.Flux } // clockNow is the reconciler's wall clock. @@ -167,6 +178,10 @@ func (r *GitTargetReconciler) clockNow() time.Time { // +kubebuilder:rbac:groups="",resources=secrets,verbs=get;create;update // +kubebuilder:rbac:groups="",resources=events,verbs=create;patch +// The Flux kinds a spec.applier names are NOT granted here: no GitTarget needs them until one names +// an applier, and an install that does not use Flux should not hold patch rights on Flux objects. +// config/rbac/flux-applier-role.yaml and the chart's applierAccess.flux grant them. + // newGitTargetReadiness starts the accumulator that owns this kind's kstatus trio. Its converged // outcome is what Ready reports when no gate objected. func newGitTargetReadiness() *readiness { @@ -281,11 +296,13 @@ func (r *GitTargetReconciler) Reconcile(ctx context.Context, req ctrl.Request) ( st.setValue(GitTargetConditionRenderMatchesLive, observed.axes.Render) refs := r.publishReferenceReadiness(st, &target, providerNS, gitProvider, sourceProvider) + applierValue, hasApplier := r.observeApplier(ctx, st, &target, gitProvider) rd := newGitTargetReadiness() convergeAsSuspended(rd, &target) gitTargetReadinessGates(rd, observed, r.parentBranchReadiness(&target, providerNS, repoIdentityOf(gitProvider)), refs.gitProvider, refs.clusterProvider, refs.sourceCluster, publicationCondition(publication)) + applierReadiness(rd, applierValue, hasApplier) parentRecoveryReadiness(rd, recoveringParent) st.applyReadiness(rd) diff --git a/internal/git/apply_request.go b/internal/git/apply_request.go new file mode 100644 index 00000000..4f9943dc --- /dev/null +++ b/internal/git/apply_request.go @@ -0,0 +1,30 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +import ( + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// ApplyRequest is one request, decided by a branch worker, that the appliers of some GitTargets on +// its branch apply a commit. The worker decides WHEN; delivering it is the ApplyRequester's job, off +// the worker's queue, so a slow applier never holds the branch. +type ApplyRequest struct { + // Branch is the branch the request is about. + Branch BranchKey + // Reason is why the appliers are asked. + Reason v1alpha3.ApplyReason + // Commit is the commit to apply: the branch tip when the worker decided, which the worker has + // pushed or proved by a fetch. Empty when the remote does not carry the branch. + Commit string + // Targets are the GitTargets whose appliers are asked. Several may name one applier; the + // requester asks it once. + Targets []itypes.ResourceReference +} + +// ApplyRequester delivers apply requests. RequestApply must not block: it is called on the branch +// worker's loop goroutine. +type ApplyRequester interface { + RequestApply(req ApplyRequest) +} diff --git a/internal/git/apply_trigger.go b/internal/git/apply_trigger.go new file mode 100644 index 00000000..de30d0ec --- /dev/null +++ b/internal/git/apply_trigger.go @@ -0,0 +1,335 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +import ( + "context" + "fmt" + "sort" + "sync" + "time" + + "sigs.k8s.io/controller-runtime/pkg/client" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// This file is the branch worker's half of the apply trigger: deciding WHEN the appliers of the +// GitTargets on this branch may apply. See docs/design/push-notification-and-reconcile-trigger.md +// part 3. +// +// An apply is safe only when nothing live is missing from the commit being applied, and only this +// worker can see the edits that are live and not yet pushed: they are its open commit window and its +// pending writes. So a request goes out only on a QUIET branch, with neither, and the applier's own +// triggers are switched off by the installer. Three things owe a request: +// +// - ForeignCommit: an observation of the remote found the branch somewhere this worker did not +// push it. Every applier on the branch is asked as soon as the branch is quiet, and an open +// window is closed rather than waited out, because a user who keeps editing would otherwise +// hold a Git-side change back for as long as they type. +// - Refusal: a refused edit on a target with spec.onRefusal: Reapply (refusal_reapply.go). +// - CatchUp: this worker's own pushes. Flux applies on its own timers and on a restart from +// whatever artifact it holds, and an artifact behind these pushes would revert edits already in +// Git. Asked only once the branch has stayed quiet for the target's spec.applier.quietPeriod, so +// the apply does not land in the middle of an editing session. + +// applyDebt is what the branch's appliers are owed because of what was OBSERVED about the remote. It +// is fed from publishObservation, the one point every observation passes through, which can run +// before the loop starts (the first clone), hence the lock. +type applyDebt struct { + mu sync.Mutex + // known is whether anything has been observed yet, and tip is the last commit observed. Empty + // means the remote does not carry the branch. + known bool + tip string + // foreign and catchUp are owed until the loop takes them on a quiet branch. + foreign bool + catchUp bool +} + +// observe classifies one observation. The first one is a baseline, not a move: nothing says what +// the appliers last applied, so they are owed a catch-up, never a ForeignCommit. A request on a +// restart would otherwise apply the tip before the watches have captured the edits made while the +// operator was down. +// +// After that, a commit that differs from the last one observed is either this worker's push, proved +// by the push itself, or somebody else's, because every other observation is a fetch. When in doubt +// the classification errs towards foreign, which costs one apply of the tip on a quiet branch. +func (d *applyDebt) observe(observed RemoteObservation) { + d.mu.Lock() + defer d.mu.Unlock() + if !d.known { + d.known, d.tip, d.catchUp = true, observed.Commit, true + return + } + if observed.Commit == d.tip { + return + } + d.tip = observed.Commit + if observed.By == ObservedByPush { + d.catchUp = true + return + } + d.foreign = true +} + +// take returns what is owed and clears it. The caller restores it if it cannot act on it. +func (d *applyDebt) take() (string, bool, bool) { + d.mu.Lock() + defer d.mu.Unlock() + tip, foreign, catchUp := d.tip, d.foreign, d.catchUp + d.foreign, d.catchUp = false, false + return tip, foreign, catchUp +} + +// restore puts back what take returned, for a caller that could not act on it. +func (d *applyDebt) restore(foreign, catchUp bool) { + d.mu.Lock() + defer d.mu.Unlock() + d.foreign = d.foreign || foreign + d.catchUp = d.catchUp || catchUp +} + +// foreignOwed reports whether a ForeignCommit is owed, without taking it. +func (d *applyDebt) foreignOwed() bool { + d.mu.Lock() + defer d.mu.Unlock() + return d.foreign +} + +// applierRetryInterval is how soon a wake retries after it could not list the branch's appliers. +const applierRetryInterval = 10 * time.Second + +// branchApplier is one GitTarget on this branch that names an applier. +type branchApplier struct { + target itypes.ResourceReference + quietPeriod time.Duration +} + +// appliersOnBranch lists the GitTargets on this worker's (provider, branch) that name an applier. A +// GitTarget names its GitProvider in its own namespace, so the provider's namespace is the only one +// to look in. The client is cached, so this costs no round trip. +func (w *BranchWorker) appliersOnBranch(ctx context.Context) ([]branchApplier, error) { + var targets v1alpha3.GitTargetList + if err := w.Client.List(ctx, &targets, client.InNamespace(w.GitProviderNamespace)); err != nil { + return nil, fmt.Errorf("list GitTargets: %w", err) + } + appliers := make([]branchApplier, 0, len(targets.Items)) + for i := range targets.Items { + target := &targets.Items[i] + if !target.DeletionTimestamp.IsZero() || target.Spec.Applier == nil || + target.Spec.GitProviderRef.Name != w.GitProviderRef || target.Spec.Branch != w.Branch { + continue + } + appliers = append(appliers, branchApplier{ + target: itypes.NewResourceReference(target.Name, target.Namespace), + quietPeriod: target.Spec.Applier.EffectiveQuietPeriod(), + }) + } + sort.Slice(appliers, func(i, j int) bool { return appliers[i].target.String() < appliers[j].target.String() }) + return appliers, nil +} + +// writeBusy reports whether the loop holds anything live that is not yet on the remote, or is in a +// state where it cannot say. A request is never sent while it does. +func (l *branchWorkerEventLoop) writeBusy() bool { + return l.openWindow != nil || len(l.pendingWrites) > 0 || len(l.deferredHeals) > 0 || + l.recovery.active || l.w.worktreeDirty() +} + +// closeWindowForForeignCommit closes the open commit window early when the branch has moved under +// it, or a Git host has said it may have, so its writes are published, and replayed onto the move, +// now rather than when the window's timers allow. See the file comment for why not wait. +func (l *branchWorkerEventLoop) closeWindowForForeignCommit() { + hinted := l.pushHinted + l.pushHinted = false + if l.openWindow == nil || l.w.applyRequester == nil { + return + } + if !hinted && !l.w.debt.foreignOwed() { + return + } + l.w.Log.Info("The branch moved, or may have; closing the commit window to publish it now", + "branch", l.w.Branch) + l.finalizeOpenWindowWithReason(windowFinalizeReasonForeignCommit) +} + +// serviceAppliers sends whatever apply requests are owed, if the branch is quiet. It runs at the end +// of every wake, after the wake's writes have been decided and committed, so "quiet" is what the +// wake left behind. +func (l *branchWorkerEventLoop) serviceAppliers() { + w := l.w + if w.applyRequester == nil { + return + } + if l.writeBusy() { + l.quietSince = time.Time{} + l.stopApplierTimer() + return + } + now := time.Now() + if l.quietSince.IsZero() { + l.quietSince = now + } + if len(w.eventQueue) > 0 { + // More work is already queued and the next wake will look again. The branch is still quiet + // until that work opens a window, so quietSince is kept. + return + } + + tip, foreign, catchUp := w.debt.take() + if !foreign && !catchUp && len(l.reapplyOwed) == 0 && len(l.catchUpOwed) == 0 { + return + } + declared, ok := l.declaredAppliers(foreign, catchUp) + if !ok { + return + } + if catchUp { + l.oweCatchUp(declared) + } + if foreign && len(declared) > 0 { + // The tip is the foreign commit with every pending write replayed onto it: the branch is + // quiet, so anything that was pending has been published on top. Every applier on the + // branch is asked, which also answers whatever else they were owed. + l.requestApply(v1alpha3.ApplyReasonForeignCommit, tip, sortedTargets(declared)) + clear(l.reapplyOwed) + clear(l.catchUpOwed) + return + } + l.serviceRefusals(tip, declared) + l.serviceCatchUp(tip, declared, now) +} + +// declaredAppliers maps every GitTarget on this branch that names an applier to its quiet period. +// On a failure to list them, what was taken is put back and a retry is armed. +func (l *branchWorkerEventLoop) declaredAppliers( + foreign, catchUp bool, +) (map[itypes.ResourceReference]time.Duration, bool) { + appliers, err := l.w.appliersOnBranch(l.w.ctx) + if err != nil { + l.w.debt.restore(foreign, catchUp) + l.w.Log.Error(err, "Cannot list the appliers on this branch; retrying", "branch", l.w.Branch) + l.resetApplierTimer(applierRetryInterval) + return nil, false + } + declared := make(map[itypes.ResourceReference]time.Duration, len(appliers)) + for _, applier := range appliers { + declared[applier.target] = applier.quietPeriod + } + return declared, true +} + +// oweCatchUp records that every declared applier is owed a catch-up with this worker's pushes. +func (l *branchWorkerEventLoop) oweCatchUp(declared map[itypes.ResourceReference]time.Duration) { + if l.catchUpOwed == nil { + l.catchUpOwed = map[itypes.ResourceReference]time.Duration{} + } + for target, quietPeriod := range declared { + l.catchUpOwed[target] = quietPeriod + } +} + +// serviceRefusals sends each owed Refusal request to its target's applier alone. +func (l *branchWorkerEventLoop) serviceRefusals( + tip string, declared map[itypes.ResourceReference]time.Duration, +) { + refusals := make([]itypes.ResourceReference, 0, len(l.reapplyOwed)) + for target := range l.reapplyOwed { + refusals = append(refusals, target) + } + sortReferences(refusals) + for _, target := range refusals { + delete(l.reapplyOwed, target) + if _, ok := declared[target]; !ok { + continue // its applier was removed since the refusal; consent went with it + } + // A reapply applies the tip, which is everything a catch-up would have asked for. + delete(l.catchUpOwed, target) + l.requestApply(v1alpha3.ApplyReasonRefusal, tip, []itypes.ResourceReference{target}) + } +} + +// serviceCatchUp asks every applier whose quiet period has run out to catch up, and arms the timer +// for the earliest one still waiting. +func (l *branchWorkerEventLoop) serviceCatchUp( + tip string, declared map[itypes.ResourceReference]time.Duration, now time.Time, +) { + var due []itypes.ResourceReference + var earliest time.Time + for target, quietPeriod := range l.catchUpOwed { + if _, ok := declared[target]; !ok { + delete(l.catchUpOwed, target) + continue + } + at := l.quietSince.Add(quietPeriod) + if !now.Before(at) { + due = append(due, target) + delete(l.catchUpOwed, target) + continue + } + if earliest.IsZero() || at.Before(earliest) { + earliest = at + } + } + if len(due) > 0 { + sortReferences(due) + l.requestApply(v1alpha3.ApplyReasonCatchUp, tip, due) + } + if earliest.IsZero() { + l.stopApplierTimer() + return + } + l.resetApplierTimer(earliest.Sub(now)) +} + +// requestApply hands one request to the requester, which delivers it off this goroutine. +func (l *branchWorkerEventLoop) requestApply( + reason v1alpha3.ApplyReason, commit string, targets []itypes.ResourceReference, +) { + l.w.Log.Info("Asking the appliers to apply", + "branch", l.w.Branch, "reason", string(reason), "commit", commit, "gitTargets", len(targets)) + l.w.applyRequester.RequestApply(ApplyRequest{ + Branch: BranchKey{ + RepoNamespace: l.w.GitProviderNamespace, + RepoName: l.w.GitProviderRef, + Branch: l.w.Branch, + }, + Reason: reason, + Commit: commit, + Targets: targets, + }) +} + +func (l *branchWorkerEventLoop) applierTimerC() <-chan time.Time { + if l.applierTimer == nil { + return nil + } + return l.applierTimer.C +} + +func (l *branchWorkerEventLoop) resetApplierTimer(delay time.Duration) { + l.stopApplierTimer() + l.applierTimer = time.NewTimer(max(delay, 0)) +} + +func (l *branchWorkerEventLoop) stopApplierTimer() { + if l.applierTimer != nil { + l.applierTimer.Stop() + l.applierTimer = nil + } +} + +func sortedTargets(set map[itypes.ResourceReference]time.Duration) []itypes.ResourceReference { + targets := make([]itypes.ResourceReference, 0, len(set)) + for target := range set { + targets = append(targets, target) + } + sortReferences(targets) + return targets +} + +func sortReferences(refs []itypes.ResourceReference) { + sort.Slice(refs, func(i, j int) bool { return refs[i].String() < refs[j].String() }) +} diff --git a/internal/git/apply_trigger_test.go b/internal/git/apply_trigger_test.go new file mode 100644 index 00000000..93d34ec2 --- /dev/null +++ b/internal/git/apply_trigger_test.go @@ -0,0 +1,368 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +// The branch worker's half of the apply trigger, from +// docs/design/push-notification-and-reconcile-trigger.md part 3: what the appliers are owed, and the +// rule that nothing is asked while anything live is unpublished. + +import ( + "context" + "testing" + "time" + + meta "github.com/fluxcd/pkg/apis/meta" + "github.com/go-logr/logr" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + configv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +func TestApplyDebt_FirstObservationIsABaseline(t *testing.T) { + var debt applyDebt + debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByFetch}) + + tip, foreign, catchUp := debt.take() + assert.Equal(t, "aaa", tip) + assert.False(t, foreign, "nothing says the applier has not applied what was there at startup") + assert.True(t, catchUp, "but nothing says it has, so it is owed a catch-up") +} + +func TestApplyDebt_ClassifiesEachMove(t *testing.T) { + cases := []struct { + name string + next RemoteObservation + wantForeign bool + wantCatchUp bool + }{ + {name: "an unchanged tip owes nothing", next: RemoteObservation{Commit: "aaa", By: ObservedByFetch}}, + {name: "our own push owes a catch-up", next: RemoteObservation{Commit: "bbb", By: ObservedByPush}, + wantCatchUp: true}, + {name: "a fetch that found the branch moved owes a ForeignCommit", + next: RemoteObservation{Commit: "bbb", By: ObservedByFetch}, wantForeign: true}, + {name: "a deleted branch is a move like any other", + next: RemoteObservation{By: ObservedByFetch}, wantForeign: true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + var debt applyDebt + debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByFetch}) + debt.take() + + debt.observe(tc.next) + tip, foreign, catchUp := debt.take() + assert.Equal(t, tc.next.Commit, tip) + assert.Equal(t, tc.wantForeign, foreign) + assert.Equal(t, tc.wantCatchUp, catchUp) + }) + } +} + +func TestApplyDebt_RestorePutsBackWhatWasTaken(t *testing.T) { + var debt applyDebt + debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByFetch}) + debt.observe(RemoteObservation{Commit: "bbb", By: ObservedByFetch}) + _, foreign, catchUp := debt.take() + require.True(t, foreign) + + debt.restore(foreign, catchUp) + assert.True(t, debt.foreignOwed()) +} + +// applierTestTarget is a GitTarget on the trigger worker's provider and branch unless overridden. +func applierTestTarget(name string, applier bool, mutate ...func(*configv1alpha3.GitTarget)) *configv1alpha3.GitTarget { + target := &configv1alpha3.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "team-a"}, + Spec: configv1alpha3.GitTargetSpec{ + GitProviderRef: meta.LocalObjectReference{Name: "test-provider"}, + Branch: "main", + Path: "apps/" + name, + }, + } + if applier { + target.Spec.Applier = testApplier(name) + } + for _, m := range mutate { + m(target) + } + return target +} + +// applyTriggerLoop is an event loop on provider team-a/test-provider, branch main, whose fake client +// holds the given GitTargets and whose requests are recorded. +func applyTriggerLoop(t *testing.T, targets ...client.Object) (*branchWorkerEventLoop, *recordingRequester) { + t.Helper() + scheme := runtime.NewScheme() + require.NoError(t, clientgoscheme.AddToScheme(scheme)) + require.NoError(t, configv1alpha3.AddToScheme(scheme)) + + w := newMetricsTestWorker() + w.GitProviderNamespace = "team-a" + w.Client = fake.NewClientBuilder().WithScheme(scheme).WithObjects(targets...).Build() + w.Log = logr.Discard() + w.ctx = context.Background() + requester := &recordingRequester{} + w.applyRequester = requester + loop := newBranchWorkerEventLoop(w, time.Minute) + t.Cleanup(loop.stopTimers) + return loop, requester +} + +// owesForeign makes the worker's debt hold a ForeignCommit at commit "bbb". +func owesForeign(w *BranchWorker) { + w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByFetch}) + w.debt.take() + w.debt.observe(RemoteObservation{Commit: "bbb", By: ObservedByFetch}) +} + +func TestServiceAppliers_ForeignCommitAsksEveryApplierOnTheBranch(t *testing.T) { + loop, requester := applyTriggerLoop(t, + applierTestTarget("alpha", true), + applierTestTarget("bravo", true, func(g *configv1alpha3.GitTarget) { g.Spec.Suspend = true }), + applierTestTarget("no-applier", false), + applierTestTarget("other-branch", true, func(g *configv1alpha3.GitTarget) { g.Spec.Branch = "release" }), + applierTestTarget("other-provider", true, func(g *configv1alpha3.GitTarget) { + g.Spec.GitProviderRef.Name = "elsewhere" + }), + ) + owesForeign(loop.w) + + loop.serviceAppliers() + + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, requests[0].Reason) + assert.Equal(t, "bbb", requests[0].Commit) + assert.Equal(t, BranchKey{RepoNamespace: "team-a", RepoName: "test-provider", Branch: "main"}, requests[0].Branch) + assert.Equal(t, []itypes.ResourceReference{ + itypes.NewResourceReference("alpha", "team-a"), + itypes.NewResourceReference("bravo", "team-a"), + }, requests[0].Targets, "a suspended target stops writing, not deployments") +} + +func TestServiceAppliers_NothingIsSentWhileAnythingLiveIsUnpublished(t *testing.T) { + cases := []struct { + name string + busy func(l *branchWorkerEventLoop) + }{ + {name: "an open commit window", busy: func(l *branchWorkerEventLoop) { l.openWindow = &openWindow{} }}, + {name: "a pending write", busy: func(l *branchWorkerEventLoop) { l.pendingWrites = []PendingWrite{{}} }}, + {name: "a parked heal", busy: func(l *branchWorkerEventLoop) { l.deferredHeals = []*ResyncRequest{{}} }}, + {name: "a worktree a failed write left dirty", busy: func(l *branchWorkerEventLoop) { + l.w.markWorktreeDirty("test") + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) + owesForeign(loop.w) + tc.busy(loop) + + loop.serviceAppliers() + + assert.Empty(t, requester.taken()) + assert.True(t, loop.quietSince.IsZero(), "a busy branch is not quiet") + assert.True(t, loop.w.debt.foreignOwed(), "and the debt waits for it") + }) + } +} + +func TestServiceAppliers_CatchUpWaitsForTheQuietPeriod(t *testing.T) { + loop, requester := applyTriggerLoop(t, + applierTestTarget("soon", true, func(g *configv1alpha3.GitTarget) { + g.Spec.Applier.QuietPeriod = &metav1.Duration{Duration: time.Minute} + }), + applierTestTarget("late", true, func(g *configv1alpha3.GitTarget) { + g.Spec.Applier.QuietPeriod = &metav1.Duration{Duration: time.Hour} + }), + ) + loop.w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByPush}) + + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "the branch has only just become quiet") + require.NotNil(t, loop.applierTimer, "the earliest catch-up is armed") + require.Len(t, loop.catchUpOwed, 2) + + // Two minutes of quiet: the one-minute applier is due, the one-hour one is not. + loop.quietSince = time.Now().Add(-2 * time.Minute) + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, configv1alpha3.ApplyReasonCatchUp, requests[0].Reason) + assert.Equal(t, "aaa", requests[0].Commit) + assert.Equal(t, []itypes.ResourceReference{itypes.NewResourceReference("soon", "team-a")}, requests[0].Targets) + assert.Contains(t, loop.catchUpOwed, itypes.NewResourceReference("late", "team-a")) + assert.NotNil(t, loop.applierTimer) +} + +func TestServiceAppliers_AWriteRestartsTheQuietPeriod(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) + loop.w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByPush}) + loop.serviceAppliers() + loop.quietSince = time.Now().Add(-time.Hour) + + loop.openWindow = &openWindow{} + loop.serviceAppliers() + assert.True(t, loop.quietSince.IsZero()) + assert.Nil(t, loop.applierTimer) + + loop.openWindow = nil + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "the quiet period restarted when the write arrived") + assert.WithinDuration(t, time.Now(), loop.quietSince, time.Second) +} + +func TestServiceAppliers_AForeignCommitAnswersEverythingElseOwed(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) + loop.oweReapply(refusalKeyFor(itypes.NewResourceReference("alpha", "team-a"), "deployments"), "refused", "obs") + owesForeign(loop.w) + + loop.serviceAppliers() + + requests := requester.taken() + require.Len(t, requests, 1, "one apply of the tip covers the refusal and the catch-up") + assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, requests[0].Reason) + assert.Empty(t, loop.reapplyOwed) + assert.Empty(t, loop.catchUpOwed) +} + +func TestServiceAppliers_NoApplierOnTheBranchClearsTheDebt(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("plain", false)) + owesForeign(loop.w) + loop.w.debt.observe(RemoteObservation{Commit: "ccc", By: ObservedByPush}) + + loop.serviceAppliers() + + assert.Empty(t, requester.taken()) + assert.False(t, loop.w.debt.foreignOwed()) + assert.Empty(t, loop.catchUpOwed) + assert.Nil(t, loop.applierTimer) +} + +func TestServiceAppliers_NoRequesterSwitchesTheTriggerOff(t *testing.T) { + loop, _ := applyTriggerLoop(t, applierTestTarget("alpha", true)) + loop.w.applyRequester = nil + owesForeign(loop.w) + + loop.serviceAppliers() + + assert.True(t, loop.w.debt.foreignOwed(), "nothing took the debt") +} + +// TestCloseWindowForForeignCommit_PublishesTheOpenWindowNow: a window closes on silence, so a user +// who keeps editing would hold a foreign commit back for as long as they type. It is closed instead, +// and its write published and replayed onto the move. +func TestCloseWindowForForeignCommit_PublishesTheOpenWindowNow(t *testing.T) { + cases := []struct { + name string + cause func(l *branchWorkerEventLoop) + }{ + {name: "a foreign commit observed", cause: func(l *branchWorkerEventLoop) { owesForeign(l.w) }}, + {name: "a Git host's push notice on a busy branch", cause: func(l *branchWorkerEventLoop) { + l.handleRefreshRequest(&RefreshRequest{PushNotice: true}) + }}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + worker, _, _ := setupCommitPushSplitWorker(t) + createPlainGitTarget(t, worker, "team-a", "team-a") + worker.applyRequester = &recordingRequester{} + loop := newBranchWorkerEventLoop(worker, time.Hour) + t.Cleanup(loop.stopTimers) + + loop.handleQueueItem(WorkItem{Request: &WriteRequest{ + Events: []Event{configMapTargetEvent("edited", "alice", "team-a")}, + CommitMode: CommitModePerEvent, + }}) + require.NotNil(t, loop.openWindow, "an hour-long window is still open") + + tc.cause(loop) + loop.closeWindowForForeignCommit() + + assert.Nil(t, loop.openWindow) + assert.Len(t, loop.pendingWrites, 1, "the window's write is decided, to be published") + }) + } +} + +// applierLedgerFixture is a real Git server, a GitTarget naming an applier, and an event loop whose +// requests are recorded. +func applierLedgerFixture(t *testing.T, slug string) (*ledgerFixture, *branchWorkerEventLoop, *recordingRequester) { + t.Helper() + f := newLedgerFixture(t, slug, true) + f.createLedgerTarget("team-a", nil) + target := &configv1alpha3.GitTarget{} + require.NoError(t, f.worker.Client.Get(f.worker.ctx, + client.ObjectKey{Name: ledgerTargetName, Namespace: "default"}, target)) + target.Spec.Applier = testApplier("apps") + require.NoError(t, f.worker.Client.Update(f.worker.ctx, target)) + + requester := &recordingRequester{} + f.worker.applyRequester = requester + loop := newBranchWorkerEventLoop(f.worker, time.Hour) + t.Cleanup(loop.stopTimers) + return f, loop, requester +} + +func foreignRequests(requests []ApplyRequest) []ApplyRequest { + var foreign []ApplyRequest + for _, req := range requests { + if req.Reason == configv1alpha3.ApplyReasonForeignCommit { + foreign = append(foreign, req) + } + } + return foreign +} + +// TestApplyTrigger_APushNoticeFindsAForeignCommit is §3.3 end to end over a real Git server: the +// notice names nothing the worker trusts, the advertisement shows a tip the worker did not push, +// and the request names that tip. +func TestApplyTrigger_APushNoticeFindsAForeignCommit(t *testing.T) { + f, loop, requester := applierLedgerFixture(t, "apply-trigger-notice") + f.publish("ours") + loop.endWake(0) + assert.Empty(t, foreignRequests(requester.taken()), "the worker's own push is not foreign") + + f.contend("OUTSIDE.md", "from-another-writer\n") + loop.handleRefreshRequest(&RefreshRequest{PushNotice: true}) + loop.endWake(0) + + requests := foreignRequests(requester.taken()) + require.Len(t, requests, 1) + assert.Equal(t, gitOut(t, f.repoDir, "rev-parse", "main"), requests[0].Commit) + assert.Equal(t, []itypes.ResourceReference{itypes.NewResourceReference(ledgerTargetName, "default")}, + requests[0].Targets) + + // A duplicate delivery of the same push is one more advertisement and no second request. + loop.handleRefreshRequest(&RefreshRequest{PushNotice: true}) + loop.endWake(0) + assert.Empty(t, foreignRequests(requester.taken())) +} + +// TestApplyTrigger_ARejectedPushOwesTheReplayedTip is §3.4: the foreign commit is found by the +// compare-and-swap, the pending write is replayed onto it, and the request names the replayed tip, +// which carries the API side's answer. +func TestApplyTrigger_ARejectedPushOwesTheReplayedTip(t *testing.T) { + f, loop, requester := applierLedgerFixture(t, "apply-trigger-rejected") + f.publish("prime") + loop.endWake(0) + requester.taken() + + f.commit("edited") + f.contend("OUTSIDE.md", "from-another-writer\n") + f.push() + loop.endWake(0) + + requests := foreignRequests(requester.taken()) + require.Len(t, requests, 1) + head := gitOut(t, f.repoDir, "rev-parse", "main") + assert.Equal(t, head, requests[0].Commit, "the replayed tip, not the foreign commit under it") + assert.Equal(t, "from-another-writer", gitOut(t, f.repoDir, "show", head+":OUTSIDE.md")) +} diff --git a/internal/git/branch_worker.go b/internal/git/branch_worker.go index 2b2791dd..a2048d7e 100644 --- a/internal/git/branch_worker.go +++ b/internal/git/branch_worker.go @@ -135,6 +135,13 @@ type BranchWorker struct { // simply not shared. remoteReporter RemoteReporter + // applyRequester delivers the apply requests this worker decides (apply_trigger.go), and debt is + // what the branch's appliers are owed from what was observed about the remote. Set by the + // WorkerManager before Start; a nil requester switches the apply trigger off (the CLI, and tests + // that do not exercise it). + applyRequester ApplyRequester + debt applyDebt + // Event processing eventQueue chan WorkItem ctx context.Context @@ -205,19 +212,19 @@ type BranchWorker struct { // See docs/design/gittarget-branch-worker-pending-writes.md. checkoutApplied atomic.Int64 - // lastRefusalTouch records when this worker last pushed an empty commit for a GitTarget, keyed - // by "namespace/name", so refusalTouchInterval can floor the rate. It is guarded by its own + // lastReapply records when this worker last owed a GitTarget's applier a Refusal request, keyed + // by "namespace/name", so refusalReapplyInterval can floor the rate. It is guarded by its own // mutex rather than repoMu: the decision is taken before any repository work starts, and // borrowing the repository lock for it would order the two for no reason. - refusalTouchMu sync.Mutex - lastRefusalTouch map[string]time.Time + refusalMu sync.Mutex + lastReapply map[string]time.Time // coveredRefusal records, per GitTarget AND WATCHED COLLECTION, the refusal observation the last - // empty commit was made for: a digest of the objects that were refused together with the + // reapply request was owed for: a digest of the objects that were refused together with the // issues raised about them. A recheck that observes exactly the same thing is already covered - // by that commit, so it is not another trigger. It is dropped when a resync for that collection is + // by that request, so it is not another trigger. It is dropped when a resync for that collection is // ACCEPTED — see refusalRecovered, which is what makes a live edit re-made after a revert // count as new, and what keeps one watched type's success from speaking for another's. - // Guarded by refusalTouchMu with the rate limit, because the two are read together. + // Guarded by refusalMu with the rate limit, because the two are read together. coveredRefusal map[refusalKey]string // branchBufferMaxBytes caps the retained in-memory event data; tripped on @@ -368,6 +375,7 @@ const ( windowFinalizeReasonIdentityChange windowFinalizeReason = "author-or-target-change" windowFinalizeReasonBufferLimit windowFinalizeReason = "buffer-limit" windowFinalizeReasonShutdown windowFinalizeReason = "shutdown" + windowFinalizeReasonForeignCommit windowFinalizeReason = "foreign-commit" ) // NewBranchWorker creates a worker for a (provider, branch) combination. A zero @@ -1135,17 +1143,30 @@ type branchWorkerEventLoop struct { pendingCRs map[commitRequestID]*pendingCommitRequest // crSeq numbers registrations, so competing requests attach first come, first served. crSeq uint64 - // refusalTimer fires at the earliest deadline in refusalPending. See refusal_touch.go. + // refusalTimer fires at the earliest deadline in refusalPending. See refusal_reapply.go. refusalTimer *time.Timer - // refusalPending holds one entry per GitTarget with a coalesced commit due. + // refusalPending holds one entry per GitTarget and watched collection with a coalesced reapply + // request due. // // It is a MAP because one branch worker serves every target on its (provider, branch), while // the rate limit and the consent check are both per target. A single slot let one target // overwrite another's pending work: A queues, B replaces it, B is then suspended, and A's - // authorized commit is gone with nothing to report it. Execution still coalesces — one commit - // on the branch satisfies every entry due at that moment — but the INTENT is kept per target - // and per watched collection, so one collection recovering cancels only its own obligation. - refusalPending map[refusalKey]pendingRefusalTouch + // authorized request is gone with nothing to report it. + refusalPending map[refusalKey]pendingRefusalReapply + // reapplyOwed holds the GitTargets whose applier is owed a Refusal request, with the refusal it + // answers, until serviceAppliers sends it on a quiet branch. See apply_trigger.go. + reapplyOwed map[itypes.ResourceReference]string + + // catchUpOwed holds, per GitTarget with an applier on this branch, how long the branch must stay + // quiet before that applier is asked to catch up with this worker's own pushes. quietSince is + // when the branch last became quiet (zero while it is busy), and applierTimer fires at the + // earliest catch-up still due. See apply_trigger.go. + catchUpOwed map[itypes.ResourceReference]time.Duration + quietSince time.Time + applierTimer *time.Timer + // pushHinted records that a Git host reported a push while the branch was busy. The wake closes + // the open window so its publication finds the move (closeWindowForForeignCommit). + pushHinted bool // attachTimer fires at the earliest attach deadline of a waiting CommitRequest, so a request // that no window reaches still resolves on time. An attached request needs no timer of its own: @@ -1201,10 +1222,13 @@ func (l *branchWorkerEventLoop) run() { // serviceCommitRequests below. case <-refusalC: l.refusalTimer = nil - l.flushPendingRefusalTouch() + l.flushPendingReapply() case <-l.retryTimerC(): l.retry.timer = nil l.runRetry() + case <-l.applierTimerC(): + l.applierTimer = nil + // serviceAppliers, at the end of the wake, sends what has come due. } l.endWake(0) } @@ -1223,12 +1247,16 @@ func (l *branchWorkerEventLoop) handleQueueItem(item WorkItem) { // - Drain any heal resync parked while a window was open, now that this wake may have finalized // it (a silence timeout, a CommitRequest finalize). A no-op while a window is still open or // nothing is parked. +// - Close the open window early if the branch moved under it (closeWindowForForeignCommit). // - Drive the pending writes: commit what the wake decided, schedule its push (advance). +// - Send the apply requests owed, if the wake left the branch quiet (serviceAppliers). // - Publish the result, moving the handled item's charge to what the loop holds. func (l *branchWorkerEventLoop) endWake(released int64) { l.serviceCommitRequests() l.applyDeferredHeals() + l.closeWindowForForeignCommit() l.advance() + l.serviceAppliers() l.publishLoopState(released) } @@ -1769,6 +1797,7 @@ func (l *branchWorkerEventLoop) stopTimers() { l.stopPushTimer() l.stopAttachTimer() l.stopRefusalTimer() + l.stopApplierTimer() if l.retry.timer != nil { l.retry.timer.Stop() l.retry.timer = nil @@ -2872,6 +2901,8 @@ func (r windowFinalizeReason) metricLabel() string { return "atomic_before_apply" case windowFinalizeReasonShutdown: return "shutdown" + case windowFinalizeReasonForeignCommit: + return "foreign_commit" case windowFinalizeReasonUnspecified: return "unspecified" default: @@ -3052,6 +3083,7 @@ func (w *BranchWorker) publishObservation(observed RemoteObservation) { // Stamped with the repository it is about, so that whoever publishes it can tell whether it // still describes the repository the GitTarget points at. See RemoteObservation.Repo. w.lastObservation.Store(&observed) + w.debt.observe(observed) // Delivered here, at the one point where an observation is made, rather than by each caller // against whichever GitTargets it happened to be holding. The fact is about the BRANCH, so it // goes to the branch, and every target on it reads the same answer with the time it was diff --git a/internal/git/commit_executor.go b/internal/git/commit_executor.go index 84d12b74..70a2295f 100644 --- a/internal/git/commit_executor.go +++ b/internal/git/commit_executor.go @@ -345,9 +345,6 @@ func (w *BranchWorker) executePendingWrite( pendingWrite PendingWrite, ) (int, plumbing.Hash, messageResolution, error) { switch pendingWrite.Kind { - case PendingWriteRefusalTouch: - created, hash, err := w.executeRefusalTouch(ctx, worktree, pendingWrite) - return created, hash, messageResolutionPreRendered, err case PendingWriteRequestRecord: return w.executeRequestRecord(ctx, worktree, pendingWrite) case PendingWriteResync: diff --git a/internal/git/dirty_worktree_recovery_test.go b/internal/git/dirty_worktree_recovery_test.go index 727b439b..fda8bad5 100644 --- a/internal/git/dirty_worktree_recovery_test.go +++ b/internal/git/dirty_worktree_recovery_test.go @@ -25,8 +25,6 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "k8s.io/apimachinery/pkg/runtime/schema" - - itypes "github.com/ConfigButler/gitops-reverser/internal/types" ) // leftoverPath is the file a half-finished write staged before it died. @@ -195,16 +193,6 @@ func TestEveryLoopCommitPathRecoversADirtyWorktree(t *testing.T) { require.Len(t, loop.pendingWrites, 2, "the record was made") }, }, - { - name: "empty commit for a refused write", - commit: func(t *testing.T, loop *branchWorkerEventLoop) { - t.Helper() - target := itypes.NewResourceReference("team-a", "default") - loop.commitRefusalTouch(refusalKeyFor(target, "configmaps"), "refused", "observation") - loop.endWake(0) - require.Len(t, loop.pendingWrites, 2, "the empty commit was made") - }, - }, } for _, tc := range cases { diff --git a/internal/git/pending_writes_loop.go b/internal/git/pending_writes_loop.go index 1481d49a..3e66dbb2 100644 --- a/internal/git/pending_writes_loop.go +++ b/internal/git/pending_writes_loop.go @@ -15,7 +15,7 @@ import ( // commits and publications. See docs/design/gittarget-branch-worker-pending-writes.md. // // Every write the loop makes, whatever produced it (a closed window, an atomic batch, a resync, a -// save's empty record, a refusal's empty commit), takes one path: +// save's empty record), takes one path: // // - decide appends it to the pending writes. The write gates and building the write have passed, so the // decision stands from here on. Deciding does nothing else: no commit, no push, no settlement. @@ -29,9 +29,8 @@ import ( // because the remote could not be reached. A write left pending needs no settling, only its // waiting resync callers an answer (replyToDeferredResyncs). // -// Settling an outcome can decide more work (a refused write's empty commit). Because deciding only -// appends, that work joins the pending writes behind the ones decided before it, and the pass already running -// commits it in order: nothing is ever re-entered. +// Settling an outcome never decides more work directly: a refused write owes its applier a reapply +// request (reapplyForRefusal), which is sent once the branch is quiet, outside the pending writes. // errWriteFailed marks a failure of the write itself, as opposed to reaching the remote or reading // the GitProvider: the plan was refused or could not be applied. It is terminal for the write, while @@ -45,24 +44,17 @@ func (e writeFailedError) Error() string { return e.err.Error() } func (e writeFailedError) Unwrap() error { return e.err } func (e writeFailedError) Is(target error) bool { return target == errWriteFailed } -// writeOrigin is what a write's outcome is settled against: the caller waiting on a resync, the -// atomic request, or the refusal an empty commit answers. They are process objects, kept beside +// writeOrigin is what a write's outcome is settled against: the caller waiting on a resync, or the +// atomic request. They are process objects, kept beside // the write's data rather than in it. type writeOrigin struct { - resync *ResyncRequest - atomic *WriteRequest - refusal *refusalTouchOrigin + resync *ResyncRequest + atomic *WriteRequest // answered records that the resync's caller was answered already: the remote held it back, // and it is applied later with nobody waiting. answered bool } -// refusalTouchOrigin is the refusal an empty commit answers, recorded once the commit exists. -type refusalTouchOrigin struct { - key refusalKey - observation string -} - // errAdmissionClosed answers a resync refused at admission. It is ErrFinalizeQueueFull to every // caller: the work was not accepted, and the producer that sent it keeps it. var errAdmissionClosed = fmt.Errorf( @@ -71,7 +63,7 @@ var errAdmissionClosed = fmt.Errorf( // pendingWriteOverheadBytes is what every retained item is charged beyond its payload: each event a // window or a write keeps (its identity, author, path), and a write that keeps none (its target // metadata, commit configuration, signer, the save it carries). Without it an item with no payload, -// a delete, a save's empty record or a refusal's empty commit, cost nothing to keep, and a branch +// a delete or a save's empty record, cost nothing to keep, and a branch // whose remote was down admitted them for as long as the outage lasted. const pendingWriteOverheadBytes = 1024 @@ -110,9 +102,8 @@ func (l *branchWorkerEventLoop) decide(pendingWrite PendingWrite) { // closes the retry and parent recovery, so no exit that settles the last one can leave intake // paused. // -// A push that lands settles the writes a replay refused, and settling one can decide more (a -// refusal's empty commit). Those are committed on the next turn; the push after them waits for the -// cooldown, so the loop ends. +// A push that lands settles the writes a replay refused. The loop ends once every decided write is +// committed and its push is scheduled. func (l *branchWorkerEventLoop) advance() { for { l.commitDecided() @@ -311,17 +302,12 @@ func (l *branchWorkerEventLoop) settleReplayRefusals(refused []PendingWrite) { } // settleCommitted acts on a write whose commit was made. A resync answers its caller with what it -// found, and one that committed nothing stops being pending: it is neither retained nor pushed. A -// refusal's empty commit records the refusal it covered, so the same observation is not a new -// trigger. +// found, and one that committed nothing stops being pending: it is neither retained nor pushed. func (l *branchWorkerEventLoop) settleCommitted(i int) { l.pendingWrites[i].committedOnce = true pendingWrite := l.pendingWrites[i] - switch { - case pendingWrite.origin.resync != nil: + if pendingWrite.origin.resync != nil { l.settleResyncApplied(i) - case pendingWrite.origin.refusal != nil: - l.w.recordRefusalObservation(pendingWrite.origin.refusal.key, pendingWrite.origin.refusal.observation) } } @@ -335,8 +321,8 @@ func (l *branchWorkerEventLoop) settleResyncApplied(i int) { l.removeAt(i) } // The plan was accepted, so the refusal standing over THIS SCOPE is over: the observation the - // last empty commit covered is forgotten and any commit still queued for it is cancelled. This - // is the recovery path the dedupe depends on: a per-type reconcile runs after the reconciler + // last reapply request covered is forgotten and any request still queued for it is cancelled. + // This is the recovery path the dedupe depends on: a per-type reconcile runs after the applier // reverts the edit, and it is what makes the NEXT refusal, including a re-made byte-identical // one, a new trigger rather than a repeat. Scoped to the collection this request evaluated, // because a ConfigMap resync succeeding is no evidence about a Deployment that is still refused. @@ -386,7 +372,7 @@ func (l *branchWorkerEventLoop) settleDropped(pendingWrite PendingWrite, err err // normally what discovers a write-boundary refusal first. var refused *manifestanalyzer.AcceptanceRefusedError if errors.As(err, &refused) { - l.touchBranchForRefusal(req.GitTargetName, req.GitTargetNamespace, err.Error(), refused, + l.reapplyForRefusal(req.GitTargetName, req.GitTargetNamespace, err.Error(), refused, refusalObservationForDesired(req.Desired, refused), req.refusalCollection()) } l.w.Log.Error(err, "Resync commit failed; dropping request", "resources", len(req.Desired)) @@ -397,14 +383,11 @@ func (l *branchWorkerEventLoop) settleDropped(pendingWrite PendingWrite, err err collection := req.refusalCollection() if isRefusal, refused := l.w.reportPathRefusal(err, req.GitTargetName, req.GitTargetNamespace, collection); isRefusal { - l.touchBranchForRefusal(req.GitTargetName, req.GitTargetNamespace, err.Error(), refused, + l.reapplyForRefusal(req.GitTargetName, req.GitTargetNamespace, err.Error(), refused, refusalObservationForDesired(req.Desired, refused), collection) } l.w.Log.Error(err, "Resync commit failed after its caller was answered; dropping it", "resources", len(req.Desired)) - case pendingWrite.origin.refusal != nil: - l.w.Log.Error(err, "The empty commit for a refused write failed", - "gitTarget", pendingWrite.origin.refusal.key.target.String()) case pendingWrite.Kind == PendingWriteRequestRecord: l.w.Log.Error(err, "The empty commit recording a CommitRequest failed", "gitTarget", pendingWrite.GitTargetNamespace+"/"+pendingWrite.GitTargetName) @@ -419,7 +402,7 @@ func (l *branchWorkerEventLoop) settleFailedChange(pendingWrite PendingWrite, er kind, name, namespace, collection := changeIdentity(pendingWrite) if isRefusal, refused := l.w.reportPathRefusal(err, name, namespace, collection); isRefusal { l.w.recordCommitFailure(kind, commitFailureRefused) - l.touchBranchForRefusal(name, namespace, err.Error(), refused, + l.reapplyForRefusal(name, namespace, err.Error(), refused, refusalObservationForEvents(pendingWrite.Events, refused), collection) } else { l.w.recordCommitFailure(kind, commitFailureError) diff --git a/internal/git/plan_flush.go b/internal/git/plan_flush.go index 2b785268..95c3a530 100644 --- a/internal/git/plan_flush.go +++ b/internal/git/plan_flush.go @@ -351,14 +351,13 @@ func (b *fileBuffer) deleted() bool { return b.current == nil && b.original != n // **File survival is not the question, and a review found the hole where it was.** The first // version asked only whether the file existed and survived, so deleting one document out of a // two-document file answered yes — another document kept the file alive — and a refused deletion -// qualified for an empty commit. An empty commit asks the reconciler to re-apply what Git holds, -// and Git still holds the document the refusal failed to remove, so that commit would RECREATE an -// object somebody deliberately deleted from the cluster. +// qualified for a reapply. A reapply asks the applier to apply what Git holds, and Git still holds +// the document the refusal failed to remove, so it would RECREATE an object somebody deliberately +// deleted from the cluster. // // The batch-wide arm is deliberate rather than per-file: the flush is all-or-nothing, so a refusal -// aborts the removal along with everything else, and the commit it would earn wakes the reconciler -// over the whole folder rather than over one file. A batch that meant to remove anything therefore -// earns no commit at all. +// aborts the removal along with everything else, and the reapply it would earn covers the whole +// folder rather than one file. A batch that meant to remove anything therefore earns no reapply. func (wb *writeBatch) holdsExistingDocument(b *fileBuffer) bool { if b.original == nil || b.deleted() || b.removedDocument { return false diff --git a/internal/git/push_notification.go b/internal/git/push_notification.go new file mode 100644 index 00000000..ad32d2c7 --- /dev/null +++ b/internal/git/push_notification.go @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +// EnqueuePushNotification queues a Git host's report that the branch was pushed to. +// +// The report is a hint and never a fact: after is logged and nothing else. It is queued as a refresh +// that reaches the remote whatever was proved recently, so the worker reads the advertisement itself +// and the observation it records decides whether the branch moved (applyDebt.observe). That one +// rule settles every awkward delivery without bookkeeping: a late delivery for an earlier push of +// this worker's finds the branch at the worker's latest push, a rollback onto an older one of its +// commits finds a tip it did not push last, and a duplicate costs one more advertisement. +// +// On a busy branch the refresh reads nothing (handleRefreshRequest): the worker closes its open +// window instead and publishes, and the compare-and-swap push discovers the move. +func (w *BranchWorker) EnqueuePushNotification(after string) { + w.Log.V(1).Info("Git host reported a push; checking where the branch is", "branch", w.Branch, "after", after) + w.EnqueueRefresh(&RefreshRequest{PushNotice: true}) +} diff --git a/internal/git/refresh.go b/internal/git/refresh.go index e6b1d941..4a1078d1 100644 --- a/internal/git/refresh.go +++ b/internal/git/refresh.go @@ -36,6 +36,10 @@ type RefreshRequest struct { // refresh a no-op with no connection at all, which is what keeps an actively-publishing // branch free: its pushes renew the record continuously. MaxAge time.Duration + // PushNotice marks a refresh a Git host asked for by reporting a push. It always reaches the + // remote on an idle branch, and on a busy one it closes the open window so the publication + // finds the move. See EnqueuePushNotification. + PushNotice bool } // EnqueueRefresh queues a refresh, dropping it if the queue is full. @@ -108,6 +112,11 @@ func (l *branchWorkerEventLoop) handleRefreshRequest(req *RefreshRequest) { // that does not re-read the folder: a layout resolved from a half-written tree is worse than // a slightly old one. A target mid-cycle is writing, and a write publishes its own layout. if len(l.pendingWrites) > 0 || l.openWindow != nil || w.worktreeDirty() { + if req.PushNotice { + // The publication this branch is about to make reads the remote anyway; closing the + // window makes it happen now, and its rejection, if the branch moved, starts the replay. + l.pushHinted = true + } w.Log.V(1).Info("Skipping refresh: this branch is mid-cycle", "branch", w.Branch, "gitTarget", req.Target.String()) return diff --git a/internal/git/refusal_observation_test.go b/internal/git/refusal_observation_test.go index cd800cd1..d35fa843 100644 --- a/internal/git/refusal_observation_test.go +++ b/internal/git/refusal_observation_test.go @@ -4,17 +4,15 @@ package git // Deduplicating one standing refusal, from the review that found the loop. // -// The rate limit alone only decides how OFTEN a refusal may commit, never whether the refusal is -// the same one. A forced recheck re-lists from the API server every minute, so a refused live edit -// that nobody corrects is observed again and again, and each observation used to be another -// trigger: an empty commit per interval, for ever, waking every reconciler on the branch each -// time. These pin that a repeat is recognised as a repeat, and — just as important — that a real +// The rate limit alone only decides how OFTEN a refusal may be requested, never whether the refusal +// is the same one. A forced recheck re-lists from the API server every minute, so a refused live +// edit that nobody corrects is observed again and again, and each observation used to be another +// trigger: a request per interval, for ever, making the applier reapply each time. These pin that a repeat is recognised as a repeat, and — just as important — that a real // second edit is not mistaken for one. import ( "os" "path/filepath" - "strconv" "strings" "testing" "time" @@ -35,7 +33,7 @@ import ( ) // standingRefusalFixture is a real HTTP Git remote holding the diamond, and a target that asked -// for the empty commit. The diamond refuses an image bump at the write boundary (two render roots +// for a reapply. The diamond refuses an image bump at the write boundary (two render roots // reach the shared base), which is the refusal the action exists for. func standingRefusalFixture(t *testing.T, slug string) *ledgerFixture { t.Helper() @@ -46,8 +44,10 @@ func standingRefusalFixture(t *testing.T, slug string) *ledgerFixture { target := &configv1alpha3.GitTarget{} require.NoError(t, f.worker.Client.Get(f.worker.ctx, client.ObjectKey{Name: ledgerTargetName, Namespace: "default"}, target)) - target.Spec.OnRefusal = configv1alpha3.RefusalActionPushEmptyCommit + target.Spec.OnRefusal = configv1alpha3.RefusalActionReapply + target.Spec.Applier = testApplier(ledgerTargetName) require.NoError(t, f.worker.Client.Update(f.worker.ctx, target)) + f.worker.applyRequester = &recordingRequester{} f.worker.mapper = deploymentMapper() return f @@ -60,12 +60,12 @@ func refuseResync(t *testing.T, f *ledgerFixture, l *branchWorkerEventLoop, imag t.Helper() // Rewind the rate-limit stamp rather than sleeping a minute: the question here is what happens // once the window is open, and the window itself has its own tests. - f.worker.refusalTouchMu.Lock() - if f.worker.lastRefusalTouch == nil { - f.worker.lastRefusalTouch = map[string]time.Time{} + f.worker.refusalMu.Lock() + if f.worker.lastReapply == nil { + f.worker.lastReapply = map[string]time.Time{} } - f.worker.lastRefusalTouch["default/"+ledgerTargetName] = time.Now().Add(-2 * refusalTouchInterval) - f.worker.refusalTouchMu.Unlock() + f.worker.lastReapply["default/"+ledgerTargetName] = time.Now().Add(-2 * refusalReapplyInterval) + f.worker.refusalMu.Unlock() l.lastPushAt = time.Now() event := overridesDeploymentEvent(image, 3) @@ -100,58 +100,58 @@ func deploymentResyncScope() ResyncScope { schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"}, "default") } -func remoteCommits(t *testing.T, f *ledgerFixture) int { +// refusalRequests is how many Refusal requests the fixture's worker has sent its applier so far. +func refusalRequests(t *testing.T, f *ledgerFixture) int { t.Helper() - count, err := strconv.Atoi(gitOut(t, f.repoDir, "rev-list", "--count", "main")) - require.NoError(t, err) - return count + requester, ok := f.worker.applyRequester.(*recordingRequester) + require.True(t, ok, "the fixture records apply requests") + return requester.count(configv1alpha3.ApplyReasonRefusal) } -// TestRefusalTouch_AnUnchangedRefusalIsNotASecondTrigger is the regression a review reproduced: -// three rechecks of one refused edit, each with the window open, produced three commits. -func TestRefusalTouch_AnUnchangedRefusalIsNotASecondTrigger(t *testing.T) { +// TestRefusalReapply_AnUnchangedRefusalIsNotASecondTrigger is the regression a review reproduced: +// three rechecks of one refused edit, each with the window open, produced three triggers. +func TestRefusalReapply_AnUnchangedRefusalIsNotASecondTrigger(t *testing.T) { f := standingRefusalFixture(t, "standing-refusal-unchanged") l := newBranchWorkerEventLoop(f.worker, time.Hour) t.Cleanup(l.stopTimers) - before := remoteCommits(t, f) + before := refusalRequests(t, f) for range 3 { refuseResync(t, f, l, "ghcr.io/example/podinfo:9.9.9") } - assert.Equal(t, 1, remoteCommits(t, f)-before, - "one unchanged refusal earns one commit, however often it is re-observed") + assert.Equal(t, 1, refusalRequests(t, f)-before, + "one unchanged refusal earns one request, however often it is re-observed") } -// TestRefusalTouch_ASecondRefusedEditEarnsItsOwnCommit is the positive control, and the reason the -// fence keys on the observation rather than simply latching after the first commit. +// TestRefusalReapply_ASecondRefusedEditEarnsItsOwnRequest is the positive control, and the reason the +// fence keys on the observation rather than simply latching after the first request. // -// A second live edit is a second thing to revert. The reconcile the first commit triggered ran -// against the first edit and has nothing to say about this one, so it needs a commit of its own. -func TestRefusalTouch_ASecondRefusedEditEarnsItsOwnCommit(t *testing.T) { +// A second live edit is a second thing to revert. The apply the first request caused ran against +// the first edit and has nothing to say about this one, so it needs a request of its own. +func TestRefusalReapply_ASecondRefusedEditEarnsItsOwnRequest(t *testing.T) { f := standingRefusalFixture(t, "standing-refusal-changed") l := newBranchWorkerEventLoop(f.worker, time.Hour) t.Cleanup(l.stopTimers) - before := remoteCommits(t, f) + before := refusalRequests(t, f) refuseResync(t, f, l, "ghcr.io/example/podinfo:9.9.9") refuseResync(t, f, l, "ghcr.io/example/podinfo:9.9.9") refuseResync(t, f, l, "ghcr.io/example/podinfo:10.0.0") - assert.Equal(t, 2, remoteCommits(t, f)-before, - "a genuinely different refused edit must still move the branch") + assert.Equal(t, 2, refusalRequests(t, f)-before, + "a genuinely different refused edit must still be requested") } -// TestRefusalTouch_AResyncsRefusalIsPushedByTheLoopItself: a refusal a resync discovers decides its -// empty commit while the resync's outcome is being settled, inside the commit pass. Nothing may push -// from inside a pass, so the pass's starter has to schedule it once the pass ends. Found by the -// refusal e2e on #413: the commit was made and never left the checkout. The fixture's refuseResync -// pushes by hand, which is how the unit tests missed it. -func TestRefusalTouch_AResyncsRefusalIsPushedByTheLoopItself(t *testing.T) { +// TestRefusalReapply_AResyncsRefusalIsRequestedByTheLoopItself: a refusal a resync discovers is owed +// while the resync's outcome is being settled, inside the commit pass, and the wake that ends the +// pass has to send it. The fixture's refuseResync pushes by hand, which is how a pushed-by-hand unit +// test once missed a trigger that never left the worker. +func TestRefusalReapply_AResyncsRefusalIsRequestedByTheLoopItself(t *testing.T) { f := standingRefusalFixture(t, "standing-refusal-pushed") l := newBranchWorkerEventLoop(f.worker, time.Hour) t.Cleanup(l.stopTimers) - before := remoteCommits(t, f) + before := refusalRequests(t, f) event := overridesDeploymentEvent("ghcr.io/example/podinfo:9.9.9", 3) scope := deploymentResyncScope() @@ -166,8 +166,8 @@ func TestRefusalTouch_AResyncsRefusalIsPushedByTheLoopItself(t *testing.T) { var refused *manifestanalyzer.AcceptanceRefusedError require.ErrorAs(t, (<-result).Err, &refused) - assert.Empty(t, l.pendingWrites, "the empty commit was published") - assert.Equal(t, 1, remoteCommits(t, f)-before, "with no push started by hand") + assert.Empty(t, l.pendingWrites, "the refused resync left nothing pending") + assert.Equal(t, 1, refusalRequests(t, f)-before, "and the request went out with no push started by hand") } // TestRefusalObservation_IsContentAddressedAndOrderIndependent pins the digest itself: same @@ -199,7 +199,7 @@ func TestRefusalObservation_IsContentAddressedAndOrderIndependent(t *testing.T) "a smaller batch is a new observation") // A different refusal about the same objects is also new: the folder changed under the edit, - // and what the previous commit asked for was answered against the old one. + // and what the previous request asked for was answered against the old one. elsewhere := &manifestanalyzer.AcceptanceRefusedError{ Issues: []manifestanalyzer.AcceptanceIssue{ {Kind: manifestanalyzer.IssueWriteFanIn, Path: "overlays/prod/deployment.yaml"}, @@ -211,11 +211,11 @@ func TestRefusalObservation_IsContentAddressedAndOrderIndependent(t *testing.T) "nothing to digest is not an observation, and must never count as covered") } -// TestRefusalObservation_NothingIsCoveredUntilACommitIsMade. The memory is written where the -// commit is, so a build or commit failure leaves the next observation free to try again. -func TestRefusalObservation_NothingIsCoveredUntilACommitIsMade(t *testing.T) { +// TestRefusalObservation_NothingIsCoveredUntilARequestIsOwed. The memory is written where the +// request is owed, and nowhere else. +func TestRefusalObservation_NothingIsCoveredUntilARequestIsOwed(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) assert.False(t, w.refusalAlreadyCovered(editingKey(), "observation-1")) @@ -238,14 +238,14 @@ func TestRefusalObservation_NothingIsCoveredUntilACommitIsMade(t *testing.T) { // regression a review found in it. // // Without any recovery the fence would be a permanent mute: a user re-making the same live edit -// after the reconciler reverted it produces byte-identical content, so only the acceptance in +// after the applier reverted it produces byte-identical content, so only the acceptance in // between can tell the second edit from the first. But recovery that clears the whole TARGET is // the loop coming back by another door: a GitTarget watches several types, a no-op ConfigMap // resync succeeds every time it runs, and the still-refused Deployment would be re-armed by every // one of them. A successful evaluation may only speak for what it evaluated. func TestRefusalObservation_RecoveryIsScopedToTheCollectionThatRecovered(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) loop := newBranchWorkerEventLoop(w, time.Minute) t.Cleanup(loop.stopTimers) @@ -274,46 +274,46 @@ func TestRefusalObservation_RecoveryIsScopedToTheCollectionThatRecovered(t *test assert.False(t, w.refusalAlreadyCovered(configMaps, "observation-2")) } -// TestRefusalObservation_RecoveryCancelsTheQueuedCommitForThatCollection is the other half of the same -// regression. +// TestRefusalObservation_RecoveryCancelsTheQueuedRequestForThatCollection is the other half of the +// same regression. // -// Dropping the digest is not enough: a commit already queued for that collection finds nothing covering -// it when its timer fires and moves the branch anyway, for a refusal that has since been accepted. -// The obligation has to go with the memory — and only that collection's obligation. -func TestRefusalObservation_RecoveryCancelsTheQueuedCommitForThatCollection(t *testing.T) { +// Dropping the digest is not enough: a request already queued for that collection finds nothing +// covering it when its timer fires and asks for a reapply anyway, for a refusal that has since been +// accepted. The obligation has to go with the memory — and only that collection's obligation. +func TestRefusalObservation_RecoveryCancelsTheQueuedRequestForThatCollection(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) loop := newBranchWorkerEventLoop(w, time.Minute) t.Cleanup(loop.stopTimers) deployments := editingKey() configMaps := refusalKeyFor(editingRef(), "configmaps") - loop.armTrailingRefusalTouch(deployments, "the deployment was refused", "observation-1", 0) - loop.armTrailingRefusalTouch(configMaps, "the configmap was refused", "observation-2", 0) + loop.armTrailingReapply(deployments, "the deployment was refused", "observation-1", 0) + loop.armTrailingReapply(configMaps, "the configmap was refused", "observation-2", 0) require.Len(t, loop.refusalPending, 2) loop.refusalRecovered(editingRef(), deployments.collection) assert.NotContains(t, loop.refusalPending, deployments, - "a queued commit must not survive the acceptance of the object it was queued for") + "a queued request must not survive the acceptance of the object it was queued for") assert.Contains(t, loop.refusalPending, configMaps, "and the collection that is still refused keeps its obligation") - // Nothing was committed for the cancelled entry, so it must not have spent the window either. + // Nothing was requested for the cancelled entry, so it must not have spent the window either. limited, _ := w.refusalRateLimited(editingRef()) assert.False(t, limited) } -// TestRefusalTouch_RemovingOneDocumentFromASharedFileIsNotCommittedFor is the second regression +// TestRefusalReapply_RemovingOneDocumentFromASharedFileIsNotRequestedFor is the second regression // the same review found, and it runs through the real planner rather than a hand-built issue. // // A DELETE of one document out of a two-document file leaves the file present, so the old // file-level evidence answered "this is an edit to a surviving document" and the refusal qualified -// for an empty commit. It is the opposite of an edit: the refusal means Git still holds the -// document, and a commit asking the reconciler to re-apply would put back an object somebody -// deliberately deleted from the cluster. -func TestRefusalTouch_RemovingOneDocumentFromASharedFileIsNotCommittedFor(t *testing.T) { +// for a reapply. It is the opposite of an edit: the refusal means Git still holds the document, and +// asking the applier to re-apply would put back an object somebody deliberately deleted from the +// cluster. +func TestRefusalReapply_RemovingOneDocumentFromASharedFileIsNotRequestedFor(t *testing.T) { writer := newContentWriter(itypes.SensitiveResourcePolicy{}) worktree := newWorktreeForTest(t) root := worktree.Filesystem().Root() @@ -335,11 +335,11 @@ func TestRefusalTouch_RemovingOneDocumentFromASharedFileIsNotCommittedFor(t *tes "a removal must not qualify because a sibling document keeps the file alive: %+v", refused.Issues) } -// TestRefusalTouch_ARemovalAnywhereInTheFlushDisqualifiesIt covers the batch-wide arm of the same +// TestRefusalReapply_ARemovalAnywhereInTheFlushDisqualifiesIt covers the batch-wide arm of the same // fence, which the sweep path needs: a mark-and-sweep resync can drop one file's document while -// refusing an edit to another, and the flush is all-or-nothing, so the commit it would earn wakes -// the reconciler over a folder whose deletion did not happen. -func TestRefusalTouch_ARemovalAnywhereInTheFlushDisqualifiesIt(t *testing.T) { +// refusing an edit to another, and the flush is all-or-nothing, so the reapply it would earn covers +// a folder whose deletion did not happen. +func TestRefusalReapply_ARemovalAnywhereInTheFlushDisqualifiesIt(t *testing.T) { wb := &writeBatch{ buffers: map[string]*fileBuffer{}, contentByPath: map[string][]byte{ @@ -354,55 +354,55 @@ func TestRefusalTouch_ARemovalAnywhereInTheFlushDisqualifiesIt(t *testing.T) { wb.recordDocumentRemoval(wb.buffer("swept.yaml")) assert.False(t, wb.holdsExistingDocument(edited), - "a flush that removes a document anywhere earns no commit, whichever file was refused") + "a flush that removes a document anywhere earns no reapply, whichever file was refused") } -// TestRefusalTouch_AQueuedCommitIsDroppedOnceSomethingElseCoveredIt closes the gap between arming -// a trailing commit and running it. +// TestRefusalReapply_AQueuedRequestIsDroppedOnceSomethingElseCoveredIt closes the gap between arming +// a trailing request and owing it. // -// A minute passes between the two, and another commit for the same target can be made in it. If -// that commit covered the same observation, this entry is asking for a second one that says -// exactly the same thing to the same reconcilers, so it is dropped rather than run. Consent is +// A minute passes between the two, and another request for the same target can be owed in it. If +// that request covered the same observation, this entry is asking for a second one that says +// exactly the same thing to the same applier, so it is dropped rather than owed. Consent is // re-checked at the same point and for the same reason: a delayed action must act on what is true // when it fires. -func TestRefusalTouch_AQueuedCommitIsDroppedOnceSomethingElseCoveredIt(t *testing.T) { +func TestRefusalReapply_AQueuedRequestIsDroppedOnceSomethingElseCoveredIt(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) loop := newBranchWorkerEventLoop(w, time.Minute) t.Cleanup(loop.stopTimers) - loop.armTrailingRefusalTouch(editingKey(), "refused while the window was closed", "observation-1", 0) + loop.armTrailingReapply(editingKey(), "refused while the window was closed", "observation-1", 0) require.Contains(t, loop.refusalPending, editingKey()) // Whatever else ran in the meantime covered exactly this. w.recordRefusalObservation(editingKey(), "observation-1") loop.stopRefusalTimer() - loop.flushPendingRefusalTouch() + loop.flushPendingReapply() loop.endWake(0) assert.Empty(t, loop.refusalPending, "the entry is consumed either way") limited, _ := w.refusalRateLimited(editingRef()) assert.False(t, limited, - "a covered entry must not spend the rate-limit window on a commit nobody needed") + "a covered entry must not spend the rate-limit window on a request nobody needed") } -// TestRefusalTouch_AnAcceptedSiblingTypeDoesNotRearmAStandingRefusal is the interaction a review +// TestRefusalReapply_AnAcceptedSiblingTypeDoesNotRearmAStandingRefusal is the interaction a review // found, driven through the real resync handler against a real Git remote. // // A GitTarget watches more than one type. The Deployment stays refused; a ConfigMap reconcile for // the same target succeeds, as an empty or unchanged one does every time it runs. If that success // is read as recovery for the whole target, the next identical Deployment observation becomes a -// new trigger, and one commit per ConfigMap pass is the loop again wearing a different hat. -func TestRefusalTouch_AnAcceptedSiblingTypeDoesNotRearmAStandingRefusal(t *testing.T) { +// new trigger, and one request per ConfigMap pass is the loop again wearing a different hat. +func TestRefusalReapply_AnAcceptedSiblingTypeDoesNotRearmAStandingRefusal(t *testing.T) { f := standingRefusalFixture(t, "standing-refusal-sibling-type") l := newBranchWorkerEventLoop(f.worker, time.Hour) t.Cleanup(l.stopTimers) - before := remoteCommits(t, f) + before := refusalRequests(t, f) refuseResync(t, f, l, "ghcr.io/example/podinfo:9.9.9") - require.Equal(t, before+1, remoteCommits(t, f), "the first refusal earns its commit") + require.Equal(t, before+1, refusalRequests(t, f), "the first refusal earns its request") // A different watched collection, with nothing to write and nothing to refuse. configMaps := ResyncScopeFor(schema.GroupVersionResource{Version: "v1", Resource: "configmaps"}, "default") @@ -418,20 +418,20 @@ func TestRefusalTouch_AnAcceptedSiblingTypeDoesNotRearmAStandingRefusal(t *testi refuseResync(t, f, l, "ghcr.io/example/podinfo:9.9.9") - assert.Equal(t, before+1, remoteCommits(t, f), + assert.Equal(t, before+1, refusalRequests(t, f), "a ConfigMap reconcile succeeding says nothing about a Deployment that is still refused") } -// TestRefusalTouch_AnEarlyRefusalStillSeesThePlannedRemoval is the third interaction. +// TestRefusalReapply_AnEarlyRefusalStillSeesThePlannedRemoval is the third interaction. // // Both flush paths apply their upserts first and abort on the first refusal, so a sweep that would // have removed a document is never reached — and the refusal, raised before it, used to carry -// evidence saying this flush removes nothing. The commit that evidence earns asks the reconciler -// to re-apply what Git holds, which recreates the object the sweep was about to delete. +// evidence saying this flush removes nothing. The reapply that evidence earns asks the applier to +// apply what Git holds, which recreates the object the sweep was about to delete. // // The shape: one Deployment whose args a kustomize patch and the live object both changed, which // cannot be placed, and a second document absent from the snapshot, which PruneAlways sweeps. -func TestRefusalTouch_AnEarlyRefusalStillSeesThePlannedRemoval(t *testing.T) { +func TestRefusalReapply_AnEarlyRefusalStillSeesThePlannedRemoval(t *testing.T) { worktree := newWorktreeForTest(t) root := worktree.Filesystem().Root() base := `apiVersion: apps/v1 @@ -520,22 +520,24 @@ func overlayRefusalFixture(t *testing.T, slug string) *ledgerFixture { target := &configv1alpha3.GitTarget{} require.NoError(t, f.worker.Client.Get(f.worker.ctx, client.ObjectKey{Name: ledgerTargetName, Namespace: "default"}, target)) - target.Spec.OnRefusal = configv1alpha3.RefusalActionPushEmptyCommit + target.Spec.OnRefusal = configv1alpha3.RefusalActionReapply + target.Spec.Applier = testApplier(ledgerTargetName) require.NoError(t, f.worker.Client.Update(f.worker.ctx, target)) + f.worker.applyRequester = &recordingRequester{} f.worker.mapper = deploymentMapper() return f } -// TestRefusalTouch_RecoveryCancelsACommitQueuedForTheSameCollection is the handler-level regression for +// TestRefusalReapply_RecoveryCancelsARequestQueuedForTheSameCollection is the handler-level regression for // recovery: the same transition as the focused state test, but driven through the real applyResync // hook against a real Git remote, so it holds the hook's PLACEMENT and not only the state it // writes. It is the case a review reproduced. // -// The sequence is the one an operator actually produces: an edit is refused and earns a commit, a -// second edit inside the rate-limit window queues a trailing commit, and then the object is put -// back to what the folder renders and is accepted. The queued commit is now asking the reconciler -// to re-apply on account of a refusal that no longer exists, so it must not fire. -func TestRefusalTouch_RecoveryCancelsACommitQueuedForTheSameCollection(t *testing.T) { +// The sequence is the one an operator actually produces: an edit is refused and earns a request, a +// second edit inside the rate-limit window queues a trailing request, and then the object is put +// back to what the folder renders and is accepted. The queued request is now asking the applier to +// re-apply on account of a refusal that no longer exists, so it must not fire. +func TestRefusalReapply_RecoveryCancelsARequestQueuedForTheSameCollection(t *testing.T) { f := overlayRefusalFixture(t, "recovery-cancels-queued") l := newBranchWorkerEventLoop(f.worker, time.Hour) t.Cleanup(l.stopTimers) @@ -578,30 +580,30 @@ func TestRefusalTouch_RecoveryCancelsACommitQueuedForTheSameCollection(t *testin require.ErrorAs(t, observe("debug"), &refused, "a base-owned field edit has nowhere to land") require.ErrorAs(t, observe("trace"), &refused, "and so does a different value for it") require.Contains(t, l.refusalPending, key, - "the second edit arrived inside the window, so it must be queued behind a trailing commit") + "the second edit arrived inside the window, so it must be queued behind a trailing request") require.NoError(t, observe(""), "the object is put back to what the folder renders, and accepted") require.NotContains(t, l.refusalPending, key, - "accepting the object cancels the commit that was queued for its refusal") + "accepting the object cancels the request that was queued for its refusal") // Fire the timer as the loop would, with both clocks advanced past their deadlines. - before := remoteCommits(t, f) - f.worker.refusalTouchMu.Lock() - f.worker.lastRefusalTouch["default/"+ledgerTargetName] = time.Now().Add(-2 * refusalTouchInterval) - f.worker.refusalTouchMu.Unlock() + before := refusalRequests(t, f) + f.worker.refusalMu.Lock() + f.worker.lastReapply["default/"+ledgerTargetName] = time.Now().Add(-2 * refusalReapplyInterval) + f.worker.refusalMu.Unlock() l.stopRefusalTimer() - l.flushPendingRefusalTouch() + l.flushPendingReapply() l.endWake(0) require.NoError(t, f.worker.pushPendingCommits(l.pendingWrites)) - assert.Equal(t, before, remoteCommits(t, f), - "a queued refusal must not move the branch after the same object was accepted") + assert.Equal(t, before, refusalRequests(t, f), + "a queued refusal must not be requested after the same object was accepted") } // TestRefusalObservation_ASelectedLiveRefusalIsRecoveredByTheNextAcceptedSnapshot: the live path // files a refusal under the selector-bearing collection its events came from, and the accepted // snapshot of the same boundary must clear it, whether the rule still selects the same objects or -// has since changed selector. Keyed inconsistently, the queued commit fired after recovery and the +// has since changed selector. Keyed inconsistently, the queued request fired after recovery and the // memory muted the next identical live refusal. func TestRefusalObservation_ASelectedLiveRefusalIsRecoveredByTheNextAcceptedSnapshot(t *testing.T) { selecting := func(selector string) ResyncScope { @@ -615,29 +617,29 @@ func TestRefusalObservation_ASelectedLiveRefusalIsRecoveredByTheNextAcceptedSnap } { t.Run(name, func(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) loop := newBranchWorkerEventLoop(w, time.Minute) t.Cleanup(loop.stopTimers) live := sourceCollectionForEvents([]Event{{SourceCollection: selecting("tier in (x)").Collection}}) key := newRefusalKey(editingRef(), live) - // A previous live refusal was committed for, so the next one is rate limited and queued. + // A previous live refusal was requested for, so the next one is rate limited and queued. w.recordRefusalObservation(key, "previous-live-refusal") - w.lastRefusalTouch = map[string]time.Time{editingRef().String(): time.Now()} + w.lastReapply = map[string]time.Time{editingRef().String(): time.Now()} refused := &manifestanalyzer.AcceptanceRefusedError{Issues: []manifestanalyzer.AcceptanceIssue{{ Kind: manifestanalyzer.IssueWriteFanIn, ExistingDocument: true, Path: "base/cm.yaml", }}} - loop.touchBranchForRefusal(editingRef().Name, editingRef().Namespace, + loop.reapplyForRefusal(editingRef().Name, editingRef().Namespace, "live edit refused", refused, "new-live-refusal", live) - require.Len(t, loop.refusalPending, 1, "the live refusal queues its trailing commit") + require.Len(t, loop.refusalPending, 1, "the live refusal queues its trailing request") snapshot := selecting(snapshotSelector) loop.refusalRecovered(editingRef(), (&ResyncRequest{Scope: &snapshot}).refusalCollection()) - assert.Empty(t, loop.refusalPending, "the accepted snapshot cancels the queued commit") + assert.Empty(t, loop.refusalPending, "the accepted snapshot cancels the queued request") assert.False(t, w.refusalAlreadyCovered(key, "previous-live-refusal"), - "and forgets what the last commit covered, so the same live refusal is new again") + "and forgets what the last request covered, so the same live refusal is new again") }) } } diff --git a/internal/git/refusal_touch.go b/internal/git/refusal_reapply.go similarity index 63% rename from internal/git/refusal_touch.go rename to internal/git/refusal_reapply.go index daf1ff05..adc42827 100644 --- a/internal/git/refusal_touch.go +++ b/internal/git/refusal_reapply.go @@ -13,23 +13,20 @@ import ( "github.com/ConfigButler/gitops-reverser/internal/manifestanalyzer" - gogit "github.com/go-git/go-git/v6" - "github.com/go-git/go-git/v6/plumbing" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" - "sigs.k8s.io/controller-runtime/pkg/log" "github.com/ConfigButler/gitops-reverser/api/v1alpha3" itypes "github.com/ConfigButler/gitops-reverser/internal/types" ) -// refusalTouchInterval is the shortest gap between two empty commits for one GitTarget. +// refusalReapplyInterval is the shortest gap between two Refusal apply requests for one GitTarget. // // A refusal is not a one-off. The same live edit can be re-attempted, and a controller writing a -// base-owned field can produce one on every reconcile of its own, so without a floor the branch -// would fill with empty commits and every one of them would wake every reconciler watching it. -// The debounce is per target rather than per object because the commit is per branch: two refusals -// a second apart are served by one commit, and the reconcile it triggers covers both. -const refusalTouchInterval = time.Minute +// base-owned field can produce one on every reconcile of its own, so without a floor the applier +// would be asked to reapply on every one of those reconciles. The floor is per target because the +// request is: two refusals a second apart are served by one request, and the apply it causes +// covers both. +const refusalReapplyInterval = time.Minute // refusalKey identifies ONE unresolved refusal: the GitTarget that owns it, and the watched collection // it is about. @@ -37,7 +34,7 @@ const refusalTouchInterval = time.Minute // The collection is in the key because a review found what a target-wide key gets wrong in both // directions. A target watches several types; a successful ConfigMap resync says nothing about a // Deployment that is still refused, so clearing the whole target on it re-armed the standing -// refusal and the commit loop came back. And a queued commit for one collection must survive another +// refusal and the request loop came back. And a queued request for one collection must survive another // collection recovering. This is the same granularity the GitPathAccepted condition already uses — the // watch layer blocks and clears per collection — so the memory and the status now agree about what is // still refused. @@ -49,7 +46,7 @@ const refusalTouchInterval = time.Minute // types.CollectionKey), and the rule compiler allows one selector per boundary. Every producer // must agree on the key: the live path files a refusal under the selector-bearing collection its // events came from, and when the accepted snapshot that recovers it was keyed differently, its -// queued commit fired after recovery and its memory muted the next identical refusal. Kept with +// queued request fired after recovery and its memory muted the next identical refusal. Kept with // the selector, a rule that changed selector left the old key behind the same way. type refusalKey struct { target itypes.ResourceReference @@ -71,36 +68,10 @@ func (k refusalKey) String() string { return k.target.String() + " " + sourceCollectionForLog(k.collection) } -// executeRefusalTouch creates the empty commit. The tree is untouched on purpose, so -// AllowEmptyCommits is what makes it a commit at all. -// -// Nothing here decides WHETHER to do it. That is settled before the write is built, because by the -// time a write reaches the executor it has already been retained, and a write that declined to -// commit here would sit in the retained set advancing the push cooldown for nothing. -func (w *BranchWorker) executeRefusalTouch( - ctx context.Context, - worktree *gogit.Worktree, - pendingWrite PendingWrite, -) (int, plumbing.Hash, error) { - message, options, _, err := pendingWrite.commitMetadata() - if err != nil { - return 0, plumbing.ZeroHash, err - } - options.AllowEmptyCommits = true - - hash, err := worktree.Commit(message, options) - if err != nil { - return 0, plumbing.ZeroHash, fmt.Errorf("failed to create refusal commit: %w", err) - } - log.FromContext(ctx).Info("Empty commit created to trigger a reconcile after a refused write", - "gitTarget", pendingWrite.GitTargetNamespace+"/"+pendingWrite.GitTargetName, - "branch", w.Branch, "commit", hash.String()) - return 1, hash, nil -} - -// refusalConsent reports whether this GitTarget asked for an empty commit on refusal at all, -// ignoring rate limiting. It is re-read every time, including when a coalesced commit finally -// fires, so a target that was suspended or set back to Ignore in the meantime is honoured. +// refusalConsent reports whether this GitTarget asked for its applier to reapply on refusal, +// ignoring rate limiting. It is re-read every time, including when a coalesced request finally +// fires, so a target that was suspended, set back to Ignore, or stripped of its applier in the +// meantime is honoured. // // The GitTarget is read live rather than from the write's resolved metadata because a refusal has // no write to carry metadata: the plan was aborted before anything was built. A read failure @@ -108,99 +79,52 @@ func (w *BranchWorker) executeRefusalTouch( func (w *BranchWorker) refusalConsent(ctx context.Context, target itypes.ResourceReference) bool { gitTarget, err := w.getGitTarget(ctx, target.Name, target.Namespace) if err != nil { - w.Log.V(1).Info("Cannot read GitTarget to decide the refusal action; not committing", + w.Log.V(1).Info("Cannot read GitTarget to decide the refusal action; not requesting a reapply", "gitTarget", target.String(), "error", err.Error()) return false } - if gitTarget.EffectiveOnRefusal() != v1alpha3.RefusalActionPushEmptyCommit { + if gitTarget.EffectiveOnRefusal() != v1alpha3.RefusalActionReapply || gitTarget.Spec.Applier == nil { return false } if gitTarget.Spec.Suspend { - // A suspended target writes nothing, and an empty commit is a write. Driving somebody - // else's reconciler from a target the operator has deliberately parked would be the one - // write suspension does not stop. + // A suspended target has stopped publishing the live edits a reapply would be weighed + // against. Driving its applier on behalf of a target the operator has deliberately parked + // would be acting on a mirror nobody is keeping. return false } return true } -// refusalRateLimited reports whether a commit for this target is inside the rate-limit window, and -// stamps the clock when it is not. +// refusalRateLimited reports whether a request for this target is inside the rate-limit window, +// and stamps the clock when it is not. // -// The caller must NOT simply drop a rate-limited refusal. Dropping loses it: the reconcile the -// earlier commit triggered may already have finished before this refusal happened, so nothing -// covers it and the live edit stays forever. touchBranchForRefusal coalesces it into a trailing -// commit instead. +// The caller must NOT simply drop a rate-limited refusal. Dropping loses it: the apply the earlier +// request caused may already have finished before this refusal happened, so nothing covers it and +// the live edit stays forever. reapplyForRefusal coalesces it into a trailing request instead. func (w *BranchWorker) refusalRateLimited(target itypes.ResourceReference) (bool, time.Duration) { - w.refusalTouchMu.Lock() - defer w.refusalTouchMu.Unlock() - if w.lastRefusalTouch == nil { - w.lastRefusalTouch = map[string]time.Time{} + w.refusalMu.Lock() + defer w.refusalMu.Unlock() + if w.lastReapply == nil { + w.lastReapply = map[string]time.Time{} } key := target.String() - if last, seen := w.lastRefusalTouch[key]; seen { - if elapsed := time.Since(last); elapsed < refusalTouchInterval { - return true, refusalTouchInterval - elapsed + if last, seen := w.lastReapply[key]; seen { + if elapsed := time.Since(last); elapsed < refusalReapplyInterval { + return true, refusalReapplyInterval - elapsed } } - w.lastRefusalTouch[key] = time.Now() + w.lastReapply[key] = time.Now() return false, 0 } -// buildRefusalTouchWrite assembles the empty commit's write. It borrows the target's resolved -// commit configuration so the commit is phrased and signed like every other commit this target -// makes; only its tree is empty. -func (w *BranchWorker) buildRefusalTouchWrite( - ctx context.Context, - target itypes.ResourceReference, - detail string, -) (*PendingWrite, error) { - metadata, err := w.resolveTargetMetadata(ctx, target.Name, target.Namespace) - if err != nil { - return nil, err - } - provider, err := w.getGitProvider(ctx) - if err != nil { - return nil, err - } - signer, err := getCommitSigner(ctx, w.Client, provider) - if err != nil { - return nil, err - } - return &PendingWrite{ - Kind: PendingWriteRefusalTouch, - CommitMessage: refusalCommitMessage(target, detail), - CommitConfig: ResolveCommitConfig(provider.Spec.Commit). - WithTargetMessage(metadata.CommitMessage), - Signer: signer, - GitTargetName: target.Name, - GitTargetNamespace: target.Namespace, - Targets: map[pendingTargetKey]ResolvedTargetMetadata{ - {Name: target.Name, Namespace: target.Namespace}: metadata, - }, - }, nil -} - -// refusalCommitMessage is the whole user-visible product of this feature when nobody is watching -// conditions, so it says what happened and why the commit is empty rather than looking like a -// stray no-op somebody force-pushed. -func refusalCommitMessage(target itypes.ResourceReference, detail string) string { - return fmt.Sprintf( - "chore: reconcile after a refused write to %s\n\n"+ - "This commit changes no file. It exists to move the branch so the GitOps reconciler "+ - "re-applies the desired state, reverting a live edit that has no destination in Git.\n\n"+ - "Refused: %s\n", - target.String(), detail) -} - -// touchBranchForRefusal is the loop-side half: it turns a reported refusal into an empty commit -// and schedules the push. +// reapplyForRefusal is the loop-side half: it turns a reported refusal into a Refusal apply request +// owed to the target's applier, sent once the branch is quiet (serviceAppliers). // // It runs on the event loop, after the refusal has already been reported, and it is deliberately -// best-effort. A refusal is already surfaced as GitPathAccepted=False; failing to add a commit on -// top of that is a missed acceleration, not a second fault, so every failure here is logged and +// best-effort. A refusal is already surfaced as GitPathAccepted=False; failing to ask for a reapply +// on top of that is a missed recovery, not a second fault, so every failure here is logged and // swallowed rather than escalated into the caller's error path. -func (l *branchWorkerEventLoop) touchBranchForRefusal( +func (l *branchWorkerEventLoop) reapplyForRefusal( targetName, targetNamespace, detail string, refused *manifestanalyzer.AcceptanceRefusedError, observation string, @@ -209,7 +133,7 @@ func (l *branchWorkerEventLoop) touchBranchForRefusal( if targetName == "" || targetNamespace == "" { return } - // Checked BEFORE consent and the rate limit, so a refusal we would never commit for does not + // Checked BEFORE consent and the rate limit, so a refusal we would never request for does not // consume the window and starve one we would. if !refusalIsAWriteBoundary(refused) { return @@ -220,26 +144,26 @@ func (l *branchWorkerEventLoop) touchBranchForRefusal( } key := newRefusalKey(target, collection) - // An observation this collection's last commit already covered is not a new trigger. Checked here, - // after consent and before the rate limit, so a recheck of unchanged input neither commits nor - // arms the trailing timer: arming it would turn the rate limit into a schedule, which is the - // loop this fence exists to close. + // An observation this collection's last request already covered is not a new trigger. Checked + // here, after consent and before the rate limit, so a recheck of unchanged input neither + // requests nor arms the trailing timer: arming it would turn the rate limit into a schedule, + // which is the loop this fence exists to close. if l.w.refusalAlreadyCovered(key, observation) { - l.w.Log.V(1).Info("Refusal unchanged since the last empty commit; not committing again", + l.w.Log.V(1).Info("Refusal unchanged since the last reapply request; not requesting again", "gitTarget", target.String(), "sourceCollection", sourceCollectionForLog(collection), "branch", l.w.Branch) return } // Inside the rate-limit window, COALESCE rather than drop. Dropping is a correctness bug, not - // a missed optimisation: the reconcile the previous commit triggered may already have - // completed, so nothing would ever cover this refusal and the live edit would stay for good. - // One trailing commit covers every refusal that arrived during the window. + // a missed optimisation: the apply the previous request caused may already have completed, so + // nothing would ever cover this refusal and the live edit would stay for good. One trailing + // request covers every refusal that arrived during the window. if limited, wait := l.w.refusalRateLimited(target); limited { - l.armTrailingRefusalTouch(key, detail, observation, wait) + l.armTrailingReapply(key, detail, observation, wait) return } - l.commitRefusalTouch(key, detail, observation) + l.oweReapply(key, detail, observation) } // refusalObservation digests what was refused: the objects the write would have produced, and the @@ -250,8 +174,8 @@ func (l *branchWorkerEventLoop) touchBranchForRefusal( // It digests CONTENT rather than a resourceVersion on purpose. A forced recheck re-lists from the // API server and every snapshot it takes carries a fresh collection version, so versions would // make every recheck look new, which is precisely the loop being closed. Sanitized content is also -// exactly what the refused write was going to be, so it is the thing the previous commit's -// reconcile either covered or did not. +// exactly what the refused write was going to be, so it is the thing the previous request's +// apply either covered or did not. // // The digest is order-independent: the per-object digests are sorted before they are folded in, so // the same set observed in a different order is the same observation. @@ -267,7 +191,7 @@ func refusalObservation( if err != nil { // An object that cannot be encoded cannot be identified, so it is treated as a // distinct observation every time: the fence errs towards committing, which costs an - // empty commit, rather than towards silence, which costs a live edit its revert. + // reapply request, rather than towards silence, which costs a live edit its revert. return "" } parts = append(parts, fmt.Sprintf("%x", sha256.Sum256(encoded))) @@ -286,7 +210,7 @@ func refusalObservation( } // refusalObservationForEvents digests the objects a live write batch carried. A DELETE carries no -// object; it contributes nothing, which is harmless because a removal never reaches the commit +// object; it contributes nothing, which is harmless because a removal never reaches the request // anyway (see refusalIsAWriteBoundary). func refusalObservationForEvents( events []Event, refused *manifestanalyzer.AcceptanceRefusedError, @@ -310,7 +234,7 @@ func refusalObservationForDesired( return refusalObservation(objects, refused) } -// refusalAlreadyCovered reports whether an empty commit has already been made for exactly this +// refusalAlreadyCovered reports whether a reapply has already been requested for exactly this // observation, with nothing accepted for the target since. // // An empty observation (nothing to digest, or an object that would not encode) is never covered: @@ -319,18 +243,18 @@ func (w *BranchWorker) refusalAlreadyCovered(key refusalKey, observation string) if observation == "" { return false } - w.refusalTouchMu.Lock() - defer w.refusalTouchMu.Unlock() + w.refusalMu.Lock() + defer w.refusalMu.Unlock() return w.coveredRefusal[newRefusalKey(key.target, key.collection)] == observation } -// recordRefusalObservation remembers what the commit just made covers. +// recordRefusalObservation remembers what the request just owed covers. func (w *BranchWorker) recordRefusalObservation(key refusalKey, observation string) { if observation == "" { return } - w.refusalTouchMu.Lock() - defer w.refusalTouchMu.Unlock() + w.refusalMu.Lock() + defer w.refusalMu.Unlock() if w.coveredRefusal == nil { w.coveredRefusal = map[refusalKey]string{} } @@ -338,19 +262,19 @@ func (w *BranchWorker) recordRefusalObservation(key refusalKey, observation stri } // refusalRecovered records that an evaluation for this target and collection was ACCEPTED, so whatever -// refusal was outstanding there is over: the memory of what the last commit covered is dropped, -// and any commit still queued for it is cancelled. +// refusal was outstanding there is over: the memory of what the last request covered is dropped, +// and any request still queued for it is cancelled. // // **Only a resync calls this, and only for its own scope.** That is the rule the GitPathAccepted // condition already follows, for the same reason: a live write that happens to avoid the offending // file proves nothing about the rest of the subtree, and one watched type succeeding proves // nothing about another. Clearing more widely than the evidence supports is what let a no-op -// ConfigMap resync re-arm a standing Deployment refusal, and the commit loop with it. +// ConfigMap resync re-arm a standing Deployment refusal, and the request loop with it. // // Dropping the memory is what keeps the fence from becoming a permanent mute: a refused live edit -// earns a commit, the reconciler reverts it, and the same edit is made again — sanitized content +// earns a request, the applier reverts it, and the same edit is made again — sanitized content // cannot tell the second from the first, so only the acceptance in between can. Cancelling the -// queued commit is the other half: an obligation whose object has since been accepted has nothing +// queued request is the other half: an obligation whose object has since been accepted has nothing // left to ask for, and firing it anyway would move the branch for a refusal that no longer exists. func (l *branchWorkerEventLoop) refusalRecovered(target itypes.ResourceReference, collection itypes.CollectionKey) { if target.Name == "" || target.Namespace == "" { @@ -366,11 +290,11 @@ func (l *branchWorkerEventLoop) refusalRecovered(target itypes.ResourceReference l.rearmRefusalTimer() } -// clearRefusalObservations forgets what this collection's last empty commit covered. The ZERO collection is a +// clearRefusalObservations forgets what this collection's last reapply request covered. The ZERO collection is a // whole-GitTarget evaluation, which does speak for every collection, so it clears all of them. func (w *BranchWorker) clearRefusalObservations(target itypes.ResourceReference, collection itypes.CollectionKey) { - w.refusalTouchMu.Lock() - defer w.refusalTouchMu.Unlock() + w.refusalMu.Lock() + defer w.refusalMu.Unlock() if collection != (itypes.CollectionKey{}) { delete(w.coveredRefusal, newRefusalKey(target, collection)) return @@ -382,28 +306,28 @@ func (w *BranchWorker) clearRefusalObservations(target itypes.ResourceReference, } } -// pendingRefusalTouch is one target's coalesced commit: what to say, and when it may run. -type pendingRefusalTouch struct { +// pendingRefusalReapply is one target's coalesced request: what to say, and when it may run. +type pendingRefusalReapply struct { key refusalKey detail string - // observation is what this entry's commit will cover. It is carried rather than re-derived + // observation is what this entry's request will cover. It is carried rather than re-derived // because the refusal that queued it is long gone by the time the timer fires. observation string dueAt time.Time } -// armTrailingRefusalTouch records one target's coalesced commit and arms the shared timer for the +// armTrailingReapply records one target's coalesced request and arms the shared timer for the // earliest deadline among all of them. // // A second refusal for the SAME target inside its window refreshes the detail and keeps that -// target's deadline rather than pushing it out, so a steady stream still produces a commit every +// target's deadline rather than pushing it out, so a steady stream still produces a request every // interval instead of starving while refusals keep arriving. A refusal for a DIFFERENT target // never displaces one already recorded, which is the bug this map exists for. -func (l *branchWorkerEventLoop) armTrailingRefusalTouch( +func (l *branchWorkerEventLoop) armTrailingReapply( key refusalKey, detail, observation string, wait time.Duration, ) { if l.refusalPending == nil { - l.refusalPending = map[refusalKey]pendingRefusalTouch{} + l.refusalPending = map[refusalKey]pendingRefusalReapply{} } entry, existing := l.refusalPending[key] entry.key, entry.detail, entry.observation = key, detail, observation @@ -429,13 +353,13 @@ func (l *branchWorkerEventLoop) rearmRefusalTimer() { l.refusalTimer = time.NewTimer(max(time.Until(earliest), 0)) } -// flushPendingRefusalTouch runs every coalesced commit that has come due. +// flushPendingReapply owes every coalesced request that has come due. // // Consent is re-checked per entry rather than trusted from when the refusal arrived, because a // minute is long enough for a target to have been suspended or set back to Ignore, and acting on // stale consent is exactly what a delayed action gets wrong. One target withdrawing consent drops // only its own entry: every other target's stays, which is the whole point of keeping them apart. -func (l *branchWorkerEventLoop) flushPendingRefusalTouch() { +func (l *branchWorkerEventLoop) flushPendingReapply() { now := time.Now() for _, key := range sortedRefusalKeys(l.refusalPending) { entry := l.refusalPending[key] @@ -447,7 +371,7 @@ func (l *branchWorkerEventLoop) flushPendingRefusalTouch() { if !l.w.refusalConsent(l.w.ctx, entry.key.target) { continue } - // Re-checked here as well as on arrival: another commit for this collection may have been made + // Re-checked here as well as on arrival: another request for this collection may have been owed // while this entry waited, and if it covered the same observation this one has nothing // left to ask for. An entry whose collection RECOVERED in the meantime is not here at all — // refusalRecovered removed it. @@ -455,19 +379,19 @@ func (l *branchWorkerEventLoop) flushPendingRefusalTouch() { continue } if limited, wait := l.w.refusalRateLimited(entry.key.target); limited { - // The window moved under us (another refusal for this target committed while this one - // waited). Re-arm rather than commit early. - l.armTrailingRefusalTouch(entry.key, entry.detail, entry.observation, wait) + // The window moved under us (another refusal for this target was requested while this one + // waited). Re-arm rather than request early. + l.armTrailingReapply(entry.key, entry.detail, entry.observation, wait) continue } - l.commitRefusalTouch(entry.key, entry.detail, entry.observation) + l.oweReapply(entry.key, entry.detail, entry.observation) } l.rearmRefusalTimer() } // sortedRefusalKeys makes the flush order deterministic, so a test that queues two targets sees -// the same one committed first every run. -func sortedRefusalKeys(pending map[refusalKey]pendingRefusalTouch) []refusalKey { +// the same one owed first every run. +func sortedRefusalKeys(pending map[refusalKey]pendingRefusalReapply) []refusalKey { keys := make([]refusalKey, 0, len(pending)) for key := range pending { keys = append(keys, key) @@ -476,21 +400,19 @@ func sortedRefusalKeys(pending map[refusalKey]pendingRefusalTouch) []refusalKey return keys } -func (l *branchWorkerEventLoop) commitRefusalTouch(key refusalKey, detail, observation string) { - target := key.target - pendingWrite, err := l.w.buildRefusalTouchWrite(l.w.ctx, target, detail) - if err != nil { - l.w.Log.Error(err, "Cannot build the empty commit for a refused write", - "gitTarget", target.String()) - return - } - - // Decided like every other write, and recorded once its commit exists (settleCommitted): a build - // or commit failure has produced no trigger, so it must leave the next observation of the same - // refusal free to make one. Deciding it while a refused write is being settled appends it to the - // pending writes for the running pass, behind the writes decided before it. - pendingWrite.origin.refusal = &refusalTouchOrigin{key: key, observation: observation} - l.decide(*pendingWrite) +// oweReapply records that the target's applier is owed a Refusal request, which serviceAppliers +// sends once the branch is quiet. Several refusals for one target before then share one request. +func (l *branchWorkerEventLoop) oweReapply(key refusalKey, detail, observation string) { + // Recorded as covered now, when the request is owed, rather than when it is delivered: the + // request is the worker's decision, and a recheck of the same refusal while the branch is still + // busy must not owe a second one. Delivery failures are the requester's to retry. + l.w.recordRefusalObservation(key, observation) + if l.reapplyOwed == nil { + l.reapplyOwed = map[itypes.ResourceReference]string{} + } + l.reapplyOwed[key.target] = detail + l.w.Log.Info("Refused edit owes its applier a reapply, sent once the branch is quiet", + "gitTarget", key.target.String(), "branch", l.w.Branch, "refused", detail) } // refusalIsAWriteBoundary reports whether this refusal is one where the FOLDER is accepted and @@ -498,7 +420,7 @@ func (l *branchWorkerEventLoop) commitRefusalTouch(key refusalKey, detail, obser // // **This is the pruning fence, and it replaces an earlier one that got it wrong.** The first // version stat-ed the object's canonical file under the GitTarget's write path, on the reasoning -// that an object Git does not manage must not be committed for. It disabled the feature for its +// that an object Git does not manage must not be requested for. It disabled the feature for its // main case: a kustomize overlay's base-owned field lives in `../../base`, outside the write // scope, so the canonical case read as "not managed" and was skipped. // @@ -512,7 +434,7 @@ func (l *branchWorkerEventLoop) commitRefusalTouch(key refusalKey, detail, obser // somebody else's delete is what we are avoiding, and a write-boundary refusal cannot cause // one. // -// Every other refusal means the folder is unusable. An empty commit cannot fix a folder, only a +// Every other refusal means the folder is unusable. Reapplying Git cannot fix a folder, only a // human can, and the reconciler may be mid-way through its own corrections there, so those are // left alone. They also reach a different reporter entirely: a folder-level refusal is normally // found by the per-type reconcile, which blocks the collection through the event router rather than @@ -527,7 +449,7 @@ func refusalIsAWriteBoundary(refused *manifestanalyzer.AcceptanceRefusedError) b } // The kind is not enough on its own, and a review found the hole: IssueRenderRefused fires for // a BRAND-NEW resource whose image an existing images: entry would override, and a write that - // is removing a document had one before it. Both would have been committed for. So every issue + // is removing a document had one before it. Both would have been requested for. So every issue // must additionally carry evidence, set only where the pre-write buffer was in hand, that the // folder already holds this document and this write is not removing it. // diff --git a/internal/git/refusal_touch_test.go b/internal/git/refusal_reapply_test.go similarity index 53% rename from internal/git/refusal_touch_test.go rename to internal/git/refusal_reapply_test.go index 6c72d99d..ae8b3418 100644 --- a/internal/git/refusal_touch_test.go +++ b/internal/git/refusal_reapply_test.go @@ -2,12 +2,13 @@ package git -// The empty commit that reverts a refused write, from -// docs/design/push-notification-and-reconcile-trigger.md, part 3 option 7. +// The reapply request that reverts a refused write, from +// docs/design/push-notification-and-reconcile-trigger.md §3.7. // -// The commit itself is two lines; every interesting property is in the decision to make one. A -// refusal is not a one-off, the action is opt-in per GitTarget, and an operator who parked a -// target must not have it driving somebody else's reconciler. Those are what these tests pin. +// The request itself is one call; every interesting property is in the decision to make one. A +// refusal is not a one-off, the action is opt-in per GitTarget and needs an applier, and an +// operator who parked a target must not have it driving the applier. Those are what these tests +// pin. import ( "context" @@ -15,9 +16,11 @@ import ( "os/exec" "path/filepath" "strings" + "sync" "testing" "time" + meta "github.com/fluxcd/pkg/apis/meta" "github.com/go-logr/logr" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -32,7 +35,8 @@ import ( itypes "github.com/ConfigButler/gitops-reverser/internal/types" ) -// refusalTouchWorker builds a worker whose fake client holds one GitTarget with the given spec. +// refusalTouchWorker builds a worker whose fake client holds one GitTarget with the given spec. A +// spec asking for Reapply gets an applier, which the API requires of it. func refusalTouchWorker(t *testing.T, spec configv1alpha3.GitTargetSpec) *BranchWorker { t.Helper() scheme := runtime.NewScheme() @@ -44,6 +48,9 @@ func refusalTouchWorker(t *testing.T, spec configv1alpha3.GitTargetSpec) *Branch if spec.Path == "" { spec.Path = "apps" } + if spec.OnRefusal == configv1alpha3.RefusalActionReapply && spec.Applier == nil { + spec.Applier = testApplier("editing") + } target := &configv1alpha3.GitTarget{ ObjectMeta: metav1.ObjectMeta{Name: "editing", Namespace: "team-a"}, Spec: spec, @@ -51,6 +58,9 @@ func refusalTouchWorker(t *testing.T, spec configv1alpha3.GitTargetSpec) *Branch client := fake.NewClientBuilder().WithScheme(scheme).WithObjects(target).Build() w := newMetricsTestWorker() + // A GitTarget names its GitProvider in its own namespace, which is where a worker looks for + // the appliers on its branch. + w.GitProviderNamespace = target.Namespace w.Client = client w.Log = logr.Discard() w.ctx = context.Background() @@ -69,17 +79,17 @@ func refusalKeyFor(target itypes.ResourceReference, resource string) refusalKey func editingKey() refusalKey { return refusalKeyFor(editingRef(), "deployments") } -// TestRefusalTouch_OffByDefault. The action deletes objects where the reconciler prunes, so the +// TestRefusalReapply_OffByDefault. The action deletes objects where the applier prunes, so the // default has to be the one that does nothing. -func TestRefusalTouch_OffByDefault(t *testing.T) { +func TestRefusalReapply_OffByDefault(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{}) assert.False(t, w.refusalConsent(w.ctx, editingRef()), - "a GitTarget that did not ask for an empty commit must not get one") + "a GitTarget that did not ask for a reapply must not get one") } -// TestRefusalTouch_ExplicitIgnoreIsAlsoOff guards the enum rather than only the zero value. -func TestRefusalTouch_ExplicitIgnoreIsAlsoOff(t *testing.T) { +// TestRefusalReapply_ExplicitIgnoreIsAlsoOff guards the enum rather than only the zero value. +func TestRefusalReapply_ExplicitIgnoreIsAlsoOff(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ OnRefusal: configv1alpha3.RefusalActionIgnore, }) @@ -87,117 +97,102 @@ func TestRefusalTouch_ExplicitIgnoreIsAlsoOff(t *testing.T) { assert.False(t, w.refusalConsent(w.ctx, editingRef())) } -// TestRefusalTouch_AllowedWhenAskedFor is the positive case. -func TestRefusalTouch_AllowedWhenAskedFor(t *testing.T) { +// TestRefusalReapply_AllowedWhenAskedFor is the positive case. +func TestRefusalReapply_AllowedWhenAskedFor(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) assert.True(t, w.refusalConsent(w.ctx, editingRef())) } -// TestRefusalTouch_SuspendedTargetIsNeverTouched. An empty commit is a write, and it drives -// another controller. Suspension has to stop the one write that reaches outside this operator. -func TestRefusalTouch_SuspendedTargetIsNeverTouched(t *testing.T) { +// TestRefusalReapply_SuspendedTargetIsNeverTouched. A suspended target has stopped publishing the +// live edits a reapply would be weighed against, so it must not drive its applier. +func TestRefusalReapply_SuspendedTargetIsNeverTouched(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, Suspend: true, }) assert.False(t, w.refusalConsent(w.ctx, editingRef()), - "a suspended target must not drive the reconciler") + "a suspended target must not drive the applier") } -// TestRefusalTouch_RateLimitedPerTarget. A controller rewriting a base-owned field produces a -// refusal on every one of its reconciles; without a floor the branch fills with empty commits and -// every one of them wakes every reconciler watching it. -func TestRefusalTouch_RateLimitedPerTarget(t *testing.T) { +// TestRefusalReapply_NoApplierIsNoConsent. The API rejects Reapply without an applier, but a target +// stored before that rule must not reach for an applier it does not name. +func TestRefusalReapply_NoApplierIsNoConsent(t *testing.T) { + w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{}) + target := &configv1alpha3.GitTarget{} + require.NoError(t, w.Client.Get(w.ctx, client.ObjectKey{Name: "editing", Namespace: "team-a"}, target)) + target.Spec.OnRefusal = configv1alpha3.RefusalActionReapply + require.NoError(t, w.Client.Update(w.ctx, target)) + + assert.False(t, w.refusalConsent(w.ctx, editingRef())) +} + +// TestRefusalReapply_RateLimitedPerTarget. A controller rewriting a base-owned field produces a +// refusal on every one of its reconciles; without a floor the applier would be asked to reapply on +// every one of them. +func TestRefusalReapply_RateLimitedPerTarget(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) limited, _ := w.refusalRateLimited(editingRef()) - require.False(t, limited, "the first refusal commits") + require.False(t, limited, "the first refusal is requested") limited, wait := w.refusalRateLimited(editingRef()) assert.True(t, limited, "an immediate second one is rate limited") assert.Positive(t, wait, "a rate-limited refusal must say when it may be retried") - assert.LessOrEqual(t, wait, refusalTouchInterval) + assert.LessOrEqual(t, wait, refusalReapplyInterval) // Age the record past the floor: the next refusal is allowed again. - w.refusalTouchMu.Lock() - w.lastRefusalTouch[editingRef().String()] = time.Now().Add(-2 * refusalTouchInterval) - w.refusalTouchMu.Unlock() + w.refusalMu.Lock() + w.lastReapply[editingRef().String()] = time.Now().Add(-2 * refusalReapplyInterval) + w.refusalMu.Unlock() limited, _ = w.refusalRateLimited(editingRef()) assert.False(t, limited, "the floor is a rate, not a one-shot") } -// TestRefusalTouch_UnreadableTargetIsNotConsent. An unreadable GitTarget is missing evidence, not -// evidence of permission, and the action is destructive where the reconciler prunes. -func TestRefusalTouch_UnreadableTargetIsNotConsent(t *testing.T) { +// TestRefusalReapply_UnreadableTargetIsNotConsent. An unreadable GitTarget is missing evidence, not +// evidence of permission, and the action is destructive where the applier prunes. +func TestRefusalReapply_UnreadableTargetIsNotConsent(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) assert.False(t, w.refusalConsent(w.ctx, itypes.NewResourceReference("absent", "team-a")), "a GitTarget that cannot be read must not be treated as opted in") } -// TestRefusalCommitMessage_ExplainsTheEmptyDiff. The message is the whole user-visible product of -// this feature for anyone reading git log, and an unexplained empty commit looks like a mistake. -func TestRefusalCommitMessage_ExplainsTheEmptyDiff(t *testing.T) { - message := refusalCommitMessage(editingRef(), "unsupported folder content") - - assert.Contains(t, message, "team-a/editing") - assert.Contains(t, message, "changes no file") - assert.Contains(t, message, "unsupported folder content") -} - -// TestRefusalTouch_LandsAnEmptyCommitOnTheRemote is the claim the whole option rests on: the -// branch moves, and no file changed. Both halves matter. A commit that changed something would be -// a write nobody asked for; a branch that did not move would trigger nothing, because a new -// revision is the entire mechanism (Argo CD's selfHeal=false skip is keyed on having already -// synced THIS revision, and Flux needs a new artifact revision). -// -// It runs over a real Git server rather than file://, because go-git's in-process receive-pack -// never compares cmd.Old, so a compare-and-swap assertion over file:// passes vacuously. -func TestRefusalTouch_LandsAnEmptyCommitOnTheRemote(t *testing.T) { - f := newLedgerFixture(t, "refusal-touch", true) - f.createLedgerTarget("team-a", nil) - f.publish("before") - - beforeSHA, beforeTree := remoteHeadAndTree(t, f.repoDir, "main") - - write, err := f.worker.buildRefusalTouchWrite( - f.worker.ctx, itypes.NewResourceReference("target-a", "default"), "unsupported folder content") - require.NoError(t, err) - - batch := []PendingWrite{*write} - require.NoError(t, f.worker.commitPendingWrites(batch)) - f.pending = append(f.pending, batch...) - f.push() - - afterSHA, afterTree := remoteHeadAndTree(t, f.repoDir, "main") - - assert.NotEqual(t, beforeSHA, afterSHA, "the branch must move, or nothing is triggered") - assert.Equal(t, beforeTree, afterTree, "the commit must change no file") - assert.Equal(t, beforeSHA, parentOf(t, f.repoDir, afterSHA), - "the empty commit must sit directly on the previous tip") -} - -// remoteHeadAndTree reads the bare repository directly, so the assertion is about what the SERVER -// holds rather than about what our worker believes it pushed. -func remoteHeadAndTree(t *testing.T, repoDir, branch string) (string, string) { - t.Helper() - return gitOut(t, repoDir, "rev-parse", branch), gitOut(t, repoDir, "rev-parse", branch+"^{tree}") -} +// TestRefusalReapply_OwedRequestWaitsForAQuietBranch is the claim the action rests on: the request +// names the refusing target alone, and it is not sent while anything live is unpublished, because an +// apply then would revert that too. +func TestRefusalReapply_OwedRequestWaitsForAQuietBranch(t *testing.T) { + w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{OnRefusal: configv1alpha3.RefusalActionReapply}) + requester := &recordingRequester{} + w.applyRequester = requester + loop := newBranchWorkerEventLoop(w, time.Minute) + t.Cleanup(loop.stopTimers) -func parentOf(t *testing.T, repoDir, sha string) string { - t.Helper() - return gitOut(t, repoDir, "rev-parse", sha+"^") + loop.openWindow = &openWindow{} + loop.oweReapply(editingKey(), "a base-owned field", "observation-1") + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "nothing is sent while a commit window is open") + + loop.openWindow = nil + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, configv1alpha3.ApplyReasonRefusal, requests[0].Reason) + assert.Equal(t, []itypes.ResourceReference{editingRef()}, requests[0].Targets, + "only the refusing target's applier is asked") + assert.True(t, w.refusalAlreadyCovered(editingKey(), "observation-1"), + "the observation is covered once the request is owed") } +// gitOut runs git against a repository on disk and returns its trimmed output. func gitOut(t *testing.T, repoDir string, args ...string) string { t.Helper() out, err := exec.Command("git", append([]string{"-C", repoDir}, args...)...).CombinedOutput() @@ -205,40 +200,40 @@ func gitOut(t *testing.T, repoDir string, args ...string) string { return strings.TrimSpace(string(out)) } -// TestRefusalTouch_RateLimitedRefusalIsCoalescedNotDropped is the regression for a review finding. +// TestRefusalReapply_RateLimitedRefusalIsCoalescedNotDropped is the regression for a review finding. // -// Dropping a rate-limited refusal is a correctness bug, not a missed optimisation. The reconcile -// the previous commit triggered may already have finished by the time the second refusal happens, -// so nothing would ever cover it and the live edit would stay in the cluster for good. The second -// refusal has to leave a trailing commit armed. -func TestRefusalTouch_RateLimitedRefusalIsCoalescedNotDropped(t *testing.T) { +// Dropping a rate-limited refusal is a correctness bug, not a missed optimisation. The apply the +// previous request caused may already have finished by the time the second refusal happens, so +// nothing would ever cover it and the live edit would stay in the cluster for good. The second +// refusal has to leave a trailing request armed. +func TestRefusalReapply_RateLimitedRefusalIsCoalescedNotDropped(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) loop := newBranchWorkerEventLoop(w, time.Minute) t.Cleanup(loop.stopTimers) - // Consume the window, as a first refusal that committed would. + // Consume the window, as a first refusal that was requested would. limited, _ := w.refusalRateLimited(editingRef()) require.False(t, limited) - loop.armTrailingRefusalTouch(editingKey(), "a second refusal", "observation-2", time.Minute) + loop.armTrailingReapply(editingKey(), "a second refusal", "observation-2", time.Minute) - assert.NotNil(t, loop.refusalTimer, "a coalesced refusal must leave a trailing commit armed") + assert.NotNil(t, loop.refusalTimer, "a coalesced refusal must leave a trailing request armed") require.Contains(t, loop.refusalPending, editingKey()) assert.Equal(t, "a second refusal", loop.refusalPending[editingKey()].detail) } -// TestRefusalTouch_TrailingCommitRechecksConsent. A minute is long enough for the target to be +// TestRefusalReapply_TrailingRequestRechecksConsent. A minute is long enough for the target to be // suspended or set back to Ignore, and acting on consent captured a minute ago is exactly what a // delayed action gets wrong. -func TestRefusalTouch_TrailingCommitRechecksConsent(t *testing.T) { +func TestRefusalReapply_TrailingRequestRechecksConsent(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) loop := newBranchWorkerEventLoop(w, time.Minute) t.Cleanup(loop.stopTimers) - loop.armTrailingRefusalTouch(editingKey(), "queued while consent still stood", "observation-1", 0) + loop.armTrailingReapply(editingKey(), "queued while consent still stood", "observation-1", 0) setSpec := func(spec configv1alpha3.GitTargetSpec) { target := &configv1alpha3.GitTarget{} @@ -253,17 +248,17 @@ func TestRefusalTouch_TrailingCommitRechecksConsent(t *testing.T) { // The select arm owns the timer (it nils it before dispatching), so the flush is called here // the way the loop calls it. loop.stopRefusalTimer() - loop.flushPendingRefusalTouch() + loop.flushPendingReapply() loop.endWake(0) assert.Empty(t, loop.refusalPending, - "a target set back to Ignore must not get the commit queued under the old setting") + "a target set back to Ignore must not get the request queued under the old setting") assert.Nil(t, loop.refusalTimer, "and nothing must be re-armed for a target that said no") } -// TestRefusalTouch_OnlyWriteBoundaryRefusalsCommit is the first half of the pruning fence: the +// TestRefusalReapply_OnlyWriteBoundaryRefusalsRequest is the first half of the pruning fence: the // refusal has to be one where the edit had nowhere to land while the folder itself is accepted. -func TestRefusalTouch_OnlyWriteBoundaryRefusalsCommit(t *testing.T) { +func TestRefusalReapply_OnlyWriteBoundaryRefusalsRequest(t *testing.T) { cases := []struct { name string kind manifestanalyzer.IssueKind @@ -293,12 +288,12 @@ func TestRefusalTouch_OnlyWriteBoundaryRefusalsCommit(t *testing.T) { } } -// TestRefusalTouch_RequiresEvidenceTheDocumentExists is the second half, and the one a review +// TestRefusalReapply_RequiresEvidenceTheDocumentExists is the second half, and the one a review // found missing. The refusal KIND cannot establish that the folder holds the object: a write that // is REMOVING a document is a write-boundary refusal too, and so is one for an object that was // never there. Re-applying does nothing for the second and prunes the first, and hurrying somebody // else's delete is what the fence exists to prevent. -func TestRefusalTouch_RequiresEvidenceTheDocumentExists(t *testing.T) { +func TestRefusalReapply_RequiresEvidenceTheDocumentExists(t *testing.T) { boundary := manifestanalyzer.IssueWriteEscapesScope withEvidence := &manifestanalyzer.AcceptanceRefusedError{ @@ -310,10 +305,10 @@ func TestRefusalTouch_RequiresEvidenceTheDocumentExists(t *testing.T) { Issues: []manifestanalyzer.AcceptanceIssue{{Kind: boundary, Path: "a.yaml"}}, } assert.False(t, refusalIsAWriteBoundary(noEvidence), - "an object the folder does not already hold must not be committed for") + "an object the folder does not already hold must not be requested for") // A batch is only safe when EVERY issue carries the evidence: one unestablished document is - // enough to make the commit a guess. + // enough to make the request a guess. mixedEvidence := &manifestanalyzer.AcceptanceRefusedError{ Issues: []manifestanalyzer.AcceptanceIssue{ {Kind: boundary, Path: "a.yaml", ExistingDocument: true}, @@ -323,9 +318,9 @@ func TestRefusalTouch_RequiresEvidenceTheDocumentExists(t *testing.T) { assert.False(t, refusalIsAWriteBoundary(mixedEvidence)) } -// TestRefusalTouch_MixedRefusalDoesNotCommit. A batch carrying any folder-level issue is a folder +// TestRefusalReapply_MixedRefusalDoesNotRequest. A batch carrying any folder-level issue is a folder // problem, so it is left alone even though a write-boundary issue rides along with it. -func TestRefusalTouch_MixedRefusalDoesNotCommit(t *testing.T) { +func TestRefusalReapply_MixedRefusalDoesNotRequest(t *testing.T) { refused := &manifestanalyzer.AcceptanceRefusedError{ Issues: []manifestanalyzer.AcceptanceIssue{ {Kind: manifestanalyzer.IssueUnplaceableEdit, Path: "a.yaml", ExistingDocument: true}, @@ -334,72 +329,18 @@ func TestRefusalTouch_MixedRefusalDoesNotCommit(t *testing.T) { } assert.False(t, refusalIsAWriteBoundary(refused)) - assert.False(t, refusalIsAWriteBoundary(nil), "a non-refusal never commits") + assert.False(t, refusalIsAWriteBoundary(nil), "a non-refusal never requests") assert.False(t, refusalIsAWriteBoundary(&manifestanalyzer.AcceptanceRefusedError{}), "a refusal with no issues says nothing about the folder") } -// TestRefusalTouch_ReplayKeepsAcceptedWritesAndTheEmptyDiff is the contention case a review asked -// for, and it lives here rather than in e2e on purpose: the property is about ARRIVAL ORDER -// between our push and somebody else's, and the bi-directional corner does not control that. -// Here the contending commit is made on disk, between our commit and our push, so the rejection is -// deterministic instead of hoped for. -// -// The three things it pins, which is what makes a mixed batch safe: -// -// - The accepted write survives the replay. A refusal must not cost the writes that were fine. -// - The empty commit is still empty after being replanned onto a moved tree. Replay re-plans -// each retained write, and a write with no content must come back with no content rather than -// picking anything up from the new base. -// - The other writer's file is still there. We rebase onto their work, never over it. -func TestRefusalTouch_ReplayKeepsAcceptedWritesAndTheEmptyDiff(t *testing.T) { - f := newLedgerFixture(t, "refusal-replay", true) - f.createLedgerTarget("team-a", nil) - f.publish("prime") - - // An ordinary accepted write, committed and retained but not yet pushed. - f.commit("accepted") - require.Len(t, f.pending, 1) - - // The refusal's empty commit joins the same retained batch, exactly as the event loop would - // append it while an earlier write is still waiting for the push cooldown. - touch, err := f.worker.buildRefusalTouchWrite( - f.worker.ctx, itypes.NewResourceReference("target-a", "default"), "unsupported folder content") - require.NoError(t, err) - batch := []PendingWrite{*touch} - require.NoError(t, f.worker.commitPendingWrites(batch)) - f.pending = append(f.pending, batch...) - - // Somebody else moves the branch before our push, so the compare-and-swap rejects it and the - // worker replays BOTH retained writes onto their tip. - f.contend("OUTSIDE.md", "from-another-writer\n") - - f.push() - - head := gitOut(t, f.repoDir, "rev-parse", "main") - tip, parent := head, gitOut(t, f.repoDir, "rev-parse", head+"^") - - // Identify the commits by their messages rather than trusting the replay's ordering, so a - // reordering shows up as a failure here instead of silently retargeting the assertions below. - assert.Contains(t, gitOut(t, f.repoDir, "log", "-1", "--format=%B", tip), "refused write", - "the tip must be the refusal commit") - assert.Empty(t, gitOut(t, f.repoDir, "diff-tree", "--no-commit-id", "--name-only", "-r", tip), - "the refusal commit must still be empty after being replayed onto a moved branch") - assert.Contains(t, gitOut(t, f.repoDir, "diff-tree", "--no-commit-id", "--name-only", "-r", parent), - "accepted", "the accepted write must survive the replay") - assert.Equal(t, "from-another-writer", - gitOut(t, f.repoDir, "show", "main:OUTSIDE.md"), - "the other writer's commit must be rebased onto, never over") -} - -// TestRefusalTouch_ANewResourceAnEntryWouldOverrideIsNotCommittedFor drives the fence through the +// TestRefusalReapply_ANewResourceAnEntryWouldOverrideIsNotRequestedFor drives the fence through the // REAL analyzer rather than a hand-built issue, because that is how a review found the hole. // // A brand-new Deployment whose image an existing images: entry would override is refused, and the // refusal is a write-boundary one. Nothing in Git holds that object, so re-applying would not -// correct it: with pruning on it is removed, and otherwise nothing happens. Committing for it was -// the promise in spec.onRefusal's own documentation being false. -func TestRefusalTouch_ANewResourceAnEntryWouldOverrideIsNotCommittedFor(t *testing.T) { +// correct it: with pruning on it is removed, and otherwise nothing happens. +func TestRefusalReapply_ANewResourceAnEntryWouldOverrideIsNotRequestedFor(t *testing.T) { writer := newContentWriter(itypes.SensitiveResourcePolicy{}) worktree := newWorktreeForTest(t) root := worktree.Filesystem().Root() @@ -423,14 +364,14 @@ images: var refused *manifestanalyzer.AcceptanceRefusedError require.ErrorAs(t, err, &refused, "the analyzer must still refuse this write") assert.False(t, refusalIsAWriteBoundary(refused), - "an object Git does not hold must not earn a commit, whatever the refusal kind says") + "an object Git does not hold must not earn a request, whatever the refusal kind says") } -// TestRefusalTouch_AnEditToAnExistingDocumentIsCommittedFor is the positive control for the test +// TestRefusalReapply_AnEditToAnExistingDocumentIsRequestedFor is the positive control for the test // above: the same machinery, an object the folder DOES hold, and the fence opens. // // Without this, the fence could pass its negative tests by never opening at all. -func TestRefusalTouch_AnEditToAnExistingDocumentIsCommittedFor(t *testing.T) { +func TestRefusalReapply_AnEditToAnExistingDocumentIsRequestedFor(t *testing.T) { issue := manifestanalyzer.AcceptanceIssue{ Kind: manifestanalyzer.IssueWriteEscapesScope, Path: "base/deployment.yaml", @@ -443,13 +384,13 @@ func TestRefusalTouch_AnEditToAnExistingDocumentIsCommittedFor(t *testing.T) { &manifestanalyzer.AcceptanceRefusedError{Issues: []manifestanalyzer.AcceptanceIssue{issue}})) } -// TestRefusalTouch_OneTargetDoesNotDropAnothersPendingCommit is the regression a review found. +// TestRefusalReapply_OneTargetDoesNotDropAnothersPendingRequest is the regression a review found. // // One branch worker serves every GitTarget on its (provider, branch), while the rate limit and // the consent check are both per target. With a single pending slot, B's refusal replaced A's; B -// was then suspended, its consent check dropped it, and A's authorized commit was gone with -// nothing left to report it. Keeping the intent per target is what makes coalesced EXECUTION safe. -func TestRefusalTouch_OneTargetDoesNotDropAnothersPendingCommit(t *testing.T) { +// was then suspended, its consent check dropped it, and A's authorized request was gone with +// nothing left to report it. +func TestRefusalReapply_OneTargetDoesNotDropAnothersPendingRequest(t *testing.T) { scheme := runtime.NewScheme() require.NoError(t, clientgoscheme.AddToScheme(scheme)) require.NoError(t, configv1alpha3.AddToScheme(scheme)) @@ -459,14 +400,15 @@ func TestRefusalTouch_OneTargetDoesNotDropAnothersPendingCommit(t *testing.T) { ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "team-a"}, Spec: configv1alpha3.GitTargetSpec{ Branch: "main", Path: "apps/" + name, - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, + Applier: testApplier(name), Suspend: suspend, }, } target.Spec.GitProviderRef.Name = "test-provider" return target } - // B is suspended, so its delayed commit must be declined when the timer fires. A is not. + // B is suspended, so its delayed request must be declined when the timer fires. A is not. client := fake.NewClientBuilder().WithScheme(scheme). WithObjects(newTarget("alpha", false), newTarget("bravo", true)).Build() @@ -481,31 +423,31 @@ func TestRefusalTouch_OneTargetDoesNotDropAnothersPendingCommit(t *testing.T) { // A zero wait makes both entries due the moment they are recorded, so the flush below cannot // race their deadlines. The deadline itself is covered by its own test. - loop.armTrailingRefusalTouch(refusalKeyFor(alpha, "deployments"), "alpha was refused", "alpha-observation", 0) - loop.armTrailingRefusalTouch(refusalKeyFor(bravo, "deployments"), "bravo was refused", "bravo-observation", 0) + loop.armTrailingReapply(refusalKeyFor(alpha, "deployments"), "alpha was refused", "alpha-observation", 0) + loop.armTrailingReapply(refusalKeyFor(bravo, "deployments"), "bravo was refused", "bravo-observation", 0) require.Len(t, loop.refusalPending, 2, - "two targets on one worker must hold two pending commits, not one") + "two targets on one worker must hold two pending requests, not one") // bravo's consent is withdrawn; alpha's is not, and alpha must survive it. loop.stopRefusalTimer() - loop.flushPendingRefusalTouch() + loop.flushPendingReapply() loop.endWake(0) assert.Empty(t, loop.refusalPending, "every due entry is consumed exactly once") limitedAlpha, _ := w.refusalRateLimited(alpha) assert.True(t, limitedAlpha, - "alpha's commit must have run, consuming its rate-limit window") + "alpha's request must have been owed, consuming its rate-limit window") limitedBravo, _ := w.refusalRateLimited(bravo) assert.False(t, limitedBravo, "bravo's was declined for lack of consent, so it must not have consumed a window") } -// TestRefusalTouch_TheTimerTracksTheEarliestDeadline. One timer serves every pending target, so it +// TestRefusalReapply_TheTimerTracksTheEarliestDeadline. One timer serves every pending target, so it // has to wake for whichever is due first or a later entry would delay an earlier one. -func TestRefusalTouch_TheTimerTracksTheEarliestDeadline(t *testing.T) { +func TestRefusalReapply_TheTimerTracksTheEarliestDeadline(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{ - OnRefusal: configv1alpha3.RefusalActionPushEmptyCommit, + OnRefusal: configv1alpha3.RefusalActionReapply, }) loop := newBranchWorkerEventLoop(w, time.Minute) t.Cleanup(loop.stopTimers) @@ -513,8 +455,8 @@ func TestRefusalTouch_TheTimerTracksTheEarliestDeadline(t *testing.T) { late := itypes.NewResourceReference("late", "team-a") soon := itypes.NewResourceReference("soon", "team-a") - loop.armTrailingRefusalTouch(refusalKeyFor(late, "deployments"), "later", "late-observation", time.Hour) - loop.armTrailingRefusalTouch(refusalKeyFor(soon, "deployments"), "sooner", "soon-observation", 50*time.Millisecond) + loop.armTrailingReapply(refusalKeyFor(late, "deployments"), "later", "late-observation", time.Hour) + loop.armTrailingReapply(refusalKeyFor(soon, "deployments"), "sooner", "soon-observation", 50*time.Millisecond) require.Len(t, loop.refusalPending, 2) select { @@ -523,3 +465,50 @@ func TestRefusalTouch_TheTimerTracksTheEarliestDeadline(t *testing.T) { t.Fatal("the shared timer must wake for the earliest deadline, not the last one armed") } } + +// testApplier is a GitTarget applier naming a Kustomization of the given name. +func testApplier(name string) *configv1alpha3.GitTargetApplier { + return &configv1alpha3.GitTargetApplier{KustomizationRef: meta.NamespacedObjectReference{Name: name}} +} + +// recordingRequester records every apply request a worker sends. +type recordingRequester struct { + mu sync.Mutex + requests []ApplyRequest + // counted keeps the reasons of the requests taken, so count still sees them. + counted map[configv1alpha3.ApplyReason]int +} + +func (r *recordingRequester) RequestApply(req ApplyRequest) { + r.mu.Lock() + defer r.mu.Unlock() + r.requests = append(r.requests, req) +} + +// count is how many requests with this reason were recorded, taken or not. +func (r *recordingRequester) count(reason configv1alpha3.ApplyReason) int { + r.mu.Lock() + defer r.mu.Unlock() + n := r.counted[reason] + for _, req := range r.requests { + if req.Reason == reason { + n++ + } + } + return n +} + +// taken returns the requests recorded so far and forgets them. +func (r *recordingRequester) taken() []ApplyRequest { + r.mu.Lock() + defer r.mu.Unlock() + requests := r.requests + r.requests = nil + if r.counted == nil { + r.counted = map[configv1alpha3.ApplyReason]int{} + } + for _, req := range requests { + r.counted[req.Reason]++ + } + return requests +} diff --git a/internal/git/types.go b/internal/git/types.go index 99c8040a..2511cef5 100644 --- a/internal/git/types.go +++ b/internal/git/types.go @@ -259,13 +259,7 @@ const ( // content-derived mark-and-sweep against the worktree (upsert every desired // resource, drop every watched managed document the snapshot did not contain). PendingWriteResync PendingWriteKind = "resync" - // PendingWriteRefusalTouch carries no content at all. It exists to move the branch after a - // refused write, so the reconciler re-applies the desired state and reverts the live edit a - // refusal left in place. It is the only write kind that deliberately produces a commit with - // an empty tree diff, and it is created only for a GitTarget with - // spec.onRefusal: PushEmptyCommit. - PendingWriteRefusalTouch PendingWriteKind = "refusal_touch" - // PendingWriteRequestRecord carries no content either. It records a CommitRequest's message + // PendingWriteRequestRecord carries no content. It records a CommitRequest's message // in an empty commit when the request ran out of time to attach, and asked for that with // whenNothingToCommit: CommitEmpty. It is not a window: only writes open windows. PendingWriteRequestRecord PendingWriteKind = "request_record" diff --git a/internal/git/worker_manager.go b/internal/git/worker_manager.go index 8f027bbc..4182e543 100644 --- a/internal/git/worker_manager.go +++ b/internal/git/worker_manager.go @@ -163,6 +163,11 @@ type WorkerManager struct { // publicationEvents carries the GitTargets whose worker's publication report changed. See // PublicationEvents. publicationEvents chan event.GenericEvent + + // applyRequester delivers every worker's apply requests. Set once at startup + // (SetApplyRequester) before any worker is created; nil in the CLI and in tests, which switches + // the apply trigger off. + applyRequester ApplyRequester } // NewWorkerManager creates a new worker manager. limits bounds every worker this manager @@ -256,6 +261,14 @@ func (m *WorkerManager) SetScanAcceptanceReporter(reporter ScanAcceptanceReporte m.scanAcceptance = reporter } +// SetApplyRequester injects what delivers every worker's apply requests. Like SetMapper, it is called +// once at startup before any worker is created. +func (m *WorkerManager) SetApplyRequester(requester ApplyRequester) { + m.mu.Lock() + defer m.mu.Unlock() + m.applyRequester = requester +} + // recordRemoteObservation is the delivery point for one branch's confirmed observation. Every // worker is given it bound to its own key; it is called on the worker's loop goroutine. func (m *WorkerManager) recordRemoteObservation(key BranchKey, observed RemoteObservation) { @@ -373,6 +386,7 @@ func (m *WorkerManager) EnsureWorker( managerCtx := m.ctx // read under m.mu; the worker's loop calls the reporter later worker.publicationReporter = func() { go m.notifyPublication(managerCtx, key) } worker.scanAcceptance = m.scanAcceptance + worker.applyRequester = m.applyRequester worker.renderFidelityGate = m.renderFidelityGate worker.crOwners = m.commitRequests @@ -697,3 +711,15 @@ func (m *WorkerManager) liveWorkers() []*BranchWorker { func (m *WorkerManager) NeedLeaderElection() bool { return true } + +// NotifyPush tells the worker for a provider's branch that the Git host reported a push to it. It +// reports whether such a worker exists. after is the commit the delivery named; it is logged and +// never trusted: the worker reads where the branch is from the remote itself. +func (m *WorkerManager) NotifyPush(providerNamespace, providerName, branch, after string) bool { + worker, found := m.GetWorkerForTarget(providerName, providerNamespace, branch) + if !found { + return false + } + worker.EnqueuePushNotification(after) + return true +} diff --git a/internal/repourl/repourl.go b/internal/repourl/repourl.go new file mode 100644 index 00000000..115f2495 --- /dev/null +++ b/internal/repourl/repourl.go @@ -0,0 +1,107 @@ +// SPDX-License-Identifier: Apache-2.0 + +/* +Package repourl decides whether two spellings of a Git repository URL name the same repository. + +One repository has many spellings: GitHub hands out https://github.com/acme/platform.git, +git@github.com:acme/platform.git and https://github.com/acme/platform in the same payload, and a +GitProvider's spec.url can be any of them. Comparing the strings would call those three different +repositories, so a relay that names the repository one way would be ignored for a provider that +spells it another. + +The comparison is deliberately loose about transport and strict about identity: scheme, userinfo, +default ports, a trailing ".git" and case are dropped, while the host, a non-default port and the +path are kept. Case is dropped from the path too, because the hosts this is used with (GitHub, +GitLab, Gitea) resolve repository paths case-insensitively. +*/ +package repourl + +import ( + "net" + "net/url" + "strings" +) + +// isDefaultPort reports whether a URL can name port without changing which server it reaches: SSH, +// HTTPS and HTTP. Any other port is part of the identity, because it can be a different server. +func isDefaultPort(port string) bool { + switch port { + case "22", "443", "80": + return true + default: + return false + } +} + +// Normalize returns the canonical "host[:port]/path" form of a repository URL, or "" for an empty +// one. It accepts URL forms (https://, http://, ssh://, git://) and the scp-like SSH form +// "[user@]host:path". A string it cannot parse is returned lowercased and trimmed, so it can still +// only ever equal itself. +func Normalize(raw string) string { + raw = strings.TrimSpace(raw) + if raw == "" { + return "" + } + + host, path, ok := splitSCPLike(raw) + if !ok { + u, err := url.Parse(raw) + if err != nil || u.Host == "" { + return strings.ToLower(raw) + } + host, path = u.Host, u.Path + } + + return canonicalHost(host) + "/" + canonicalPath(path) +} + +// Same reports whether a and b name the same repository. Two empty URLs are not the same +// repository: an absent value is no evidence of a match. +func Same(a, b string) bool { + na := Normalize(a) + return na != "" && na == Normalize(b) +} + +// splitSCPLike recognizes the scp-like SSH form git uses, "[user@]host:path", and returns its host +// and path. It is the form only when there is no scheme and the first colon comes before the first +// slash: "host:org/repo" is scp-like, "./a:b" is a local path. +func splitSCPLike(raw string) (string, string, bool) { + if strings.Contains(raw, "://") { + return "", "", false + } + colon := strings.Index(raw, ":") + if colon <= 0 { + return "", "", false + } + if slash := strings.Index(raw, "/"); slash >= 0 && slash < colon { + return "", "", false + } + host := raw[:colon] + if at := strings.LastIndex(host, "@"); at >= 0 { + host = host[at+1:] + } + return host, raw[colon+1:], true +} + +// canonicalHost lowercases a host, drops any userinfo, and drops the port when it is a default one. +func canonicalHost(host string) string { + if at := strings.LastIndex(host, "@"); at >= 0 { + host = host[at+1:] + } + host = strings.ToLower(host) + if h, port, err := net.SplitHostPort(host); err == nil && isDefaultPort(port) { + if strings.Contains(h, ":") { + return "[" + h + "]" // an IPv6 literal keeps the brackets it has without a port + } + return h + } + return host +} + +// canonicalPath lowercases a repository path and strips the decoration that does not change which +// repository it names: surrounding slashes and a trailing ".git". +func canonicalPath(path string) string { + path = strings.ToLower(strings.Trim(path, "/")) + path = strings.TrimSuffix(path, ".git") + return strings.TrimRight(path, "/") +} diff --git a/internal/repourl/repourl_test.go b/internal/repourl/repourl_test.go new file mode 100644 index 00000000..b0d9c398 --- /dev/null +++ b/internal/repourl/repourl_test.go @@ -0,0 +1,69 @@ +// SPDX-License-Identifier: Apache-2.0 + +package repourl + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestNormalize(t *testing.T) { + tests := map[string]struct { + in string + want string + }{ + "https with .git": {"https://github.com/acme/platform.git", "github.com/acme/platform"}, + "https without .git": {"https://github.com/acme/platform", "github.com/acme/platform"}, + "trailing slash": {"https://github.com/acme/platform/", "github.com/acme/platform"}, + "trailing slash after .git": {"https://github.com/acme/platform.git/", "github.com/acme/platform"}, + "scp-like ssh": {"git@github.com:acme/platform.git", "github.com/acme/platform"}, + "scp-like without user": {"github.com:acme/platform", "github.com/acme/platform"}, + "ssh url": {"ssh://git@github.com/acme/platform.git", "github.com/acme/platform"}, + "ssh default port": {"ssh://git@github.com:22/acme/platform.git", "github.com/acme/platform"}, + "https default port": {"https://github.com:443/acme/platform", "github.com/acme/platform"}, + "http default port": {"http://gitea.local:80/acme/platform", "gitea.local/acme/platform"}, + "non-default port is kept": {"ssh://git@gitea.local:2222/acme/platform", "gitea.local:2222/acme/platform"}, + "userinfo dropped": {"https://user:secret@github.com/acme/platform", "github.com/acme/platform"}, + "case folded": {"https://GitHub.com/Acme/Platform.GIT", "github.com/acme/platform"}, + "surrounding whitespace": {" https://github.com/acme/platform ", "github.com/acme/platform"}, + "nested group path": {"https://gitlab.com/acme/team/platform.git", "gitlab.com/acme/team/platform"}, + "ipv6 with default port": {"ssh://git@[::1]:22/acme/platform", "[::1]/acme/platform"}, + "ipv6 without port": {"ssh://git@[::1]/acme/platform", "[::1]/acme/platform"}, + "empty": {"", ""}, + "whitespace only": {" ", ""}, + "unparseable stays itself": {"Not A URL", "not a url"}, + "relative path is not a url": {"./a:b", "./a:b"}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + assert.Equal(t, tc.want, Normalize(tc.in)) + }) + } +} + +func TestSame(t *testing.T) { + tests := map[string]struct { + a, b string + want bool + }{ + "https and scp-like": {"https://github.com/acme/platform.git", "git@github.com:acme/platform", true}, + "scp-like and ssh url": {"git@github.com:acme/platform.git", "ssh://git@github.com/acme/platform", true}, + "case and suffix": { + "https://GITHUB.com/acme/Platform", + "https://github.com/acme/platform.git", + true, + }, + "different repository": {"https://github.com/acme/platform", "https://github.com/acme/other", false}, + "different host": {"https://github.com/acme/platform", "https://gitlab.com/acme/platform", false}, + "different non-default port": {"ssh://git@h:2222/acme/p", "ssh://git@h:2223/acme/p", false}, + "both empty": {"", "", false}, + "one empty": {"", "https://github.com/acme/platform", false}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + assert.Equal(t, tc.want, Same(tc.a, tc.b)) + assert.Equal(t, tc.want, Same(tc.b, tc.a), "Same must be symmetric") + }) + } +} diff --git a/internal/telemetry/apply_requests.go b/internal/telemetry/apply_requests.go new file mode 100644 index 00000000..1467e5d6 --- /dev/null +++ b/internal/telemetry/apply_requests.go @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: Apache-2.0 + +package telemetry + +import "go.opentelemetry.io/otel/metric" + +// ApplyRequestsTotal counts the apply requests GitOps Reverser decided, one per applier object per +// request, labelled by {applier_kind, reason, outcome}. +// +// applier_kind names the applier: "kustomization" (a Flux Kustomization). reason is the request's +// ApplyReason (ForeignCommit, Refusal, CatchUp). outcome is a frozen enum: +// +// sent — the applier was asked and the ask landed: a requestedAt stamp was patched, or, for +// a CatchUp, the applier's source already held the commit and nothing was owed. +// failed — every attempt failed and the request was given up on. An ApplyRequestFailed +// Event names the GitTargets it was for. +// superseded — a newer request for the same applier replaced this one before it was sent. The +// newer request carries the newer commit, so nothing is lost; a steady stream of +// these says requests are arriving faster than the applier is answering. +// +// It counts decisions, not attempts: a request that fails twice and then lands is one "sent". The +// retries are in the logs. It says nothing about whether the applier APPLIED: a request asks and +// never reports, and the applier's own status is the record of what it did. +// +// No GitTarget identity: one request is for one applier, which several targets can share, so a +// target label would have to invent an owner. The ApplyRequested and ApplyRequestFailed Events on +// each GitTarget carry that identity instead. +var ApplyRequestsTotal metric.Int64Counter diff --git a/internal/telemetry/exporter.go b/internal/telemetry/exporter.go index 3b502a1c..36f8b59b 100644 --- a/internal/telemetry/exporter.go +++ b/internal/telemetry/exporter.go @@ -596,6 +596,8 @@ func registerCounters() error { {"gitopsreverser_secret_encryptions_total", &SecretEncryptionsTotal}, {"gitopsreverser_watch_plan_passes_total", &WatchPlanPassesTotal}, {"gitopsreverser_watch_plan_triggers_total", &WatchPlanTriggersTotal}, + {"gitopsreverser_apply_requests_total", &ApplyRequestsTotal}, + {"gitopsreverser_receiver_deliveries_total", &ReceiverDeliveriesTotal}, } for _, s := range counters { v, err := otelMeter.Int64Counter(s.name) diff --git a/internal/telemetry/receiver.go b/internal/telemetry/receiver.go new file mode 100644 index 00000000..feb2c914 --- /dev/null +++ b/internal/telemetry/receiver.go @@ -0,0 +1,28 @@ +// SPDX-License-Identifier: Apache-2.0 + +package telemetry + +import "go.opentelemetry.io/otel/metric" + +// ReceiverDeliveriesTotal counts the deliveries the push receiver answered for a GitProvider that +// has spec.receiver, labelled by {type, outcome}. type is the receiver type (github, gitlab, +// gitea, generic-hmac). outcome is a frozen enum: +// +// notified — verified, named a branch, and a branch worker for it was told to look. +// unmatched — verified and named a branch, but no branch worker writes it. Still a 202. +// ignored — verified and deliberately not acted on: a ping, an event other than a push, a +// tag or other non-branch ref, or a generic-hmac body naming another repository. +// rejected — refused before anything in it was read: the signature or token was absent or +// wrong, or the body was over the size cap. +// invalid — verified but not parseable, or a required field was missing (a 400). +// +// It cannot say whether the push was somebody else's: the receiver only hands the branch to its +// worker, and the worker decides foreign versus own by reading the remote. A delivery is a hint, +// so "notified" means "looked", never "found a foreign commit". +// +// Requests that never reach a provider with a receiver (an unknown route, a wrong method, a +// missing token Secret) are not counted: there is no type to label them with, and the route and +// method are unbounded inputs from whoever can reach the listener. +// +// Registered in registerCounters (exporter.go) with every other counter. +var ReceiverDeliveriesTotal metric.Int64Counter diff --git a/internal/watch/cluster_context.go b/internal/watch/cluster_context.go index 72a52a09..6cd81e32 100644 --- a/internal/watch/cluster_context.go +++ b/internal/watch/cluster_context.go @@ -4,6 +4,7 @@ package watch import ( "context" + "errors" "fmt" "sort" "strings" @@ -695,6 +696,40 @@ func (m *Manager) clusterDynamicClient(ctx context.Context, clusterID string) (d return dc, nil } +// SourceDynamicClient returns the dynamic client for a source cluster, named by its +// ClusterProvider (GitTarget.SourceCluster()). It is the client the cluster's watches already run +// on, with the ClusterProvider's identity, so anything asked through it (the applier's apply +// requests) is asked as that identity and against that cluster, never the operator's own. +// +// It serves only a cluster some GitTarget has declared. clusterDynamicClient creates a context for +// any id on first use, and only forgetGitTargetCluster tears one down, when the last GitTarget that +// declared it goes. A context created for an id no GitTarget declared would therefore live for the +// process: a resolved credential and a client set nobody ever releases. An undeclared id is also +// the moment a caller has run ahead of the GitTarget controller (startup, or a target created a +// moment ago), and the error says so; its caller retries, by which time the Declare has landed. +// +// The empty id is the config plane, which is not a source cluster and is refused rather than +// silently served as one. +func (m *Manager) SourceDynamicClient(ctx context.Context, clusterID string) (dynamic.Interface, error) { + if clusterID == configPlaneClusterID { + return nil, errors.New("no source cluster named: the config plane is not a source cluster") + } + if !m.sourceClusterDeclared(clusterID) { + return nil, fmt.Errorf("source cluster %q is not declared by any GitTarget yet", clusterID) + } + return m.clusterDynamicClient(ctx, clusterID) +} + +// sourceClusterDeclared reports whether any GitTarget's Declare captured this cluster id. +func (m *Manager) sourceClusterDeclared(clusterID string) bool { + for _, id := range m.watchPlane().clusters { + if id == clusterID { + return true + } + } + return false +} + // discoveryRESTConfig returns the config a cluster's discovery client is built from: a COPY of // the cluster's config carrying a request timeout. // diff --git a/internal/watch/source_dynamic_client_test.go b/internal/watch/source_dynamic_client_test.go new file mode 100644 index 00000000..ebd5bba4 --- /dev/null +++ b/internal/watch/source_dynamic_client_test.go @@ -0,0 +1,59 @@ +// SPDX-License-Identifier: Apache-2.0 + +package watch + +import ( + "context" + "testing" + + "github.com/go-logr/logr" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "k8s.io/client-go/rest" +) + +// TestSourceDynamicClient_RefusesTheConfigPlane: the empty id is the operator's own cluster, and a +// caller asking for a source cluster by an empty name has a bug, not a cluster. +func TestSourceDynamicClient_RefusesTheConfigPlane(t *testing.T) { + m := &Manager{Log: logr.Discard()} + + _, err := m.SourceDynamicClient(context.Background(), configPlaneClusterID) + require.Error(t, err) +} + +// TestSourceDynamicClient_RefusesAnUndeclaredCluster: asking for a cluster no GitTarget declared +// fails, and creates no context, so nothing is left behind that only a GitTarget's deletion could +// ever tear down. +func TestSourceDynamicClient_RefusesAnUndeclaredCluster(t *testing.T) { + m := &Manager{ + Log: logr.Discard(), + SourceClusters: stubSourceClusterResolver{cfg: &rest.Config{Host: "https://prod.example"}}, + } + + _, err := m.SourceDynamicClient(context.Background(), "prod-eu-1") + require.Error(t, err) + assert.Contains(t, err.Error(), "not declared") + assert.Nil(t, m.clusterContextByID("prod-eu-1"), "an undeclared id creates no cluster context") +} + +// TestSourceDynamicClient_ServesADeclaredCluster: once a GitTarget declared the cluster, the client +// is the one its watches use, built once and cached. +func TestSourceDynamicClient_ServesADeclaredCluster(t *testing.T) { + m := &Manager{ + Log: logr.Discard(), + SourceClusters: stubSourceClusterResolver{cfg: &rest.Config{Host: "https://prod.example"}, version: "v1"}, + } + m.rememberGitTargetCluster(gd("checkout"), "prod-eu-1") + + first, err := m.SourceDynamicClient(context.Background(), "prod-eu-1") + require.NoError(t, err) + require.NotNil(t, first) + + second, err := m.SourceDynamicClient(context.Background(), "prod-eu-1") + require.NoError(t, err) + assert.Same(t, first, second, "the cluster's cached client is reused, not rebuilt per call") + + cc := m.clusterContextByID("prod-eu-1") + require.NotNil(t, cc) + assert.Same(t, first, cc.dynamicClient, "it is the client the cluster's watches run on") +} diff --git a/internal/webhook/receiver.go b/internal/webhook/receiver.go new file mode 100644 index 00000000..2b496382 --- /dev/null +++ b/internal/webhook/receiver.go @@ -0,0 +1,521 @@ +// SPDX-License-Identifier: Apache-2.0 + +package webhook + +import ( + "context" + "crypto/hmac" + "crypto/sha256" + "crypto/subtle" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "io" + "net/http" + "strings" + + "github.com/go-logr/logr" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + "k8s.io/apimachinery/pkg/types" + "k8s.io/apimachinery/pkg/util/validation" + "sigs.k8s.io/controller-runtime/pkg/client" + logf "sigs.k8s.io/controller-runtime/pkg/log" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/repourl" + "github.com/ConfigButler/gitops-reverser/internal/telemetry" +) + +// ReceiverPathPrefix is the path prefix the push receiver serves. A route is +// ReceiverPathPrefix + "/", as v1alpha3.ReceiverPath builds it. +const ReceiverPathPrefix = "/git-push/" + +// DefaultReceiverMaxRequestBodyBytes caps a delivery's body. It is GitHub's own cap on a webhook +// payload (25 MiB), the largest of the hosts the receiver speaks to, so no genuine push event is +// refused for its size. A push event is a few kilobytes in practice; the cap exists so an +// unauthenticated client cannot make the receiver buffer an unbounded body before the signature +// over it can be checked. +const DefaultReceiverMaxRequestBodyBytes = int64(25 * 1024 * 1024) + +// Headers each receiver type is signed or classified by. The names are the hosts' own; the +// generic-hmac header is the one Flux's generic-hmac Receiver checks, so a relay written for Flux +// can post here unchanged. +const ( + headerGitHubSignature = "X-Hub-Signature-256" + headerGitHubEvent = "X-Github-Event" // canonical form of GitHub's X-GitHub-Event + headerGitLabToken = "X-Gitlab-Token" //nolint:gosec // a header name, not a credential + headerGitLabEvent = "X-Gitlab-Event" + headerGiteaSignature = "X-Gitea-Signature" + headerGiteaEvent = "X-Gitea-Event" + headerGenericHMAC = "X-Signature" + + sha256SignaturePrefix = "sha256=" + branchRefPrefix = "refs/heads/" +) + +// Delivery outcomes, the outcome label of gitopsreverser_receiver_deliveries_total. See +// telemetry.ReceiverDeliveriesTotal for what each one means. +const ( + deliveryNotified = "notified" + deliveryUnmatched = "unmatched" + deliveryIgnored = "ignored" + deliveryRejected = "rejected" + deliveryInvalid = "invalid" +) + +// PushNotifier hands a verified delivery's branch to the branch worker that owns it. It reports +// whether such a worker exists. git.WorkerManager implements it. +// +// after is the commit the delivery named. It is passed for logging only: a delivery is a hint, +// and the worker reads where the branch is from the remote itself rather than trusting it. +type PushNotifier interface { + NotifyPush(providerNamespace, providerName, branch, after string) bool +} + +// ReceiverHandlerConfig configures the push receiver. +type ReceiverHandlerConfig struct { + // Reader looks up the GitProvider a route names and the Secret holding its token. In the + // manager this is the manager client: GitProviders come from its cache, and Secrets are + // excluded from that cache (Cache.DisableFor), so each delivery reads the token Secret + // directly by name. A rotated token therefore takes effect on the next delivery, and the + // receiver adds no Secret informer. See docs/rbac.md. + Reader client.Reader + // Notifier receives each verified push to a branch. Required. + Notifier PushNotifier + // MaxRequestBodyBytes caps a delivery's body. Zero means DefaultReceiverMaxRequestBodyBytes. + MaxRequestBodyBytes int64 +} + +// ReceiverHandler serves POST /git-push//: the endpoint a Git host's push webhook +// points at, so GitOps Reverser learns quickly that somebody pushed to a branch. +// +// A delivery is a hint, never a fact. The handler authenticates it, extracts the branch, and asks +// the branch worker to look; the worker then reads the remote's advertisement and decides for +// itself whether the branch gained a foreign commit. Nothing the body says beyond the branch name +// is acted on, so a late, duplicated or replayed delivery costs one ref advertisement and can never +// make the worker believe something the remote does not say. +// +// Authentication comes first. The signature (or GitLab's token) is checked over the raw bytes +// before the body is parsed for anything, in constant time, so an unsigned request cannot make a +// target do work or probe which branches are tracked. +type ReceiverHandler struct { + config ReceiverHandlerConfig +} + +// NewReceiverHandler builds the push receiver handler. +func NewReceiverHandler(config ReceiverHandlerConfig) (*ReceiverHandler, error) { + if config.Reader == nil { + return nil, errors.New("receiver handler requires a Reader") + } + if config.Notifier == nil { + return nil, errors.New("receiver handler requires a Notifier") + } + if config.MaxRequestBodyBytes <= 0 { + config.MaxRequestBodyBytes = DefaultReceiverMaxRequestBodyBytes + } + return &ReceiverHandler{config: config}, nil +} + +// receiverResponse is the body of every 202. matched is 1 when a branch worker was told to look +// and 0 otherwise; it is a number rather than a boolean so a relay fanning one event out to many +// receivers can sum the answers. +type receiverResponse struct { + Matched int `json:"matched"` +} + +// pushDelivery is what a verified delivery is acted on for. +type pushDelivery struct { + branch string + after string +} + +// errDeliveryIgnored marks a verified delivery that is deliberately not acted on (a tag, another +// repository). It answers 202, not 400: none of those is the sender's mistake. +var errDeliveryIgnored = errors.New("delivery ignored") + +// ServeHTTP implements http.Handler for push deliveries. +// +// The status codes are a contract with Git hosts, which disable a hook that keeps failing: every +// verified delivery answers 202, including one that names a branch nothing writes, because that is +// not the host's fault. Only an unauthenticated request (401), an unparseable verified one (400), +// a route that does not serve (404) and a receiver that cannot verify anything yet (503) fail. +func (h *ReceiverHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { + ctx := r.Context() + log := logf.Log.WithName("push-receiver") + + if r.Method != http.MethodPost { + w.Header().Set("Allow", http.MethodPost) + http.Error(w, "method not allowed", http.StatusMethodNotAllowed) + return + } + + key, ok := receiverRoute(r.URL.Path) + if !ok { + notFound(w) + return + } + log = log.WithValues("gitProvider", key.String()) + + provider, ok := h.lookupProvider(ctx, log, w, key) + if !ok { + return + } + receiverType := provider.Spec.Receiver.Type + log = log.WithValues("type", receiverType) + + token, ok := h.lookupToken(ctx, log, w, provider) + if !ok { + return + } + + body, ok := h.authenticate(ctx, log, w, r, receiverType, token) + if !ok { + return + } + h.act(ctx, log, w, r.Header, key, provider, body) +} + +// authenticate reads the body under the size cap and verifies it, answering the request itself when +// it fails. Nothing in the body is parsed here: a request is authenticated before it is read for +// anything. +func (h *ReceiverHandler) authenticate( + ctx context.Context, + log logr.Logger, + w http.ResponseWriter, + r *http.Request, + receiverType v1alpha3.ReceiverType, + token []byte, +) ([]byte, bool) { + body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, h.config.MaxRequestBodyBytes)) + if err != nil { + var tooLarge *http.MaxBytesError + if errors.As(err, &tooLarge) { + log.Info("Refused a push delivery over the size cap", "maxBytes", h.config.MaxRequestBodyBytes) + http.Error(w, "request body too large", http.StatusRequestEntityTooLarge) + h.record(ctx, receiverType, deliveryRejected) + return nil, false + } + log.V(1).Info("Could not read a push delivery", "error", err.Error()) + http.Error(w, "could not read request body", http.StatusBadRequest) + return nil, false + } + + if !verifyDelivery(receiverType, r.Header, body, token) { + // Info, not V(1): a run of these is a misconfigured hook (a token that does not match) or + // somebody probing the endpoint, and either is worth seeing at the default level. Neither + // the token nor the body is logged. + log.Info("Rejected a push delivery: signature absent or wrong") + http.Error(w, "signature absent or wrong", http.StatusUnauthorized) + h.record(ctx, receiverType, deliveryRejected) + return nil, false + } + return body, true +} + +// act handles a verified delivery: it decides whether the delivery is a push to a branch and, if +// so, hands the branch to its worker. Every path answers 202 except a body that cannot be read. +func (h *ReceiverHandler) act( + ctx context.Context, + log logr.Logger, + w http.ResponseWriter, + header http.Header, + key types.NamespacedName, + provider *v1alpha3.GitProvider, + body []byte, +) { + receiverType := provider.Spec.Receiver.Type + if event, isPush := pushEvent(receiverType, header); !isPush { + // A host's ping (GitHub sends one when the hook is created) and any other event the hook + // was subscribed to are answered 202, so the host shows the hook as healthy. + log.V(1).Info("Ignored a verified delivery that is not a push", "event", event) + h.accept(w, false) + h.record(ctx, receiverType, deliveryIgnored) + return + } + + delivery, err := parseDelivery(receiverType, body, provider.Spec.URL) + switch { + case errors.Is(err, errDeliveryIgnored): + log.V(1).Info("Ignored a verified push delivery", "reason", err.Error()) + h.accept(w, false) + h.record(ctx, receiverType, deliveryIgnored) + return + case err != nil: + log.Info("Refused a verified push delivery that cannot be read", "error", err.Error()) + http.Error(w, err.Error(), http.StatusBadRequest) + h.record(ctx, receiverType, deliveryInvalid) + return + } + + matched := h.config.Notifier.NotifyPush(key.Namespace, key.Name, delivery.branch, delivery.after) + outcome := deliveryUnmatched + if matched { + outcome = deliveryNotified + } + log.V(1).Info("Push delivery handled", "branch", delivery.branch, "after", delivery.after, + "outcome", outcome) + h.accept(w, matched) + h.record(ctx, receiverType, outcome) +} + +// receiverRoute parses /git-push//. Anything else under the prefix, including a +// segment that cannot be a namespace or an object name, is no route: answering 404 for it without +// an API read keeps a scanner from turning into apiserver traffic. +func receiverRoute(path string) (types.NamespacedName, bool) { + rest, ok := strings.CutPrefix(path, ReceiverPathPrefix) + if !ok { + return types.NamespacedName{}, false + } + namespace, name, ok := strings.Cut(rest, "/") + if !ok || strings.Contains(name, "/") { + return types.NamespacedName{}, false + } + if len(validation.IsDNS1123Label(namespace)) > 0 || len(validation.IsDNS1123Subdomain(name)) > 0 { + return types.NamespacedName{}, false + } + return types.NamespacedName{Namespace: namespace, Name: name}, true +} + +// lookupProvider resolves the GitProvider a route names. A missing provider and one without +// spec.receiver answer the same 404, so the endpoint does not reveal which GitProviders exist. +func (h *ReceiverHandler) lookupProvider( + ctx context.Context, + log logr.Logger, + w http.ResponseWriter, + key types.NamespacedName, +) (*v1alpha3.GitProvider, bool) { + var provider v1alpha3.GitProvider + if err := h.config.Reader.Get(ctx, key, &provider); err != nil { + if apierrors.IsNotFound(err) { + log.V(1).Info("Push delivery for a GitProvider that does not exist") + notFound(w) + return nil, false + } + log.Error(err, "Could not read the GitProvider a push delivery names") + http.Error(w, "receiver not ready", http.StatusServiceUnavailable) + return nil, false + } + if provider.Spec.Receiver == nil { + log.V(1).Info("Push delivery for a GitProvider without spec.receiver") + notFound(w) + return nil, false + } + return &provider, true +} + +// lookupToken reads the receiver token from the Secret spec.receiver.secretRef names. A missing +// Secret or key is the receiver not being ready, never the sender's fault, so it answers 503 and +// not 401: a 401 would tell whoever set up the hook that their secret is wrong when it is ours +// that is absent, and a host retries a 5xx where it may give up on a 4xx. +func (h *ReceiverHandler) lookupToken( + ctx context.Context, + log logr.Logger, + w http.ResponseWriter, + provider *v1alpha3.GitProvider, +) ([]byte, bool) { + secretName := provider.Spec.Receiver.SecretRef.Name + var secret corev1.Secret + err := h.config.Reader.Get(ctx, types.NamespacedName{Namespace: provider.Namespace, Name: secretName}, &secret) + if err != nil { + if apierrors.IsNotFound(err) { + log.Info("Push receiver token Secret not found", "secret", secretName) + } else { + log.Error(err, "Could not read the push receiver token Secret", "secret", secretName) + } + http.Error(w, "receiver not ready", http.StatusServiceUnavailable) + return nil, false + } + token := secret.Data[v1alpha3.ReceiverTokenKey] + if len(token) == 0 { + log.Info("Push receiver token Secret has no token", "secret", secretName, "key", v1alpha3.ReceiverTokenKey) + http.Error(w, "receiver not ready", http.StatusServiceUnavailable) + return nil, false + } + return token, true +} + +// verifyDelivery authenticates a delivery over its raw bytes, the way its host signs it. Every +// comparison is constant time. An unknown type verifies nothing: the CRD's enum keeps one from +// existing, and failing closed keeps a future type from being served unauthenticated. +func verifyDelivery(receiverType v1alpha3.ReceiverType, header http.Header, body, token []byte) bool { + switch receiverType { + case v1alpha3.ReceiverTypeGitHub: + return verifyPrefixedHMAC(header.Get(headerGitHubSignature), body, token) + case v1alpha3.ReceiverTypeGitea: + return verifyHexHMAC(header.Get(headerGiteaSignature), body, token) + case v1alpha3.ReceiverTypeGenericHMAC: + return verifyPrefixedHMAC(header.Get(headerGenericHMAC), body, token) + case v1alpha3.ReceiverTypeGitLab: + return verifyToken(header.Get(headerGitLabToken), token) + default: + return false + } +} + +// verifyPrefixedHMAC checks a "sha256=" signature header. +func verifyPrefixedHMAC(signature string, body, token []byte) bool { + hexDigest, ok := strings.CutPrefix(signature, sha256SignaturePrefix) + if !ok { + return false + } + return verifyHexHMAC(hexDigest, body, token) +} + +// verifyHexHMAC checks a bare hex HMAC-SHA256 of body keyed by token. +func verifyHexHMAC(hexDigest string, body, token []byte) bool { + if hexDigest == "" { + return false + } + got, err := hex.DecodeString(hexDigest) + if err != nil { + return false + } + mac := hmac.New(sha256.New, token) + mac.Write(body) + return hmac.Equal(got, mac.Sum(nil)) +} + +// verifyToken compares a plain token header with the configured token. Both sides are hashed +// first so the comparison takes the same time whatever the presented token's length: +// subtle.ConstantTimeCompare alone returns early on a length mismatch, which leaks the length. +func verifyToken(presented string, token []byte) bool { + if presented == "" { + return false + } + got := sha256.Sum256([]byte(presented)) + want := sha256.Sum256(token) + return subtle.ConstantTimeCompare(got[:], want[:]) == 1 +} + +// pushEvent reports the delivery's event name and whether it is a push. generic-hmac carries no +// event header: a relay posts only pushes, so every verified generic-hmac delivery is one. +func pushEvent(receiverType v1alpha3.ReceiverType, header http.Header) (string, bool) { + switch receiverType { + case v1alpha3.ReceiverTypeGitHub: + event := header.Get(headerGitHubEvent) + return event, event == "push" + case v1alpha3.ReceiverTypeGitLab: + event := header.Get(headerGitLabEvent) + return event, event == "Push Hook" + case v1alpha3.ReceiverTypeGitea: + event := header.Get(headerGiteaEvent) + return event, event == "push" + case v1alpha3.ReceiverTypeGenericHMAC: + return "push", true + default: + return "", false + } +} + +// hostPushPayload is the part of a GitHub, GitLab or Gitea push event the receiver reads. The +// three hosts agree on these two fields, which is why one shape serves all of them. +type hostPushPayload struct { + Ref string `json:"ref"` + After string `json:"after"` +} + +// genericPushPayload is a generic-hmac relay's body. +type genericPushPayload struct { + Repository string `json:"repository"` + Branch string `json:"branch"` + After string `json:"after"` +} + +// parseDelivery extracts the branch a verified push names. It returns errDeliveryIgnored (wrapped, +// with the reason) for a push that is deliberately not acted on, and any other error for a body +// that cannot be read. +// +// after is optional in every shape. It is only ever logged, so refusing a delivery for lacking it +// would turn a field nothing depends on into a reason for a host to disable the hook. +// +// The repository check is generic-hmac's alone. A host's own hook is configured on one repository +// (or an organization), and a delivery from a sibling repository on an organization hook costs one +// ref advertisement for a branch of the same name; comparing the host's repository URLs with +// spec.url instead would silently drop every delivery for a provider that reaches the host by +// another name (an SSH host alias, ssh.github.com:443), which is the worse failure. +func parseDelivery(receiverType v1alpha3.ReceiverType, body []byte, providerURL string) (pushDelivery, error) { + if receiverType == v1alpha3.ReceiverTypeGenericHMAC { + return parseGenericDelivery(body, providerURL) + } + var payload hostPushPayload + if err := json.Unmarshal(body, &payload); err != nil { + return pushDelivery{}, fmt.Errorf("push event is not valid JSON: %w", err) + } + if payload.Ref == "" { + return pushDelivery{}, errors.New("push event has no ref") + } + branch, err := branchFromRef(payload.Ref) + if err != nil { + return pushDelivery{}, err + } + return pushDelivery{branch: branch, after: payload.After}, nil +} + +// parseGenericDelivery reads a generic-hmac body. Its branch may be a bare name or a full ref; a +// repository other than spec.url is ignored, not refused, because a relay fanning one host's +// events out to every receiver is doing exactly what it is for. +func parseGenericDelivery(body []byte, providerURL string) (pushDelivery, error) { + var payload genericPushPayload + if err := json.Unmarshal(body, &payload); err != nil { + return pushDelivery{}, fmt.Errorf("body is not valid JSON: %w", err) + } + if payload.Branch == "" { + return pushDelivery{}, errors.New("body has no branch") + } + branch := payload.Branch + if strings.HasPrefix(branch, "refs/") { + var err error + if branch, err = branchFromRef(branch); err != nil { + return pushDelivery{}, err + } + } + if payload.Repository != "" && !repourl.Same(payload.Repository, providerURL) { + return pushDelivery{}, fmt.Errorf("%w: names another repository", errDeliveryIgnored) + } + return pushDelivery{branch: branch, after: payload.After}, nil +} + +// branchFromRef returns the branch a full ref names. A ref outside refs/heads/ (a tag, a pull +// request ref) is ignored: no branch worker writes it. +func branchFromRef(ref string) (string, error) { + branch, ok := strings.CutPrefix(ref, branchRefPrefix) + if !ok { + return "", fmt.Errorf("%w: %s is not a branch", errDeliveryIgnored, ref) + } + if branch == "" { + return "", fmt.Errorf("ref %q names no branch", ref) + } + return branch, nil +} + +// notFound answers a route that does not serve. One body for every reason, so the answer does not +// say whether the GitProvider is missing or merely has no receiver. +func notFound(w http.ResponseWriter) { + http.Error(w, "no push receiver at this path", http.StatusNotFound) +} + +// accept answers a verified delivery. +func (h *ReceiverHandler) accept(w http.ResponseWriter, matched bool) { + resp := receiverResponse{} + if matched { + resp.Matched = 1 + } + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusAccepted) + _ = json.NewEncoder(w).Encode(resp) +} + +// record counts one delivery for a provider with a receiver. +func (h *ReceiverHandler) record(ctx context.Context, receiverType v1alpha3.ReceiverType, outcome string) { + if telemetry.ReceiverDeliveriesTotal == nil { + return + } + telemetry.ReceiverDeliveriesTotal.Add(ctx, 1, metric.WithAttributes( + attribute.String("type", string(receiverType)), + attribute.String("outcome", outcome), + )) +} diff --git a/internal/webhook/receiver_test.go b/internal/webhook/receiver_test.go new file mode 100644 index 00000000..f81d7fa4 --- /dev/null +++ b/internal/webhook/receiver_test.go @@ -0,0 +1,483 @@ +// SPDX-License-Identifier: Apache-2.0 + +package webhook + +import ( + "context" + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + + meta "github.com/fluxcd/pkg/apis/meta" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + clientgoscheme "k8s.io/client-go/kubernetes/scheme" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/telemetry" +) + +const ( + receiverTestNamespace = "team-a" + receiverTestToken = "s3cr3t-token" + receiverTestRepoURL = "https://github.com/acme/platform.git" + receiverTestCommit = "4f2c1ab9e0d1c2b3a4f5e6d7c8b9a0f1e2d3c4b5" +) + +// fakePushNotifier records every NotifyPush call and reports a worker for the branches in workers. +type fakePushNotifier struct { + workers map[string]bool + calls []fakePushCall +} + +type fakePushCall struct { + namespace, name, branch, after string +} + +func (f *fakePushNotifier) NotifyPush(providerNamespace, providerName, branch, after string) bool { + f.calls = append(f.calls, fakePushCall{providerNamespace, providerName, branch, after}) + return f.workers[branch] +} + +func receiverTestProvider(name string, receiver *v1alpha3.GitProviderReceiver) *v1alpha3.GitProvider { + return &v1alpha3.GitProvider{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: receiverTestNamespace}, + Spec: v1alpha3.GitProviderSpec{ + URL: receiverTestRepoURL, + AllowedBranches: []string{"*"}, + Receiver: receiver, + }, + } +} + +func receiverOf(receiverType v1alpha3.ReceiverType, secret string) *v1alpha3.GitProviderReceiver { + return &v1alpha3.GitProviderReceiver{Type: receiverType, SecretRef: meta.LocalObjectReference{Name: secret}} +} + +func receiverTestSecret(name string, data map[string][]byte) *corev1.Secret { + return &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: receiverTestNamespace}, + Data: data, + } +} + +// receiverTestObjects is one provider per receiver type sharing one token Secret, plus the +// providers that must not serve or cannot verify yet. +func receiverTestObjects() []client.Object { + return []client.Object{ + receiverTestSecret("receiver-token", map[string][]byte{v1alpha3.ReceiverTokenKey: []byte(receiverTestToken)}), + receiverTestSecret("wrong-key", map[string][]byte{"password": []byte(receiverTestToken)}), + receiverTestProvider("gh", receiverOf(v1alpha3.ReceiverTypeGitHub, "receiver-token")), + receiverTestProvider("gl", receiverOf(v1alpha3.ReceiverTypeGitLab, "receiver-token")), + receiverTestProvider("gt", receiverOf(v1alpha3.ReceiverTypeGitea, "receiver-token")), + receiverTestProvider("gen", receiverOf(v1alpha3.ReceiverTypeGenericHMAC, "receiver-token")), + receiverTestProvider("plain", nil), + receiverTestProvider("no-secret", receiverOf(v1alpha3.ReceiverTypeGitHub, "absent")), + receiverTestProvider("no-key", receiverOf(v1alpha3.ReceiverTypeGitHub, "wrong-key")), + } +} + +func newReceiverTestHandler( + t *testing.T, + notifier *fakePushNotifier, + maxBody int64, + funcs *interceptor.Funcs, +) *ReceiverHandler { + t.Helper() + scheme := runtime.NewScheme() + require.NoError(t, clientgoscheme.AddToScheme(scheme)) + require.NoError(t, v1alpha3.AddToScheme(scheme)) + builder := fake.NewClientBuilder().WithScheme(scheme).WithObjects(receiverTestObjects()...) + if funcs != nil { + builder = builder.WithInterceptorFuncs(*funcs) + } + h, err := NewReceiverHandler(ReceiverHandlerConfig{ + Reader: builder.Build(), + Notifier: notifier, + MaxRequestBodyBytes: maxBody, + }) + require.NoError(t, err) + return h +} + +func hmacHex(body, token string) string { + mac := hmac.New(sha256.New, []byte(token)) + mac.Write([]byte(body)) + return hex.EncodeToString(mac.Sum(nil)) +} + +func hostPushBody(ref string) string { + return `{"ref":"` + ref + `","before":"0000","after":"` + receiverTestCommit + `",` + + `"repository":{"clone_url":"https://github.com/acme/platform.git"}}` +} + +func genericBody(repository, branch string) string { + body, _ := json.Marshal(map[string]string{"repository": repository, "branch": branch, "after": receiverTestCommit}) + return string(body) +} + +func githubHeaders(event, body, token string) map[string]string { + return map[string]string{headerGitHubEvent: event, headerGitHubSignature: "sha256=" + hmacHex(body, token)} +} + +func giteaHeaders(event, body, token string) map[string]string { + return map[string]string{headerGiteaEvent: event, headerGiteaSignature: hmacHex(body, token)} +} + +func gitlabHeaders(event, token string) map[string]string { + return map[string]string{headerGitLabEvent: event, headerGitLabToken: token} +} + +func genericHeaders(body, token string) map[string]string { + return map[string]string{headerGenericHMAC: "sha256=" + hmacHex(body, token)} +} + +type receiverCase struct { + method string // POST when empty + path string + headers map[string]string + body string + + wantStatus int + // wantMatched is the 202 body's matched value; ignored for any other status. + wantMatched int + // wantCall is the NotifyPush the delivery must make; nil asserts none was made. + wantCall *fakePushCall + // wantType/wantOutcome is the one delivery the metric must count; empty asserts none. + wantType string + wantOutcome string +} + +func receiverCases() map[string]receiverCase { + mainPush := hostPushBody("refs/heads/main") + otherPush := hostPushBody("refs/heads/feature/x") + tagPush := hostPushBody("refs/tags/v1.0.0") + ghCall := &fakePushCall{receiverTestNamespace, "gh", "main", receiverTestCommit} + genMain := genericBody("git@github.com:acme/platform", "main") + + return map[string]receiverCase{ + // github + "github push to a worked branch is notified": { + path: "/git-push/team-a/gh", headers: githubHeaders("push", mainPush, receiverTestToken), body: mainPush, + wantStatus: http.StatusAccepted, wantMatched: 1, wantCall: ghCall, + wantType: "github", wantOutcome: deliveryNotified, + }, + "github push to a branch nothing writes is still a success": { + path: "/git-push/team-a/gh", headers: githubHeaders("push", otherPush, receiverTestToken), body: otherPush, + wantStatus: http.StatusAccepted, wantMatched: 0, + wantCall: &fakePushCall{receiverTestNamespace, "gh", "feature/x", receiverTestCommit}, + wantType: "github", wantOutcome: deliveryUnmatched, + }, + "github wrong signature is rejected": { + path: "/git-push/team-a/gh", headers: githubHeaders("push", mainPush, "not-the-token"), body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + }, + "github missing signature is rejected": { + path: "/git-push/team-a/gh", headers: map[string]string{headerGitHubEvent: "push"}, body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + }, + "github signature without the sha256 prefix is rejected": { + path: "/git-push/team-a/gh", + headers: map[string]string{ + headerGitHubEvent: "push", headerGitHubSignature: hmacHex(mainPush, receiverTestToken), + }, + body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + }, + "github signature that is not hex is rejected": { + path: "/git-push/team-a/gh", + headers: map[string]string{ + headerGitHubEvent: "push", headerGitHubSignature: "sha256=zz", + }, + body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + }, + "github ping is accepted and does nothing": { + path: "/git-push/team-a/gh", headers: githubHeaders("ping", `{"zen":"hi"}`, receiverTestToken), + body: `{"zen":"hi"}`, + wantStatus: http.StatusAccepted, wantType: "github", wantOutcome: deliveryIgnored, + }, + "github non-push event is ignored": { + path: "/git-push/team-a/gh", headers: githubHeaders("issues", `{}`, receiverTestToken), body: `{}`, + wantStatus: http.StatusAccepted, wantType: "github", wantOutcome: deliveryIgnored, + }, + "github tag push is ignored": { + path: "/git-push/team-a/gh", headers: githubHeaders("push", tagPush, receiverTestToken), body: tagPush, + wantStatus: http.StatusAccepted, wantType: "github", wantOutcome: deliveryIgnored, + }, + "github verified body that is not JSON is invalid": { + path: "/git-push/team-a/gh", headers: githubHeaders("push", "not json", receiverTestToken), + body: "not json", + wantStatus: http.StatusBadRequest, wantType: "github", wantOutcome: deliveryInvalid, + }, + "github verified push without a ref is invalid": { + path: "/git-push/team-a/gh", headers: githubHeaders("push", `{"after":"abc"}`, receiverTestToken), + body: `{"after":"abc"}`, + wantStatus: http.StatusBadRequest, wantType: "github", wantOutcome: deliveryInvalid, + }, + "github ref naming no branch is invalid": { + path: "/git-push/team-a/gh", headers: githubHeaders("push", `{"ref":"refs/heads/"}`, receiverTestToken), + body: `{"ref":"refs/heads/"}`, + wantStatus: http.StatusBadRequest, wantType: "github", wantOutcome: deliveryInvalid, + }, + + // gitlab + "gitlab push with the token is notified": { + path: "/git-push/team-a/gl", headers: gitlabHeaders("Push Hook", receiverTestToken), body: mainPush, + wantStatus: http.StatusAccepted, wantMatched: 1, + wantCall: &fakePushCall{receiverTestNamespace, "gl", "main", receiverTestCommit}, + wantType: "gitlab", wantOutcome: deliveryNotified, + }, + "gitlab wrong token is rejected": { + path: "/git-push/team-a/gl", headers: gitlabHeaders("Push Hook", "nope"), body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "gitlab", wantOutcome: deliveryRejected, + }, + "gitlab missing token is rejected": { + path: "/git-push/team-a/gl", headers: map[string]string{headerGitLabEvent: "Push Hook"}, body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "gitlab", wantOutcome: deliveryRejected, + }, + "gitlab tag push hook is ignored": { + path: "/git-push/team-a/gl", headers: gitlabHeaders("Tag Push Hook", receiverTestToken), body: tagPush, + wantStatus: http.StatusAccepted, wantType: "gitlab", wantOutcome: deliveryIgnored, + }, + + // gitea + "gitea push is notified": { + path: "/git-push/team-a/gt", headers: giteaHeaders("push", mainPush, receiverTestToken), body: mainPush, + wantStatus: http.StatusAccepted, wantMatched: 1, + wantCall: &fakePushCall{receiverTestNamespace, "gt", "main", receiverTestCommit}, + wantType: "gitea", wantOutcome: deliveryNotified, + }, + "gitea wrong signature is rejected": { + path: "/git-push/team-a/gt", headers: giteaHeaders("push", mainPush, "nope"), body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "gitea", wantOutcome: deliveryRejected, + }, + "gitea github-style prefixed signature is rejected": { + path: "/git-push/team-a/gt", + headers: map[string]string{ + headerGiteaEvent: "push", headerGiteaSignature: "sha256=" + hmacHex(mainPush, receiverTestToken), + }, + body: mainPush, + wantStatus: http.StatusUnauthorized, wantType: "gitea", wantOutcome: deliveryRejected, + }, + "gitea non-push event is ignored": { + path: "/git-push/team-a/gt", headers: giteaHeaders("create", `{}`, receiverTestToken), body: `{}`, + wantStatus: http.StatusAccepted, wantType: "gitea", wantOutcome: deliveryIgnored, + }, + + // generic-hmac + "generic-hmac push for the same repository spelled differently is notified": { + path: "/git-push/team-a/gen", headers: genericHeaders(genMain, receiverTestToken), body: genMain, + wantStatus: http.StatusAccepted, wantMatched: 1, + wantCall: &fakePushCall{receiverTestNamespace, "gen", "main", receiverTestCommit}, + wantType: "generic-hmac", wantOutcome: deliveryNotified, + }, + "generic-hmac full ref is read as its branch": { + path: "/git-push/team-a/gen", + headers: genericHeaders(genericBody("", "refs/heads/main"), receiverTestToken), + body: genericBody("", "refs/heads/main"), + wantStatus: http.StatusAccepted, wantMatched: 1, + wantCall: &fakePushCall{receiverTestNamespace, "gen", "main", receiverTestCommit}, + wantType: "generic-hmac", wantOutcome: deliveryNotified, + }, + "generic-hmac another repository is ignored, not refused": { + path: "/git-push/team-a/gen", + headers: genericHeaders(genericBody("https://github.com/acme/other", "main"), receiverTestToken), + body: genericBody("https://github.com/acme/other", "main"), + wantStatus: http.StatusAccepted, wantType: "generic-hmac", wantOutcome: deliveryIgnored, + }, + "generic-hmac tag ref is ignored": { + path: "/git-push/team-a/gen", + headers: genericHeaders(genericBody("", "refs/tags/v1"), receiverTestToken), + body: genericBody("", "refs/tags/v1"), + wantStatus: http.StatusAccepted, wantType: "generic-hmac", wantOutcome: deliveryIgnored, + }, + "generic-hmac wrong signature is rejected": { + path: "/git-push/team-a/gen", headers: genericHeaders(genMain, "nope"), body: genMain, + wantStatus: http.StatusUnauthorized, wantType: "generic-hmac", wantOutcome: deliveryRejected, + }, + "generic-hmac without a branch is invalid": { + path: "/git-push/team-a/gen", + headers: genericHeaders(`{"repository":"x"}`, receiverTestToken), + body: `{"repository":"x"}`, + wantStatus: http.StatusBadRequest, wantType: "generic-hmac", wantOutcome: deliveryInvalid, + }, + + // routes that do not serve, and a receiver that cannot verify yet + "unknown provider is 404": { + path: "/git-push/team-a/missing", + headers: githubHeaders("push", mainPush, receiverTestToken), + body: mainPush, + wantStatus: http.StatusNotFound, + }, + "provider without spec.receiver is 404": { + path: "/git-push/team-a/plain", headers: githubHeaders("push", mainPush, receiverTestToken), body: mainPush, + wantStatus: http.StatusNotFound, + }, + "route without a name is 404": { + path: "/git-push/team-a", body: mainPush, wantStatus: http.StatusNotFound, + }, + "route with an extra segment is 404": { + path: "/git-push/team-a/gh/extra", body: mainPush, wantStatus: http.StatusNotFound, + }, + "route with an empty name is 404": { + path: "/git-push/team-a/", body: mainPush, wantStatus: http.StatusNotFound, + }, + "route with an invalid namespace is 404": { + path: "/git-push/Team_A/gh", body: mainPush, wantStatus: http.StatusNotFound, + }, + "path outside the prefix is 404": { + path: "/other/team-a/gh", body: mainPush, wantStatus: http.StatusNotFound, + }, + "missing token Secret is 503, not 401": { + path: "/git-push/team-a/no-secret", headers: githubHeaders("push", mainPush, receiverTestToken), + body: mainPush, wantStatus: http.StatusServiceUnavailable, + }, + "token Secret without the token key is 503": { + path: "/git-push/team-a/no-key", headers: githubHeaders("push", mainPush, receiverTestToken), + body: mainPush, wantStatus: http.StatusServiceUnavailable, + }, + "GET is not allowed": { + method: http.MethodGet, path: "/git-push/team-a/gh", wantStatus: http.StatusMethodNotAllowed, + }, + } +} + +func TestReceiverHandler_ServeHTTP(t *testing.T) { + for name, tc := range receiverCases() { + t.Run(name, func(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + notifier := &fakePushNotifier{workers: map[string]bool{"main": true}} + h := newReceiverTestHandler(t, notifier, 0, nil) + + method := tc.method + if method == "" { + method = http.MethodPost + } + req := httptest.NewRequest(method, tc.path, strings.NewReader(tc.body)) + for k, v := range tc.headers { + req.Header.Set(k, v) + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + + require.Equal(t, tc.wantStatus, rec.Code, "body: %s", rec.Body.String()) + if tc.wantStatus == http.StatusAccepted { + var resp receiverResponse + require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &resp)) + assert.Equal(t, tc.wantMatched, resp.Matched) + assert.Equal(t, "application/json", rec.Header().Get("Content-Type")) + } + + if tc.wantCall == nil { + assert.Empty(t, notifier.calls, "a delivery that is not acted on must not reach a worker") + } else { + assert.Equal(t, []fakePushCall{*tc.wantCall}, notifier.calls) + } + + total, _ := telemetry.CollectInt64Sum(reader, "gitopsreverser_receiver_deliveries_total", nil) + if tc.wantOutcome == "" { + assert.Zero(t, total, "a request that reaches no receiver is not a delivery") + return + } + assert.Equal(t, int64(1), total, "exactly one delivery is counted") + got, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_receiver_deliveries_total", + map[string]string{"type": tc.wantType, "outcome": tc.wantOutcome}) + require.True(t, ok, "no %s/%s delivery recorded", tc.wantType, tc.wantOutcome) + assert.Equal(t, int64(1), got) + }) + } +} + +// TestReceiverHandler_OversizedBodyIsRefusedBeforeVerification pins the cap: a body over it is +// refused whatever its signature says, because verifying it would mean buffering all of it. +func TestReceiverHandler_OversizedBodyIsRefusedBeforeVerification(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + notifier := &fakePushNotifier{workers: map[string]bool{"main": true}} + h := newReceiverTestHandler(t, notifier, 64, nil) + + body := hostPushBody("refs/heads/main") + require.Greater(t, len(body), 64) + req := httptest.NewRequest(http.MethodPost, "/git-push/team-a/gh", strings.NewReader(body)) + for k, v := range githubHeaders("push", body, receiverTestToken) { + req.Header.Set(k, v) + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + + assert.Equal(t, http.StatusRequestEntityTooLarge, rec.Code) + assert.Empty(t, notifier.calls) + got, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_receiver_deliveries_total", + map[string]string{"type": "github", "outcome": deliveryRejected}) + require.True(t, ok) + assert.Equal(t, int64(1), got) +} + +// TestReceiverHandler_UnreadableProviderIs503 separates "no such route" from "cannot tell yet": an +// API error reading the GitProvider must not answer 404, which a host may treat as a dead hook. +func TestReceiverHandler_UnreadableProviderIs503(t *testing.T) { + notifier := &fakePushNotifier{} + h := newReceiverTestHandler(t, notifier, 0, &interceptor.Funcs{ + Get: func(_ context.Context, _ client.WithWatch, _ client.ObjectKey, _ client.Object, _ ...client.GetOption) error { + return errors.New("apiserver unavailable") + }, + }) + + body := hostPushBody("refs/heads/main") + req := httptest.NewRequest(http.MethodPost, "/git-push/team-a/gh", strings.NewReader(body)) + for k, v := range githubHeaders("push", body, receiverTestToken) { + req.Header.Set(k, v) + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + + assert.Equal(t, http.StatusServiceUnavailable, rec.Code) + assert.Empty(t, notifier.calls) +} + +func TestNewReceiverHandler_RequiresReaderAndNotifier(t *testing.T) { + _, err := NewReceiverHandler(ReceiverHandlerConfig{Notifier: &fakePushNotifier{}}) + require.Error(t, err) + + scheme := runtime.NewScheme() + require.NoError(t, v1alpha3.AddToScheme(scheme)) + _, err = NewReceiverHandler(ReceiverHandlerConfig{Reader: fake.NewClientBuilder().WithScheme(scheme).Build()}) + require.Error(t, err) + + h, err := NewReceiverHandler(ReceiverHandlerConfig{ + Reader: fake.NewClientBuilder().WithScheme(scheme).Build(), + Notifier: &fakePushNotifier{}, + }) + require.NoError(t, err) + assert.Equal(t, DefaultReceiverMaxRequestBodyBytes, h.config.MaxRequestBodyBytes) +} + +func TestVerifyToken_ComparesWholeToken(t *testing.T) { + assert.True(t, verifyToken("abc", []byte("abc"))) + assert.False(t, verifyToken("ab", []byte("abc"))) + assert.False(t, verifyToken("abcd", []byte("abc"))) + assert.False(t, verifyToken("", []byte("abc"))) +} + +func TestVerifyDelivery_UnknownTypeFailsClosed(t *testing.T) { + body := []byte(`{}`) + header := http.Header{} + header.Set(headerGenericHMAC, "sha256="+hmacHex(string(body), receiverTestToken)) + assert.False(t, verifyDelivery("bitbucket", header, body, []byte(receiverTestToken))) +} diff --git a/test/e2e/flux_bi_directional_e2e_test.go b/test/e2e/flux_bi_directional_e2e_test.go index 0f8a0b83..4aa8b134 100644 --- a/test/e2e/flux_bi_directional_e2e_test.go +++ b/test/e2e/flux_bi_directional_e2e_test.go @@ -337,27 +337,21 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order run.consistentlyExpectRemoteCommitCount(revertBaselineCommitCount+2, biStableCountMediumWait) }) - // The mechanism behind GitTarget spec.onRefusal: PushEmptyCommit - // (docs/design/push-notification-and-reconcile-trigger.md, part 3 option 7). + // The Flux property GitOps Reverser's apply requests rest on + // (docs/design/push-notification-and-reconcile-trigger.md §3.6): **a new GitRepository revision + // is enough**, on its own, to make the Kustomization apply and revert live drift, even when the + // manifests it builds are byte-identical to the ones it applied last time. A request that finds + // the artifact behind the commit stamps only the GitRepository and relies on exactly this. // - // The wiring from a refused write to the commit is unit-tested, including that the commit - // lands with an empty tree diff. What only real Flux can answer is the claim the whole option - // rests on: **a new revision is enough**, on its own, to make the reconciler re-apply and - // revert live drift, even though the manifests it builds are byte-identical to the ones it - // applied last time. - // - // So this spec isolates exactly that claim and nothing else. It pushes the empty commit by - // hand rather than provoking a refusal, because combining the two would test the mechanism - // LESS precisely: a failure would not say which half broke. - // - // Two details carry the argument, and removing either makes the spec pass for the wrong - // reason: + // So this spec isolates that claim and nothing else, with a commit that changes no file pushed + // by hand. Two details carry the argument, and removing either makes the spec pass for the + // wrong reason: // // - The GitTarget is SUSPENDED first. Otherwise the operator publishes the drift to Git and // there is no drift left for Flux to correct, so the revert would prove nothing. // - Only the GitRepository is reconciled, never the Kustomization. The Kustomization is on // a 30m interval, so if it applies inside this spec it is because a new source revision - // woke it, which is the property a push webhook would supply in production. + // woke it. It("reverts live drift from an empty commit, because a new revision is enough", func() { By("suspending the GitTarget so the drift stays drift instead of being published") _, err := kubectlRunInNamespace(testNs, "patch", "gittarget", run.gitTargetName, @@ -423,6 +417,88 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order "a new revision must be enough to make Flux re-apply and revert the drift") }, 2*time.Minute, 2*time.Second).Should(Succeed()) }) + + // The race GitOps Reverser's apply trigger exists for, from + // docs/design/bi-directional-right.md §7: two API edits to one field a few seconds apart. With + // Flux woken by every push, the first commit is applied over the second edit and the branch + // ping-pongs. With the Kustomization named as the GitTarget's applier, Flux is asked to catch up + // only once the branch has been quiet, so the later value survives and Flux still ends on the + // tip, without anybody reconciling it by hand. + It("asks Flux to apply only on a quiet branch, so the later of two quick API edits survives", func() { + By("naming the live Kustomization as the GitTarget's applier") + _, err := kubectlRunInNamespace(testNs, "patch", "gittarget", run.gitTargetName, "--type=merge", "-p", + fmt.Sprintf(`{"spec":{"applier":{"kustomizationRef":{"namespace":"flux-system","name":%q},`+ + `"quietPeriod":"5s"}}}`, run.fluxLiveName)) + Expect(err).NotTo(HaveOccurred(), "failed to set spec.applier") + verifyResourceCondition("gittarget", run.gitTargetName, testNs, "ApplierConfigured", "True", "Succeeded", "") + run.waitForGitTargetReady() + + Expect(run.gitPull()).To(Succeed()) + baseline := run.waitForStableRemoteCommitCount(biStableCountMediumWait) + + By("editing the second order through the API, and again seconds after that edit is pushed") + _, err = kubectlRunInNamespace(testNs, "patch", iceCreamCRDName(crdGroupBiDirectional), + run.secondOrderName, "--type=merge", "-p", `{"spec":{"container":"Cone"}}`) + Expect(err).NotTo(HaveOccurred()) + run.waitForCommittedOrderSpec(run.secondOrderName, "Cone", "Strawberry", "WhippedCream") + _, err = kubectlRunInNamespace(testNs, "patch", iceCreamCRDName(crdGroupBiDirectional), + run.secondOrderName, "--type=merge", "-p", `{"spec":{"container":"WaffleBowl"}}`) + Expect(err).NotTo(HaveOccurred()) + run.waitForCommittedOrderSpec(run.secondOrderName, "WaffleBowl", "Strawberry", "WhippedCream") + head := run.gitHEAD() + + By("Flux catches up with those pushes on its own, once the branch is quiet") + run.waitForFluxKustomizationRevision(run.fluxLiveName, head) + + By("and the later value survives, with no commit after it") + Consistently(func(g Gomega) { + g.Expect(liveOrderContainer(g, run, run.secondOrderName)).To(Equal("WaffleBowl")) + g.Expect(run.gitPull()).To(Succeed()) + count, countErr := run.gitMainCommitCount() + g.Expect(countErr).NotTo(HaveOccurred()) + g.Expect(count).To(Equal(baseline+2), + "an apply of the first edit's commit over the second edit would show as more commits") + }, 30*time.Second, 2*time.Second).Should(Succeed()) + }) + + // A Git-side push with an API edit on top of it, from + // docs/design/push-notification-and-reconcile-trigger.md §3.4. Nothing tells GitOps Reverser + // about the push here, so it finds it the hard way: its own push is rejected, it replays the API + // edit onto the Git-side commit, and only then asks Flux to apply, so the commit Flux applies + // already carries the API side's answer. Nobody reconciles Flux by hand. + It("applies a Git-side push only after replaying the API edit made on top of it", func() { + Expect(run.gitPull()).To(Succeed()) + + By("pushing a Git-side change to the first order") + run.writeLiveOrder(iceCreamOrderFile{ + Name: run.firstOrderName, + Namespace: testNs, + CustomerName: "Alice", + Container: "Cup", + Scoops: []iceCreamScoop{ + {Flavor: run.reverseFlavor, Quantity: 4}, + }, + Toppings: []string{run.reverseTopping}, + }) + Expect(run.commitAllAndPush("bi-directional: a Git-side change to the first order")).To(Succeed()) + + By("and editing the second order through the API before GitOps Reverser has heard of that push") + _, err := kubectlRunInNamespace(testNs, "patch", iceCreamCRDName(crdGroupBiDirectional), + run.secondOrderName, "--type=merge", "-p", `{"spec":{"container":"Cup"}}`) + Expect(err).NotTo(HaveOccurred()) + + By("its push is rejected, replayed onto the Git-side change, and Flux applies the result") + run.waitForCommittedOrderSpec(run.secondOrderName, "Cup", "Strawberry", "WhippedCream") + run.waitForCommittedOrderSpec(run.firstOrderName, "Cup", run.reverseFlavor, run.reverseTopping) + Expect(run.gitPull()).To(Succeed()) + run.waitForFluxKustomizationRevision(run.fluxLiveName, run.gitHEAD()) + run.waitForOrderSpec(run.firstOrderName, "Cup", run.reverseFlavor, run.reverseTopping) + + By("and the API edit is not reverted by it") + Consistently(func(g Gomega) { + g.Expect(liveOrderContainer(g, run, run.secondOrderName)).To(Equal("Cup")) + }, 20*time.Second, 2*time.Second).Should(Succeed()) + }) }) // emptyDiffAt reports whether the commit changed no file, which is the property that separates diff --git a/test/e2e/refusal_empty_commit_e2e_test.go b/test/e2e/refusal_reapply_e2e_test.go similarity index 55% rename from test/e2e/refusal_empty_commit_e2e_test.go rename to test/e2e/refusal_reapply_e2e_test.go index ff9ad150..e526edbf 100644 --- a/test/e2e/refusal_empty_commit_e2e_test.go +++ b/test/e2e/refusal_reapply_e2e_test.go @@ -14,12 +14,13 @@ import ( . "github.com/onsi/gomega" ) -// GitTarget spec.onRefusal: PushEmptyCommit, with a REAL refusal. +// GitTarget spec.onRefusal: Reapply, with a REAL refusal and a real Flux Kustomization as the +// applier. // // The refusal has to be the right KIND, and choosing it is most of what this spec is about. A // folder-level refusal (a foreign file, unparseable YAML) is normally found by the per-type -// reconcile, which blocks the collection through the event router and never reaches the live-write path -// the action hangs off. It is also the case where an empty commit achieves nothing: only a human +// reconcile, which blocks the collection through the event router and never reaches the live-write +// path the action hangs off. It is also the case where reapplying achieves nothing: only a human // can repair a folder. // // The case the action exists for is a WRITE-BOUNDARY refusal, where the folder is accepted and @@ -28,29 +29,30 @@ import ( // var, which no kustomize declaration in the overlay can express. Read scope reaches the base; // write scope never does. // -// The other half of the loop, that a new revision alone makes Flux re-apply and revert, is proven -// against real Flux in the bi-directional corner. The seam between the two specs is a commit on -// the branch with an empty diff: this one proves the operator produces it, that one proves Flux -// acts on it. Splitting there is deliberate, because a single spec would need real Flux driving a -// kustomize overlay to say anything this pair does not. -var _ = Describe("Manager Refusal Empty Commit", Label("manager", "refusal-commit"), Ordered, func() { +// Flux applies the overlay when its Kustomization is created and then waits out a 24h interval, so +// a revert of the drifted env var inside this spec is GitOps Reverser's request and nothing else. +var _ = Describe("Manager Refusal Reapply", Label("manager", "refusal-reapply"), Ordered, func() { var ( - testNs string - repo *RepoArtifacts - providerName = "refusal-commit-provider" - destName = "refusal-commit-dest" - ruleName = "refusal-commit-rule" - gitPath = "apps/checkout/overlays/prod" + testNs string + repo *RepoArtifacts + providerName = "refusal-reapply-provider" + destName = "refusal-reapply-dest" + ruleName = "refusal-reapply-rule" + gitPath = "apps/checkout/overlays/prod" + fluxName string + fluxSecretName string ) BeforeAll(func() { By("creating the test namespace") - testNs = testNamespaceFor("manager-refusal-commit") + testNs = testNamespaceFor("manager-refusal-reapply") _, _ = kubectlRun("create", "namespace", testNs) + fluxName = "refusal-reapply-" + testNs + fluxSecretName = fluxName + "-git" By("setting up Gitea repo and credentials") repo = SetupRepo(resolveE2EContext(), testNs, - fmt.Sprintf("e2e-refusal-commit-%d", GinkgoRandomSeed())) + fmt.Sprintf("e2e-refusal-reapply-%d", GinkgoRandomSeed())) _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") applySOPSAgeKeyToNamespace(testNs) @@ -64,141 +66,153 @@ var _ = Describe("Manager Refusal Empty Commit", Label("manager", "refusal-commi cleanupWatchRule(ruleName, testNs) cleanupGitTarget(destName, testNs) _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", providerName, "--ignore-not-found=true") + cleanupNamespacedResource("flux-system", "kustomization", fluxName) + cleanupNamespacedResource("flux-system", "gitrepository", fluxName) + cleanupNamespacedResource("flux-system", "secret", fluxSecretName) cleanupNamespace(testNs) }) - It("commits an empty commit when the live object has nowhere in Git to land", func() { + It("asks the applier to reapply when the live object has nowhere in Git to land", func() { By("seeding a base-and-overlay folder that renders into this test's namespace") seedFolder := writeBaseOwnedFieldFolder(testNs) DeferCleanup(func() { _ = os.RemoveAll(seedFolder) }) seedRenderedFolderIntoRepo(repo, testNs, seedFolder, "apps/checkout") - // Applied with the env var ALREADY diverged from what the folder renders. The divergence is - // the point: an empty commit asks the reconciler to re-apply, and without something to - // correct the commit would be asking for nothing. - By("applying the rendered Deployment, drifted from what Git says") - _, err := kubectlRunInNamespace(testNs, "apply", "-f", filepath.Join(seedFolder, "base", "deployment.yaml")) - Expect(err).NotTo(HaveOccurred(), "failed to apply the base Deployment") - _, err = kubectlRunInNamespace(testNs, "set", "env", "deployment/checkout", "LOG_LEVEL=debug") - Expect(err).NotTo(HaveOccurred(), "failed to drift the base-owned env var") + By("letting Flux apply the overlay once, then leaving it on a 24h interval") + applyRefusalFluxApplier(testNs, repo, fluxName, fluxSecretName, "./"+gitPath) + Eventually(func(g Gomega) { + g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info")) + }, 2*time.Minute, 2*time.Second).Should(Succeed()) - seedCount := remoteCommitCount(repo.CheckoutDir) + // The divergence is the point: a reapply that has nothing to correct would prove nothing. + By("drifting the base-owned env var away from what Git says") + _, err := kubectlRunInNamespace(testNs, "set", "env", "deployment/checkout", "LOG_LEVEL=debug") + Expect(err).NotTo(HaveOccurred(), "failed to drift the base-owned env var") - By("creating the GitTarget on the OVERLAY, asking it to commit on refusal") + By("creating the GitTarget on the OVERLAY, naming the Kustomization and asking it to reapply") createGitTarget(destName, testNs, providerName, gitPath, "main") - _, err = kubectlRunInNamespace(testNs, "patch", "gittarget", destName, - "--type=merge", "-p", `{"spec":{"onRefusal":"PushEmptyCommit"}}`) - Expect(err).NotTo(HaveOccurred(), "failed to set spec.onRefusal") + _, err = kubectlRunInNamespace(testNs, "patch", "gittarget", destName, "--type=merge", "-p", + fmt.Sprintf(`{"spec":{"applier":{"kustomizationRef":{"namespace":"flux-system","name":%q}},`+ + `"onRefusal":"Reapply"}}`, fluxName)) + Expect(err).NotTo(HaveOccurred(), "failed to set spec.applier and spec.onRefusal") + verifyResourceCondition("gittarget", destName, testNs, "ApplierConfigured", "True", "Succeeded", "") // The overlay renders into THIS namespace, so the rule watches its own and needs no - // sourceNamespace override (which would need authorizing, and would be testing that - // instead of this). + // sourceNamespace override. ConfigMaps ride along so this target has an ACCEPTED watched + // type beside the refused one: a reconcile of one type succeeding is not evidence about + // another that is still refused. Expect(applyFromTemplate("test/e2e/templates/manager/watchrule-resources.tmpl", struct { Name string Namespace string DestinationName string Resources string }{Name: ruleName, Namespace: testNs, DestinationName: destName, - // ConfigMaps ride along so this target has an ACCEPTED watched type beside the refused - // one. A review found the interaction: a reconcile of one type succeeding is not - // evidence about another that is still refused, and reading it as recovery re-armed - // the standing refusal once per pass. Resources: `"deployments","configmaps"`}, testNs)).To(Succeed(), "failed to apply the WatchRule") - By("the write is refused at the BOUNDARY, which is the kind that earns a commit") + By("the write is refused at the BOUNDARY, which is the kind that earns a reapply") waitForGitTargetGitPathRefused(destName, testNs, "WriteBoundaryRefused") verifyResourceCondition("gittarget", destName, testNs, "GitPathAccepted", "False", "WriteBoundaryRefused", "escapes the GitTarget write scope") - By("the operator commits a commit that changes no file") - var head string + By("the applier is asked, and Flux reverts the refused edit") Eventually(func(g Gomega) { - g.Expect(remoteCommitCount(repo.CheckoutDir)).To(BeNumerically(">", seedCount), - "a refused write on an opted-in target must produce a commit") - head = remoteHead(g, repo.CheckoutDir) - g.Expect(emptyDiffAtSHA(repo.CheckoutDir, head)).To(BeTrue(), - "the operator's commit must change no file; a refused write writes nothing") - }, 90*time.Second, 2*time.Second).Should(Succeed()) - - Expect(commitMessageAtSHA(repo.CheckoutDir, head)).To(ContainSubstring(destName), - "the commit must name the GitTarget whose write was refused, or git log reads as a stray no-op") - - // Past TWO full trailing intervals, not one. A window that ends inside the second interval - // only proves there was no immediate repeat, which the rate limit guarantees for free; a - // commit-per-interval loop is not visible until the third commit would be due. The refusal - // is still standing throughout, and a forced recheck re-observes it every minute, so this - // is the worst case the dedupe exists for: rechecking an unchanged refusal must not keep - // moving the branch. - // - // The bound is the initial commit plus one trailing commit. Two, not one, because the live - // write and the per-type reconcile observe the same refusal from different populations, so - // the second of them can legitimately be a first observation of its own. What must never - // grow is the count per interval after that. - By("and it settles past two trailing intervals instead of committing on a loop") - emptyBefore := emptyCommitCount(repo.CheckoutDir) + g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info"), + "a refused edit on a Reapply target must be reverted by its applier") + }, 3*time.Minute, 2*time.Second).Should(Succeed()) + Expect(emptyCommitCount(repo.CheckoutDir)).To(BeZero(), + "the request goes to the applier; nothing moves the branch for it") + + // Past two rate-limit intervals. The revert itself is a new observation of the refusal + // (the env var changed back), so one more request may follow it. What must never grow is + // the count after that: a forced recheck re-observes the standing refusal every minute. + By("and it settles instead of asking on every recheck") + seen := map[string]struct{}{fluxRequestStamps(fluxName): {}} Consistently(func(g Gomega) { - g.Expect(emptyCommitCount(repo.CheckoutDir)).To(BeNumerically("<=", emptyBefore+1), - "a standing refusal must not produce a commit per interval forever") + seen[fluxRequestStamps(fluxName)] = struct{}{} + g.Expect(len(seen)).To(BeNumerically("<=", 2), + "a standing refusal must not be requested once per recheck") }, 150*time.Second, 10*time.Second).Should(Succeed()) - // The accepted sibling type, exercised while the Deployment is still refused. Its write is - // ordinary and makes an ordinary commit; what it must NOT do is make the unchanged - // Deployment refusal look new again, which is why the assertions above and below count - // EMPTY commits rather than commits. By("a ConfigMap on the same target is mirrored normally") Expect(applyFromTemplate("test/e2e/templates/manager/configmap.tmpl", struct { Name string Namespace string }{Name: "refusal-sibling", Namespace: testNs}, testNs)).To(Succeed(), "failed to apply the sibling ConfigMap") - - afterSibling := emptyCommitCount(repo.CheckoutDir) Eventually(func(g Gomega) { g.Expect(runGitIn(repo.CheckoutDir, "fetch", "origin", "main")).To(Succeed()) g.Expect(gitShowFiles(repo.CheckoutDir)).To(ContainSubstring("refusal-sibling"), "the accepted type must reach Git, or this proves nothing about it") }, 120*time.Second, 2*time.Second).Should(Succeed()) - By("without re-arming the refusal that is still standing") - Consistently(func(g Gomega) { - g.Expect(emptyCommitCount(repo.CheckoutDir)).To(Equal(afterSibling), - "a reconcile of one watched type succeeding must not turn another type's unchanged "+ - "refusal into a new trigger") - }, 90*time.Second, 10*time.Second).Should(Succeed()) - - // A quiet branch on its own is two readings, and only one of them is the feature: the - // refusal was recognised as one already answered, or the target stopped evaluating - // anything at all. A review asked for the second to be ruled out. Editing the SAME field - // to a NEW value does that and covers the dedupe's other half at the same time — a second - // real edit is a second thing to revert, and the commit made for the first says nothing - // about it. - // // The target is NOT asked to converge, because this shape cannot: the live object carries // the API server's defaults and the base document does not, so a write for it is planned, - // and refused, whatever the env var says. Correcting the value would leave the refusal - // exactly where it is. What is provable here is that the operator keeps evaluating and - // still distinguishes a new edit from a repeat. + // and refused, whatever the env var says. What is provable is that the operator keeps + // evaluating and treats a genuinely new edit as new. By("editing the same field again, to a value nobody has been refused for yet") - beforeSecondEdit := emptyCommitCount(repo.CheckoutDir) _, err = kubectlRunInNamespace(testNs, "set", "env", "deployment/checkout", "LOG_LEVEL=trace") Expect(err).NotTo(HaveOccurred(), "failed to make the second drifting edit") - By("which earns an empty commit of its own") + By("which is reverted too") Eventually(func(g Gomega) { - g.Expect(emptyCommitCount(repo.CheckoutDir)).To(BeNumerically(">", beforeSecondEdit), - "a genuinely new refused edit must still move the branch") - }, 120*time.Second, 2*time.Second).Should(Succeed()) - - By("and then settles again, rather than resuming the loop") - settled := emptyCommitCount(repo.CheckoutDir) - Consistently(func(g Gomega) { - g.Expect(emptyCommitCount(repo.CheckoutDir)).To(BeNumerically("<=", settled+1), - "a second standing refusal must settle exactly as the first one did") - }, 150*time.Second, 10*time.Second).Should(Succeed()) + g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info"), + "a genuinely new refused edit must be requested again") + }, 3*time.Minute, 2*time.Second).Should(Succeed()) + Expect(emptyCommitCount(repo.CheckoutDir)).To(BeZero()) }) }) +// applyRefusalFluxApplier creates a GitRepository and a Kustomization for path on the spec's +// repository, both on a 24h interval, and waits for the first apply. +func applyRefusalFluxApplier(testNs string, repo *RepoArtifacts, name, secretName, path string) { + GinkgoHelper() + // The in-cluster Gitea test credential, read from the cluster at runtime. + checkout := gitCheckout{secretNamespace: testNs, gitSecretName: repo.GitSecretHTTP} + username, password := checkout.readGitCredentialSecretDataBase64() + Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-gitrepository-http.tmpl", struct { + Namespace, SecretName, Name, RepoURL, Branch, Interval, Username, Password string + }{ + Namespace: "flux-system", SecretName: secretName, Name: name, RepoURL: repo.RepoURLHTTP, + Branch: "main", Interval: "24h", Username: username, + Password: password, + }, "flux-system")).To(Succeed(), "failed to apply the Flux GitRepository") + Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-kustomization.tmpl", struct { + Namespace, Name, Path, SourceName, Interval, DependsOn string + Prune, Wait bool + TargetNS string + HasTargetNS bool + DecryptionProvider, DecryptionSecretName string + }{ + Namespace: "flux-system", Name: name, Path: path, SourceName: name, Interval: "24h", + }, "flux-system")).To(Succeed(), "failed to apply the Flux Kustomization") + _, err := kubectlRunInNamespace("flux-system", "wait", "--for=condition=Ready", + "kustomization/"+name, "--timeout=3m") + Expect(err).NotTo(HaveOccurred(), "the Kustomization must apply the overlay once") +} + +// checkoutLogLevel reads the LOG_LEVEL env var of the live checkout Deployment. +func checkoutLogLevel(g Gomega, testNs string) string { + out, err := kubectlRunInNamespace(testNs, "get", "deployment", "checkout", "-o", + `jsonpath={.spec.template.spec.containers[0].env[?(@.name=="LOG_LEVEL")].value}`) + g.Expect(err).NotTo(HaveOccurred()) + return strings.TrimSpace(out) +} + +// fluxRequestStamps is the reconcile.fluxcd.io/requestedAt annotation of the GitRepository and the +// Kustomization together: one apply request changes one of them. +func fluxRequestStamps(name string) string { + GinkgoHelper() + stamps := make([]string, 0, 2) + for _, kind := range []string{"gitrepository", "kustomization"} { + out, err := kubectlRunInNamespace("flux-system", "get", kind, name, "-o", + `jsonpath={.metadata.annotations.reconcile\.fluxcd\.io/requestedAt}`) + Expect(err).NotTo(HaveOccurred()) + stamps = append(stamps, strings.TrimSpace(out)) + } + return strings.Join(stamps, "|") +} + // writeBaseOwnedFieldFolder builds corpus shape 8 with the overlay pointed at the caller's // namespace, so the rendered object lands where a plain WatchRule already watches. // @@ -267,9 +281,8 @@ func remoteCommitCount(checkoutDir string) int { return count } -// emptyCommitCount counts the commits on the remote branch whose diff is empty, which is what the -// refusal action produces. Once a second watched type writes ordinary commits to the same branch, -// counting commits stops answering "did the refusal fire again" and this does. +// emptyCommitCount counts the commits on the remote branch whose diff is empty. A refusal moves the +// branch for nothing, so it must stay at zero however often the refusal is answered. func emptyCommitCount(checkoutDir string) int { GinkgoHelper() Expect(runGitIn(checkoutDir, "fetch", "origin", "main")).To(Succeed()) @@ -308,13 +321,6 @@ func emptyDiffAtSHA(checkoutDir, sha string) bool { return strings.TrimSpace(string(out)) == "" } -func commitMessageAtSHA(checkoutDir, sha string) string { - GinkgoHelper() - out, err := exec.Command("git", "-C", checkoutDir, "log", "-1", "--format=%B", sha).CombinedOutput() - Expect(err).NotTo(HaveOccurred(), "git log: %s", out) - return string(out) -} - func runGitIn(dir string, args ...string) error { cmd := exec.Command("git", append([]string{"-C", dir}, args...)...) if out, err := cmd.CombinedOutput(); err != nil { From ee94924e0280492453de875f3b4bfe44fce34110 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 09:24:51 +0000 Subject: [PATCH 05/36] test(e2e): count the refusal spec's empty commits from a baseline, and prove the revert was requested The seeded root commit has no parent to diff against and reads as empty, so the spec now compares against the count before the GitTarget exists. It also asserts the GitTarget's ApplyRequested Event names Refusal, so the revert is shown to come from GitOps Reverser's request rather than Flux's own schedule. Co-Authored-By: Claude Opus 5.5 (1M context) --- test/e2e/refusal_reapply_e2e_test.go | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/test/e2e/refusal_reapply_e2e_test.go b/test/e2e/refusal_reapply_e2e_test.go index e526edbf..edfe8655 100644 --- a/test/e2e/refusal_reapply_e2e_test.go +++ b/test/e2e/refusal_reapply_e2e_test.go @@ -89,6 +89,9 @@ var _ = Describe("Manager Refusal Reapply", Label("manager", "refusal-reapply"), _, err := kubectlRunInNamespace(testNs, "set", "env", "deployment/checkout", "LOG_LEVEL=debug") Expect(err).NotTo(HaveOccurred(), "failed to drift the base-owned env var") + // Relative, because the seeded root commit has no parent to diff against and reads as empty. + emptyBefore := emptyCommitCount(repo.CheckoutDir) + By("creating the GitTarget on the OVERLAY, naming the Kustomization and asking it to reapply") createGitTarget(destName, testNs, providerName, gitPath, "main") _, err = kubectlRunInNamespace(testNs, "patch", "gittarget", destName, "--type=merge", "-p", @@ -120,7 +123,11 @@ var _ = Describe("Manager Refusal Reapply", Label("manager", "refusal-reapply"), g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info"), "a refused edit on a Reapply target must be reverted by its applier") }, 3*time.Minute, 2*time.Second).Should(Succeed()) - Expect(emptyCommitCount(repo.CheckoutDir)).To(BeZero(), + Eventually(func(g Gomega) { + g.Expect(applyRequestedMessages(g, testNs, destName)).To(ContainSubstring("(Refusal)"), + "the revert must come from GitOps Reverser's Refusal request, not from Flux's own schedule") + }, 30*time.Second, 2*time.Second).Should(Succeed()) + Expect(emptyCommitCount(repo.CheckoutDir)).To(Equal(emptyBefore), "the request goes to the applier; nothing moves the branch for it") // Past two rate-limit intervals. The revert itself is a new observation of the refusal @@ -159,7 +166,7 @@ var _ = Describe("Manager Refusal Reapply", Label("manager", "refusal-reapply"), g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info"), "a genuinely new refused edit must be requested again") }, 3*time.Minute, 2*time.Second).Should(Succeed()) - Expect(emptyCommitCount(repo.CheckoutDir)).To(BeZero()) + Expect(emptyCommitCount(repo.CheckoutDir)).To(Equal(emptyBefore)) }) }) @@ -191,6 +198,15 @@ func applyRefusalFluxApplier(testNs string, repo *RepoArtifacts, name, secretNam Expect(err).NotTo(HaveOccurred(), "the Kustomization must apply the overlay once") } +// applyRequestedMessages is every ApplyRequested Event message recorded on a GitTarget. +func applyRequestedMessages(g Gomega, testNs, gitTarget string) string { + out, err := kubectlRunInNamespace(testNs, "get", "events", "--field-selector", + "involvedObject.kind=GitTarget,involvedObject.name="+gitTarget+",reason=ApplyRequested", + "-o", "jsonpath={.items[*].message}") + g.Expect(err).NotTo(HaveOccurred()) + return out +} + // checkoutLogLevel reads the LOG_LEVEL env var of the live checkout Deployment. func checkoutLogLevel(g Gomega, testNs string) string { out, err := kubectlRunInNamespace(testNs, "get", "deployment", "checkout", "-o", From 11ec79c575cf85dec5cc1aed544604b62fc5492e Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 09:31:37 +0000 Subject: [PATCH 06/36] fix(git): restart the apply trigger's quiet period on every decided write The bi-directional e2e reproduced the race the trigger exists to prevent. A window with a zero idle timeout opened, closed, committed and pushed inside one wake that ended quiet, so the quiet period was inherited from the quiet wakes before it and the catch-up went out at once. Flux applied that commit over the next API edit, and the revert was mirrored as Flux's commit. Deciding a write now restarts the quiet period wherever in the wake it happens, so a catch-up waits quietPeriod from the last write. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/git/apply_trigger_test.go | 20 ++++++++++++++++++++ internal/git/pending_writes_loop.go | 7 +++++++ 2 files changed, 27 insertions(+) diff --git a/internal/git/apply_trigger_test.go b/internal/git/apply_trigger_test.go index 93d34ec2..ce4d6b51 100644 --- a/internal/git/apply_trigger_test.go +++ b/internal/git/apply_trigger_test.go @@ -219,6 +219,26 @@ func TestServiceAppliers_AWriteRestartsTheQuietPeriod(t *testing.T) { assert.WithinDuration(t, time.Now(), loop.quietSince, time.Second) } +// TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod is the regression the +// bi-directional e2e found. A window with a zero idle timeout opens, closes, commits and pushes in a +// single wake that ends quiet, so a quiet period read only from how wakes END was inherited from +// the quiet wakes before it: the catch-up went out at once, Flux applied that commit over the next +// API edit, and the revert was mirrored as Flux's commit. +func TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) + loop.serviceAppliers() + loop.quietSince = time.Now().Add(-time.Hour) + + loop.decide(PendingWrite{}) + loop.pendingWrites, loop.pendingWritesBytes = nil, 0 // the same wake pushed it + loop.w.debt.observe(RemoteObservation{Commit: "bbb", By: ObservedByPush}) + loop.serviceAppliers() + + assert.Empty(t, requester.taken(), "the catch-up waits a quiet period from the push, not from before it") + assert.WithinDuration(t, time.Now(), loop.quietSince, time.Second) + assert.NotNil(t, loop.applierTimer) +} + func TestServiceAppliers_AForeignCommitAnswersEverythingElseOwed(t *testing.T) { loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) loop.oweReapply(refusalKeyFor(itypes.NewResourceReference("alpha", "team-a"), "deployments"), "refused", "obs") diff --git a/internal/git/pending_writes_loop.go b/internal/git/pending_writes_loop.go index 3e66dbb2..f4d6f478 100644 --- a/internal/git/pending_writes_loop.go +++ b/internal/git/pending_writes_loop.go @@ -6,6 +6,7 @@ import ( "errors" "fmt" "slices" + "time" "github.com/ConfigButler/gitops-reverser/internal/manifestanalyzer" itypes "github.com/ConfigButler/gitops-reverser/internal/types" @@ -83,6 +84,12 @@ func retainedCharge(pendingWrite *PendingWrite) int64 { // A request riding it is held from here on, so the controller never fails a save whose write can // still land. Committing it is advance's job, at the end of the wake. func (l *branchWorkerEventLoop) decide(pendingWrite PendingWrite) { + // Every decided write restarts the quiet period the apply trigger waits out, wherever in the + // wake it is decided. A wake can open a window, close it, commit and push before it ends, and + // end quiet; reading "busy" only from the state a wake ends in would let that wake inherit the + // quiet period of the wakes before it and ask the applier to catch up at once, over the next + // edit. See serviceAppliers. + l.quietSince = time.Time{} pendingWrite.charge = retainedCharge(&pendingWrite) pendingWrite.decidedAt = l.w.now() l.pendingWrites = append(l.pendingWrites, pendingWrite) From 83a5e60bf03c23b25a1de8bab8cecd12e5277990 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 09:41:47 +0000 Subject: [PATCH 07/36] docs: describe onRefusal: Reapply in the API-first publication guide Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/api-first-publication.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/api-first-publication.md b/docs/api-first-publication.md index 55146979..6edce7eb 100644 --- a/docs/api-first-publication.md +++ b/docs/api-first-publication.md @@ -187,9 +187,9 @@ A refusal is not a transient error and is not retried into success. It is the co edit never appears in Git while the worker looks healthy, so check the `GitTarget` conditions before the worker logs. -[`spec.onRefusal: PushEmptyCommit`](bi-directional.md#choosing-speconrefusal) can request Git -re-application for eligible refused edits. It leaves these gates in force, and publishing the -empty commit does not establish that the live value was restored or the target recovered. +[`spec.onRefusal: Reapply`](bi-directional.md#choosing-speconrefusal) asks the target's applier to +reapply Git for eligible refused edits. It leaves these gates in force, and a delivered request does +not establish that the live value was restored or the target recovered. Separately, a window whose finalize fails is **dropped**, not retried: its events are gone until the next resync re-derives them from the current live state. That is a deliberate choice, because From 2e5f801a7851f9bb2aa2b8f718304cd89962fe1e Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 09:53:46 +0000 Subject: [PATCH 08/36] docs(design): record the apply trigger's follow-ups What comes after the Flux applier: the Argo CD applier and how to build it, the PushEmptyCommit regression for appliers GitOps Reverser cannot name, the smaller loose ends, and the parts of the trigger the tests do not reach yet. The index entry for the trigger design now describes it as built for Flux. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/INDEX.md | 3 +- docs/design/apply-trigger-follow-ups.md | 228 ++++++++++++++++++++++++ 2 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 docs/design/apply-trigger-follow-ups.md diff --git a/docs/INDEX.md b/docs/INDEX.md index f9c48148..6ca67c5c 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -151,7 +151,8 @@ so you never have to read a proposal to find out it already shipped. | [`watch-and-catalog-architecture.md`](design/watch-and-catalog-architecture.md) | the target three-layer watch model — **needs a human call before building** | | [`metrics-observability-plan.md`](design/metrics-observability-plan.md) | **built.** the canonical metrics document, rewritten 2026-09 and shipped in the same wave. Reads the product as two ingestion paths — object state through watch, authorship through audit — joining at the resolver, and gives each stage one counter with a bounded `outcome` so the flow and its loss paths are drawable. Records what three review rounds corrected, including two reversals worth not re-litigating: the fact-loss counters must NOT be merged (an eviction counts a fact, a trim gap an occurrence, a decode error a whole batch), and a cumulative counter cannot replace a current count, which is why `status.streams` and `status.retainedDocuments` stayed. The instrument inventory lives in [`interpreting-metrics.md`](interpreting-metrics.md), not here. Open: the Grafana JSON and alert rules | | [`reconcile-triggering.md`](design/reconcile-triggering.md) | which controllers still fail to wake up | -| [`push-notification-and-reconcile-trigger.md`](design/push-notification-and-reconcile-trigger.md) | **part shipped, the rest is a menu.** Merges the old `inbound-push-notification.md` and `support-boundary/orchestrator-reconcile-trigger.md`, which argued two halves of one subject: the messages Reverser and the reconciler do not send each other. Part 1 is built and measured: the pre-push fetch is gone because the compare-and-swap push already reads the advertisement and a cycle that commits nothing still reaches it, carried by a three-flag invariant (`baseTrusted`/`worktreeDirty`/`replayRequired`, each bracketing a different span), a measured round-trip ledger, and a per-reason fetch counter. Part 2 is the two gaps: nothing tells an idle target its branch moved, and a refused write produces no commit so nothing reverts it. It also states **which side wins** (a captured API write beats Git silently; the cell worth protecting is the object a push changed that we hold no intent about) and why wiring a webhook to the `requestedAt` annotation is a correctness bug rather than a shortcut. Part 3 is six options with consequences, **doing nothing first**: a maximum age on `baseTrusted`, telling the reconciler after we push (which closes the API-first loop with **no Git-host webhook at all**), receiving a notification, taking the webhook over and forwarding it (so **we never have to work out which orchestrator object syncs a folder** — neither Argo CD nor Flux derives that mapping either), holding the push as an optimization that cannot be a barrier, triggering a reconcile to revert a refused edit, and **pushing an empty commit**, which closes that same gap for nothing: a new SHA defeats Argo CD's `selfHeal: false` skip legitimately (the skip is gated on having already synced *this revision*) and gives Flux a new artifact revision, so neither needs us to name an object, patch one, or hold an Argo sync authority. Its costs are a branch-wide blast radius and what it does under `prune: true`. Option 8 hands the action to the installer instead, and records that **executing a script mounted from a ConfigMap is declined**: it is arbitrary code execution as a process holding Git credentials, SSH and age keys and remote kubeconfigs, and Argo CD removed exactly that pattern in 2.8 in favour of a sidecar. The seam is a structured Event carrying the refused object and reason, which the existing `EventRecorder` makes nearly free. Options 7 and 8 together are why option 6, and the ownership interpreters it needs, may never be built | +| [`push-notification-and-reconcile-trigger.md`](design/push-notification-and-reconcile-trigger.md) | **built for Flux.** GitOps Reverser owns the apply trigger: a `GitTarget` names its applier, the applier's own short triggers are switched off, and the branch worker sends apply requests only on a quiet branch (no open commit window, no pending writes) — `ForeignCommit` after replaying pending writes onto somebody else's push, `Refusal` for `onRefusal: Reapply`, `CatchUp` after a quiet period so Flux's artifact never falls behind. Part 1 keeps the push-is-the-read invariant and the measured round trips; the receiver contract, status, metrics and tests are its reference part. Argo CD and the webhook applier are designed and not built | +| [`apply-trigger-follow-ups.md`](design/apply-trigger-follow-ups.md) | **open.** What comes after the Flux applier: the Argo CD applier and how to build it, the `PushEmptyCommit` regression for appliers GitOps Reverser cannot name, smaller loose ends, and the parts of the trigger the tests do not reach yet | | [`push-cooldown.md`](design/push-cooldown.md) | **partly built.** Whether the fixed `5s` push cooldown still earns its place now that the commit window exists. Ordinary timer closures already space one author writing one target, but identity changes, saves with shorter timers and buffer-limit closures commit sooner, and serialization does not batch queued commits, so the cooldown is what batches them. Option C, the failure backoff, is built in #412 and shares the push timer; removing the success wait would now remove only `lastPushAt` and its branch, so that decision waits on the event-loop measurements in §9 | | [`multi-source-audit-ingress-hardening.md`](design/multi-source-audit-ingress-hardening.md) | how independent sources authenticate to a named audit route, when annotation routing is trustworthy, and how multi-provider ingestion remains fair | | [`release-image-reuse-plan.md`](design/release-image-reuse-plan.md) | PRs 2–5 unstarted | diff --git a/docs/design/apply-trigger-follow-ups.md b/docs/design/apply-trigger-follow-ups.md new file mode 100644 index 00000000..dc80a6ed --- /dev/null +++ b/docs/design/apply-trigger-follow-ups.md @@ -0,0 +1,228 @@ +# Apply trigger: follow-ups + +> **design**: open. Written 2026-10-16, after the Flux applier landed on `docs-bi-directional-right` +> (#419). Index: [`../INDEX.md`](../INDEX.md) +> The design this continues: [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md). +> The user-facing model: [`../bi-directional.md`](../bi-directional.md). + +The branch makes GitOps Reverser decide when a Flux `Kustomization` applies a folder. This page is +what comes next, in order: Argo CD on the same branch, the regression the branch ships until then, +the smaller loose ends, and the parts of the feature the tests do not reach yet. + +## 1. Where the branch stands + +Built and tested: + +- **The decision**, on the branch worker ([`apply_trigger.go`](../../internal/git/apply_trigger.go)). + `applyDebt` turns every remote observation into what the appliers are owed: a `ForeignCommit` + for a tip the worker did not push, a `CatchUp` for its own pushes and for its first observation. + `serviceAppliers` sends requests only on a quiet branch, closes an open window when the branch + moved under it, and waits `quietPeriod` from the last decided write before a catch-up. +- **The refusal action**, `onRefusal: Reapply`, which owes the refusing target's applier a + `Refusal` request ([`refusal_reapply.go`](../../internal/git/refusal_reapply.go)). +- **Delivery**, in [`internal/applier`](../../internal/applier/): the Flux client (stamp the + `GitRepository`, or the `Kustomization` when the artifact already holds the commit), the + `Validate` behind `ApplierConfigured`, and the dispatcher (per-applier superseding, retry with + backoff, Events, the ledger behind `status.applier`). +- **The receiver**, `GitProvider.spec.receiver` on `--receiver-bind-address` + ([`receiver.go`](../../internal/webhook/receiver.go)), which turns a verified delivery into a + refresh that always reaches the remote. + +Not built: the Argo CD applier, the webhook applier. The API accepts `kustomizationRef` only. + +## 2. Next: the Argo CD applier + +### 2.1 What it has to do + +The design is +[§3.6 of the trigger design](push-notification-and-reconcile-trigger.md#36-what-a-request-does). +In short: GitOps Reverser sets `operation.sync` on the `Application` exactly as Argo CD's own +automated sync builds it +([`appcontroller.go`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2628-L2641)), +pinned to the commit the worker checked, and the `Application` runs with +`syncPolicy.automated.enabled: false` (Argo CD 3.1+) so nothing else syncs it. + +Three properties make it different from Flux, and each one has to survive into the code: + +- **No `CatchUp`.** With automated sync off, Argo CD applies nothing on its own, so an + `Application` behind GitOps Reverser's own pushes has nothing stale to apply. And every sync runs + the `Application`'s resource hooks, so syncing for no change would run them for no reason. + `ForeignCommit` and `Refusal` only. +- **The commit is pinned.** The operation names `revision`, so the sync applies what the worker + checked, not whatever the tip has become by the time Argo CD gets to it. Flux cannot do this, + which is why it gets `CatchUp`; Argo CD can, which is why it does not need it. +- **A running operation is waited for, never replaced.** `operation` is a single slot, and + Argo CD's own `SetAppOperation` refuses to overwrite one. A request that finds it occupied retries + later; the dispatcher's backoff already covers this. + +### 2.2 How I would build it + +1. **API.** Add `applicationRef` (`meta.NamespacedObjectReference`) beside `kustomizationRef`, make + both optional, and add a CEL rule on `GitTargetApplier` requiring exactly one. Document that + `quietPeriod` is ignored for `applicationRef`. Regenerate, and extend + [`hack/crdfields/fields.yaml`](../../hack/crdfields/fields.yaml). +2. **An applier interface in `internal/applier`.** Today the dispatcher calls `Flux` directly + (`attempt` builds `NewFlux`; the controller calls `Flux.Validate`). Introduce: + + ```go + type Applier interface { + Request(ctx context.Context, commit string, reason v1alpha3.ApplyReason) (Outcome, error) + Validate(ctx context.Context, repositoryURL, branch string) Verdict + } + ``` + + and a constructor that picks the implementation from the `GitTargetApplier`. Add the kind to + `applierKey`, so a `Kustomization` and an `Application` with one name never supersede each + other. Keep `applier_kind` on the metric (`kustomization`, `application`). +3. **`argocd.go`.** Read the `Application` (`argoproj.io/v1alpha1`, `applications`) through the + dynamic client. + - `Request`: refuse `CatchUp` (outcome `Current`, nothing sent), and otherwise build the + operation from the `Application`'s own spec: `prune` from `syncPolicy.automated.prune`, + `syncOptions` from `syncPolicy.syncOptions`, `retry` from `syncPolicy.retry` or `{limit: 5}`, + `source` from `spec.source` (or `sources` and `revisions` for a multi-source `Application`), + `initiatedBy.username: gitops-reverser`. Write it with an **update**, not a patch, guarded by + the `resourceVersion` the read returned. A conflict or an existing `operation` is a retry. + - `Validate`: `ApplierNotFound`; `SourceMismatch` when no source has a `repoURL` that is the + same repository (`repourl.Same`) with `targetRevision` equal to the branch (`HEAD` and an empty + `targetRevision` mean the remote's default branch, which is a mismatch unless that is the + branch); `AutomatedSyncEnabled` when `syncPolicy.automated` is present without + `enabled: false`; `Forbidden` from an access review for `update` on `applications`. + - `AutomatedSyncEnabled` still sends requests: they are correct, only the guarantee is missing. + It should be a **stall** in `applierReadiness`, because it needs a human, beside + `SourceMismatch` and `Forbidden`. +4. **The worker.** `serviceCatchUp` asks every applier on the branch. It must skip `applicationRef` + targets, which means `appliersOnBranch` carries the kind. That is the only worker change. +5. **RBAC.** `config/rbac/argocd-applier-role.yaml` and a binding (`get` and `update` on + `argoproj.io` `applications`, `create` on `selfsubjectaccessreviews`), included in + `config/rbac/kustomization.yaml` the way the Flux pair is, and Helm `applierAccess.argoCD` + beside `applierAccess.flux`. +6. **e2e, the Argo CD corner.** The corner's specs currently prove Argo CD's own behaviour + (`selfHeal: true` flaps, `selfHeal: false` keeps the edit). Keep the bookkeeping spec and the + split-ownership spec, and replace the `selfHeal` phases with: + - the race spec, as in the Flux corner, with `automated.enabled: false` and `applicationRef`; + - a spec that sets `timeout.reconciliation` short (the corner installs Argo CD, so this is + `argocd-cm`) and asserts the `Application`'s operation history holds only GitOps Reverser's + syncs while edits are in flight; + - the Git-side push with an API edit on top, asserting the synced revision is the replayed tip; + - the refusal spec against an `Application`. +7. **Docs.** Bring back the Argo CD section of the guide as the trigger design describes it; the + [docs-first version of the guide](https://github.com/ConfigButler/gitops-reverser/blob/6e700ec8/docs/bi-directional.md#configure-argo-cd) + on this branch's history is a good starting text. Update `configuration.md`'s applier section, + the architecture ground rule, and the trigger design's build order. + +### 2.3 Questions to settle before writing it + +- **`HEAD` as `targetRevision`.** Many `Application`s track `HEAD`. Is that a match for a + `GitTarget` on the remote's default branch? Resolving it needs the remote's HEAD, which the + worker's observation has (`ParentBranch` while the branch is absent, not otherwise). Simplest: + require an explicit branch, and say so in the `SourceMismatch` message. +- **Hooks on `ForeignCommit`.** Every applier on the branch is asked, whether or not the foreign + commit touched its folder (open question 3 in the design). For Flux that is a no-op apply; for + Argo CD it runs hooks. If that is a real complaint, the narrowing is the `Application`'s own + `source.path` compared with the diff, which is the Application's own declaration, not an + inference. +- **Hub Argo CD.** An `Application` usually lives in the Argo CD cluster, not the cluster it + deploys to. The current rule (the applier is looked up in the source cluster) makes the common + Argo CD topology unsupported. It needs a way to say where the `Application` lives: the control + cluster, or a kubeconfig. Decide this first, because it shapes the API. + +## 3. The regression this branch ships until then + +`onRefusal: PushEmptyCommit` is gone, and `Reapply` needs a Flux applier. So for a folder Argo CD +applies, a refused edit now stays `OutOfSync` where the empty commit used to revert it. The same is +true for any applier GitOps Reverser cannot name: a hub Argo CD, or a tool with no applier kind. + +Two ways to close it: + +- **Keep `PushEmptyCommit` as a third value**, documented as the fallback for an applier GitOps + Reverser cannot name, with the costs it always had (it wakes everything watching the branch, and + it can revert allowed edits still in a commit window). The machinery is in this branch's history + (`internal/git/refusal_touch.go` before the rename) and the guards carry over unchanged. +- **Land the Argo CD applier on this branch before it merges**, which is the plan, and note the gap + in [`UPGRADING.md`](../UPGRADING.md) only if it ships without it. + +My recommendation is the second, with the first as the fallback if the hub question in §2.3 turns +out to need its own release. + +## 4. The webhook applier + +After Argo CD. `webhook: {url, secretRef}`, a POST of `{"repository", "branch", "commit", +"reason"}` signed with `X-Signature: sha256=` (Flux's `generic-hmac` header), behind the same +`Applier` interface. It gets every reason, `CatchUp` included, because nothing says the tool on the +other end does not apply on its own. No in-process plugin, for the reason the trigger design gives. + +## 5. Smaller loose ends + +- **The ledger behind `status.applier` is never pruned** + ([`dispatcher.go`](../../internal/applier/dispatcher.go)). It is keyed by namespace and name, so a + `GitTarget` deleted and recreated under the same name inherits its predecessor's last request. + Key it by UID, or forget an entry from the GitTarget controller's deleted-object path, the way + `remotePublications` does. +- **The receiver listens on the elected replica only.** A standby refuses the connection, so a + `Service` across replicas loses deliveries to it. With one replica (the default) this is + invisible. Either bind on every replica and answer `503` until elected, which a Git host retries + or at least reports, or route to the leader. Open question 2 in the trigger design. +- **`ApplierConfigured` reads the source cluster on every reconcile**: two GETs and two access + reviews. A target that is not converged reconciles every ten seconds, so a broken target costs + that much. Cache the verdict per `(cluster, applier, generation)` for a minute, or check only on + the steady tick and on spec changes. +- **`SourceDynamicClient` serves only declared source clusters.** The dispatcher retries until the + GitTarget's watches have declared the cluster, which covers startup. A target whose declaration + keeps failing therefore never gets its requests delivered, and only the retry log says why. Count + it in `apply_requests_total{outcome="failed"}` with a reason, or surface it in `ApplierConfigured`. +- **The dispatcher's inbox is capped at 1024 and drops the oldest with a log line only.** That + can only fill if the loop is wedged, but a drop there is a lost `ForeignCommit`. Count it. +- **A catch-up when nothing was pushed since the last one.** The first observation after a restart + owes a catch-up, which is right; a refresh that confirms an unchanged tip owes nothing, which is + also right. Check that a worker replaced because its `GitProvider` was repointed starts with a + baseline of the new repository and not a `ForeignCommit` against the old one's tip. I believe it + does (a new worker has a fresh `applyDebt`), and nothing tests it. +- **The trigger design's open question 3**, path scoping for `ForeignCommit`, stays open. + +## 6. What the tests do not reach yet + +Ordered by how much I would worry about each. + +1. **The receiver end to end.** The config-dir install runs the listener and a `Service`, but no + e2e spec sends a delivery. Discovery of a foreign push by a delivery is unit-tested over a real + Git server (`TestApplyTrigger_APushNoticeFindsAForeignCommit`) and the handler is unit-tested, + but nothing proves the two are wired together in a deployed operator. A spec should sign a + `generic-hmac` body with the route's token, POST it to the `Service` from inside the cluster, + and assert Flux applies a Git-side push within seconds, with no API edit to make it surface. +2. **A foreign push found by the refresher.** The Flux spec finds the foreign push through a + rejected push, because the e2e refresh rides the five-minute reconcile tick. An idle branch with + no receiver depends entirely on the refresh path, and nothing exercises it end to end. A + reconcile-request annotation would not do (it forces a resync, the hazard of §2.2 of the design); + the test needs a way to fire the refresh, or a shorter tick under test. +3. **The race spec's margin.** It edits twice a few seconds apart and asserts no extra commit for + 30 seconds. It caught the one-wake bug, which is the reason it exists, but it depends on the + corner's commit window and push cooldown. Write the unit equivalent with an injected clock: a + window that opens, closes, commits and pushes in one wake, followed by a second write inside + `quietPeriod`, asserting no request in between. Part of this is + `TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod`; the second-write + half is not. +4. **A remote source cluster.** Every test of the applier uses the operator's own cluster. A + `ClusterProvider` with a kubeconfig, a `Kustomization` in that cluster, and the + `Forbidden` path when the kubeconfig's identity lacks the grant, are all untested. +5. **`ApplierConfigured` against real RBAC.** e2e covers `True` only. `Forbidden` (drop the + flux-applier binding), `SourceMismatch` (point the `Kustomization` at another branch) and + `ApplierNotFound` (name one that does not exist, then create it, and watch the target recover) + are unit-tested with a fake access check. +6. **Several GitTargets naming one `Kustomization`.** The dispatcher's dedupe is unit-tested; no + e2e has two folders on one branch applied by one `Kustomization`. +7. **A foreign commit while the remote is unreachable.** The rule is that the request waits with + the pending writes (§3.4 of the design). Nothing tests that it does not go out during the + outage, or that it goes out once the push lands. +8. **A foreign commit closing a window a `CommitRequest` is attached to.** + `closeWindowForForeignCommit` finalizes it early; the request should resolve `Committed` with + the replayed commit. Untested. +9. **Upgrade with a stored `PushEmptyCommit`.** `EffectiveOnRefusal` reads it as `Ignore` (unit + level). The CRD rejecting an update until the field changes, and the CEL rule rejecting + `Reapply` without an applier, are not exercised by an envtest. +10. **The Helm install.** Local e2e is `INSTALL_MODE=config-dir`, so the chart's receiver flag, + `Service` and `applierAccess.flux` role are covered by `helm template` tests only. CI's + quickstart leg installs the chart, but with neither value set. +11. **The race detector in CI.** `applyDebt` is fed from `publishObservation`, which can run off + the loop goroutine (the first clone), and the dispatcher delivers on goroutines of its own. Both + are lock-guarded, and their tests pass under `-race -count=3` locally (2026-10-16), but CI does + not run `-race`, so a later change can regress this unnoticed. From 65621bd81ef0deab52f857f4000d370814a2fc31 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 10:16:03 +0000 Subject: [PATCH 09/36] docs: match the apply trigger docs to what the code does A delivery waits on the worker's queue behind an in-flight push rather than being compared against a recorded in-flight commit; the foreign-commit test compares against the last observed tip; a Flux request stamps the GitRepository or the Kustomization, not both; the SourceMismatch example uses the real message; and the receiver's Helm default is empty. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 6 +++--- docs/bi-directional.md | 5 +++-- docs/configuration.md | 4 ++-- ...push-notification-and-reconcile-trigger.md | 19 ++++++++++--------- 4 files changed, 18 insertions(+), 16 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 41e6f726..6157b0c9 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -653,9 +653,9 @@ flowchart TD style REQ fill:#fff3e0,stroke:#ff9800 ``` -The worker records the commit of a push while the push is in flight, so a delivery that arrives -before `PushAtomic` returns is still recognized as its own. A rollback onto one of its own older -commits is foreign, because the advertisement names a tip that is not the last one it pushed. +A delivery is handled on the worker's own queue, so it is never judged while a push is in flight: +the push has recorded its commit before the delivery is read. A rollback onto one of the worker's +older commits is foreign, because the advertisement names a tip other than the last one observed. The same quiet-branch rule sends two other requests: `Refusal`, to the refusing target's applier when `spec.onRefusal: Reapply` is set, and `CatchUp`, to every applier on the branch once it has diff --git a/docs/bi-directional.md b/docs/bi-directional.md index e161c5d2..c34ed1bd 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -146,8 +146,9 @@ spec: GitOps Reverser looks the applier up in the **source cluster**, the cluster it applies to, with the identity the `ClusterProvider` already uses. That identity needs permission to read and patch -the applier; the chart grants it for the operator's own cluster. The applier must apply this -folder's repository and branch, or the target reports `ApplierConfigured=False`. The +the applier; the chart grants it for the operator's own cluster with `applierAccess.flux: true`. +The applier must apply this folder's repository and branch, or the target reports +`ApplierConfigured=False`. The [configuration reference](configuration.md#who-applies-this-folder-specapplier) has every field and reason. diff --git a/docs/configuration.md b/docs/configuration.md index 8faeedbd..db2c3706 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1151,7 +1151,7 @@ status: - type: ApplierConfigured status: "False" reason: SourceMismatch - message: 'Kustomization flux-system/checkout applies GitRepository flux-system/infra, whose branch is "release", not "main"' + message: 'GitRepository flux-system/infra fetches branch "release", not main' applier: requestedCommit: 4f2c1ab9... requestedAt: "2026-10-20T10:14:03Z" @@ -2460,7 +2460,7 @@ present the client certificate the audit listener requires, so the two cannot sh | Flag | Helm value | Default | What it does | | --- | --- | --- | --- | -| `--receiver-bind-address` | `receiver.bindAddress` | `0` (off) | `host:port` the receiver listens on. `0` serves no receiver | +| `--receiver-bind-address` | `receiver.bindAddress` | `0` (off); Helm: empty (off) | `host:port` the receiver listens on, such as `:9292`. `0` or empty serves no receiver | The chart creates a `-receiver` Service when the listener is on. Expose it to your Git host with an Ingress or a Gateway route of your own that terminates TLS, the way Flux's diff --git a/docs/design/push-notification-and-reconcile-trigger.md b/docs/design/push-notification-and-reconcile-trigger.md index d100a45e..74b450ed 100644 --- a/docs/design/push-notification-and-reconcile-trigger.md +++ b/docs/design/push-notification-and-reconcile-trigger.md @@ -184,8 +184,9 @@ on its own goroutine, since every write and every window passes through its queu | A rejected compare-and-swap push | Already on the worker; the replay that follows is the response | | The periodic refresh | Already on the worker, every `--git-refresh-interval` on an idle branch | -All three end in the same comparison: the remote's tip against the last commit the worker pushed. -Equal means its own work. Anything else is foreign, including a tip the worker cannot classify. +All three end in the same comparison: the remote's tip against the last tip the worker observed, +by a push or a fetch. Unchanged is nothing new; a change proved by the worker's own push is its own +work; a change any fetch found is foreign. A worker's first observation is a baseline (§3.5). **A delivery is a hint, never a fact.** The worker does not trust the commit a delivery names. On an idle branch it reads the advertisement (one request), compares, and fetches only when the tip is @@ -197,8 +198,8 @@ cases come out right without bookkeeping: tip, the worker's latest push, so it is own work. - **A rollback onto one of the worker's older commits.** The tip is not the last push, so it is foreign, and the rollback is applied. -- **A delivery that arrives before `PushAtomic` returns.** The worker records the commit it is - pushing before it sends, so the comparison is against the commit in flight. +- **A delivery that arrives while a push is in flight.** It waits on the worker's queue behind the + push, which has recorded its commit by the time the delivery is read. - **The delivery for the replayed push `F + W`.** It names the worker's own push and is dropped, which is correct: the foreign commit `F` already set the applier behind, and the request is owed whether or not this delivery arrives. @@ -263,11 +264,11 @@ applier's own triggers off, withholding the request would stall Git-side deploym ### 3.6 What a request does **Flux (`kustomizationRef`).** The request `flux reconcile kustomization --with-source` makes: -stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository` named by the `Kustomization`'s -`sourceRef`, wait until its artifact names the requested commit or a later one, then stamp the -`Kustomization` unless it already started reconciling that artifact. A `Refusal` needs the second -stamp, since a reapply at an unchanged commit produces no new artifact to wake it. A `CatchUp` stamps -only the `GitRepository`. Permissions: `get` and `patch` on `gitrepositories` and `kustomizations`. +when the `GitRepository` named by the `Kustomization`'s `sourceRef` does not hold the requested +commit yet, stamp `reconcile.fluxcd.io/requestedAt` on it, and the new artifact wakes the +`Kustomization` by itself; when it already holds the commit, stamp the `Kustomization` instead, which +is what a `Refusal` at an unchanged commit needs. A `CatchUp` whose artifact already holds the commit +stamps nothing. Permissions: `get` and `patch` on `gitrepositories` and `kustomizations`. **Argo CD (`applicationRef`).** Set `operation` on the `Application` as Argo CD's own automated sync builds it ([`appcontroller.go`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2628-L2641)), From eb9dc035a148816281e4abf932f3e749d42cef36 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 18:27:51 +0000 Subject: [PATCH 10/36] docs(design): plan the review fixes before the apply trigger merges Record the 2026-10-16 review's six findings and their shared cause (a request decided once is never checked again at the patch), the fix for each, and the hand-back that replaces giving up on a failed request. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/apply-trigger-follow-ups.md | 230 ++++++++++++++++++++---- 1 file changed, 197 insertions(+), 33 deletions(-) diff --git a/docs/design/apply-trigger-follow-ups.md b/docs/design/apply-trigger-follow-ups.md index dc80a6ed..f9671ecb 100644 --- a/docs/design/apply-trigger-follow-ups.md +++ b/docs/design/apply-trigger-follow-ups.md @@ -6,8 +6,9 @@ > The user-facing model: [`../bi-directional.md`](../bi-directional.md). The branch makes GitOps Reverser decide when a Flux `Kustomization` applies a folder. This page is -what comes next, in order: Argo CD on the same branch, the regression the branch ships until then, -the smaller loose ends, and the parts of the feature the tests do not reach yet. +what comes next, in order: the fixes the 2026-10-16 review found, which block the merge; Argo CD; +the breaking change; the smaller loose ends; and the parts of the feature the tests do not reach +yet. ## 1. Where the branch stands @@ -30,9 +31,172 @@ Built and tested: Not built: the Argo CD applier, the webhook applier. The API accepts `kustomizationRef` only. -## 2. Next: the Argo CD applier +## 2. Before merge: what the review found -### 2.1 What it has to do +The review of `21df16be` (`docs/reviews/20261016-review.md`, kept out of the tree) found six +problems. Five hold as reported, and one holds only for its wording. They have one cause: the +worker decides a request once, and nothing after that decision checks it again. The dispatcher +attempts it for up to four minutes after the worker decided, against whatever the cluster, the +`GitTarget` and the branch have become in the meantime. GitOps Reverser now controls when +deployments happen, so each decision has to hold both when it is made and when a patch is sent. + +### 2.1 A target the ClusterProvider denies still drives its applier (finding 1) + +`appliersOnBranch` lists every `GitTarget` on the provider and branch that names an applier, and +`Dispatcher.resolve` checks nothing more. `SourceDynamicClient` hands out a cluster's client when +**any** target declared that cluster, so a target with `Validated=False/NamespaceNotAuthorized` +rides on another target's declaration. That target can be kept on a running worker by any +legitimate target in its namespace on the same branch. The reviewer reproduced the patch. + +The damage is bounded: the request is a reconcile annotation, and once §2.2 is fixed it reaches only +a `Kustomization` that applies this repository and branch. It is still an authorization bypass, +and the fix is cheap. The dispatcher calls `authz.GitTargetAdmitted` for every target in a job, +on every attempt (see §2.4), and drops a target it denies. A client being available is not +permission to use it. + +One more gap sits at the same fix site, and the review did not raise it: `kustomizationRef.namespace` +is unrestricted, so an admitted target can name a `Kustomization` in any namespace of the source +cluster, while the operator's `flux-applier` role grants `patch` cluster-wide. A WatchRule needs +`allowAnySourceNamespace` on its ClusterProvider to reach outside its own namespace, and the applier +reference should be held to the same rule. + +### 2.2 A `SourceMismatch` applier is still driven (finding 2) + +`Flux.Validate` checks the `GitRepository`'s URL and branch, but `Flux.Request` does not, and the +dispatcher never reads the verdict. A target that publishes `main` and names a `Kustomization` whose +source follows `release` reports `ApplierConfigured=False/SourceMismatch`, yet every foreign commit +on `main` still stamps that `release` source. The reviewer reproduced this as well. + +`Request` already reads both objects, so it runs `sourceMismatch` on them before it stamps and +returns a `permanentError` on a mismatch. Reading the published condition instead would leave the +race between the check and the patch. This means `Request` needs the expected repository URL and +branch, which the job does not carry today. The dispatcher reads the `GitProvider` when it resolves +the job. This also shapes the `Applier` interface in §3.2: the expected source binding is an input +to `Request`, not only to `Validate`. + +### 2.3 A delayed Refusal outlives its consent and its recovery (finding 3) + +There are two waiting places for a refusal, and only one of them checks again. `flushPendingReapply`, +which handles the rate-limit trailer, calls `refusalConsent`, as that function's comment promises. +`reapplyOwed` does not. A refusal owed while the branch is busy, behind an open window or a +publication outage, is sent by `serviceRefusals` once the branch is quiet, and the only check left +is whether the target still names an applier. So it goes out after the target is suspended or set to +`Ignore`. + +`refusalRecovered` cancels `refusalPending` but leaves `reapplyOwed` alone, because `reapplyOwed` is +keyed by target and has lost the collection it would have to match. A refusal that has recovered, +which means its edit was accepted and is now pending, still asks for an apply once the branch is +quiet. By then the tip carries the accepted edit, so the apply is harmless, but it is still a +request made for a refusal that no longer exists. + +Fix both in the worker: + +- Key `reapplyOwed` by `refusalKey`, and have `refusalRecovered` delete the matching entries the way + it already does for `refusalPending`. +- Have `serviceRefusals` call `refusalConsent` before it sends. + +The dispatcher has to check consent again on its own attempts (see §2.4). A job's reason is per +applier, but consent is per target: a `Refusal` job can have absorbed a `CatchUp` for a sibling +target (`supersedingReason`). The job therefore has to keep each target's own reason. When the only +target that consented drops out, the job falls back to `CatchUp` for the rest rather than being +dropped. + +### 2.4 One rule for all three: check again at the patch + +§2.1 to §2.3 need the same check, so it should be written once. Before each attempt, the dispatcher +reads each target in the job again from the cache and keeps only the targets that: + +- still name this applier, as `resolve` already checks once, +- `authz.GitTargetAdmitted` admits (§2.1), +- consent to a refusal, for a target that is in the job only for a `Refusal` (§2.3). + +Then `Request` checks the source binding (§2.2). A job with no target left is dropped and counted. +This is the last gate before the patch, so the retries go through it too, which is the point the +review makes about "including retries". + +### 2.5 A request given up on is lost (finding 4) + +After eight failed attempts the dispatcher emits `ApplyRequestFailed` and discards the job, while +the worker cleared its debt when it sent the request. The `DefaultBackoff` comment reads "Nothing is +lost by giving up", but that assumes another request comes, and none has to. A refresh that sees +the same tip owes nothing, so a `ForeignCommit` that failed while the source API was down waits for +the next commit or for Flux's long backstop interval. A failed `Refusal` is worse: `oweReapply` +marks the observation as covered when the request is owed, so the same standing refusal never owes +another request until it recovers. + +The review asks to "retain or reconstruct" the request, and I would reconstruct it in the worker +rather than make the dispatcher retry forever. The reason is one the review does not raise: **a +retry is a request sent without the quiet check.** The worker decided it on a quiet branch, and the +dispatcher replays that decision up to four minutes later. Flux cannot pin a commit, so the retry +applies whatever the tip is then, and live edits waiting in an open window by that time are +reverted. That is the hazard this whole design exists to close. Retrying for longer makes the +window longer. + +The shape I propose: + +- The dispatcher keeps its short retries for a blip of a few seconds, since a request decided a + moment ago is still sound. It hands a job back to the worker once the job has failed for longer + than about one `quietPeriod`, or permanently, instead of giving up. "Handing back" is a new call on + the requester's side of `git.ApplyRequester`, delivered as an event on the worker's loop. +- The worker puts the debt back (`applyDebt.restore`, or `reapplyOwed` for a refusal). It re-sends + only on a quiet branch, with its own backoff capped at a few minutes so that a permanent failure + does not produce a tight loop. +- `ApplyRequestFailed` stays, emitted once each time a request goes from delivering to failing, + not on every attempt. + +A `permanentError`, which is a misconfigured `Kustomization` and now also a source mismatch, then +stays owed and is delivered once the configuration is fixed, rather than being forgotten. A +restart loses the hand-back state, which is fine: the first observation owes a `CatchUp`, and that +stamps the source when its artifact is behind. + +### 2.6 A Refusal can wake folders that share the GitRepository (finding 5) + +This one is a contract problem, and no reason to enforce isolation. When the artifact +is behind the tip, `Request` stamps the `GitRepository`, the new revision wakes every +`Kustomization` on it, and those reapply as well. A `Kustomization` that stamped only itself would be +worse: it reconciles from the old artifact and reverts edits that are already in Git. So the source +has to move, and moving the source wakes everything that consumes it. That is how Flux works, not +something GitOps Reverser can avoid. + +The harm is small. The branch is quiet, because a Refusal is sent only then, so the woken siblings +apply a tip with nothing live left out of it. They would have applied the same tip at their own +`CatchUp` one `quietPeriod` later. A `Kustomization` that GitOps Reverser does not know about would +apply it at its next interval. What is wrong is the promise. Five places say a refusal wakes no +other folder: + +- the `onRefusal` doc comment in `gittarget_types.go`, +- [`bi-directional.md`](../bi-directional.md), +- [`configuration.md`](../configuration.md), +- [`UPGRADING.md`](../UPGRADING.md), +- §4.5 of the trigger design. + +The fix is to narrow all five: only this target's applier is asked, and if its `GitRepository` has +to fetch to reach the tip, every `Kustomization` on that `GitRepository` applies the new revision, as +it would for any push. Give the folder its own `GitRepository` for strict isolation. + +### 2.7 Validation can hold the GitTarget controller indefinitely (finding 6) + +`applierVerdict` calls `ApplierClients` and `Validate` with the reconcile context, which has no +deadline. The source client has no request timeout either, by design, because it carries the +watches. The GitTarget controller runs one reconcile worker, so an API server that accepts the +connection and never answers stops every GitTarget's reconcile, not only this target's. Wrap both +calls in a bounded context. About 10 seconds is enough: `Validate` makes two GETs and two access +reviews, and a timeout is already `ReasonCheckFailed`, so `keptOrUnknown` keeps the previous verdict. +This is the same fix site as the per-reconcile cost in §6. Do the caching there together with this. + +### 2.8 Validation the review ran + +The review ran `task lint`, `task test` and `task test-e2e-bi-directional`, and all three were +green. `task test-e2e` failed one spec, audit-route attribution, which passed on its own. That spec +is the known flaky one: it has failed twice in CI on comment-only commits, and both passed on a +rerun. It is not evidence against this branch. Each fix above gets a unit test: + +- the reviewer's reproductions for findings 1 to 4 and 6, kept as regression tests, +- a test that a hand-back is re-sent only on a quiet branch. + +## 3. Next: the Argo CD applier + +### 3.1 What it has to do The design is [§3.6 of the trigger design](push-notification-and-reconcile-trigger.md#36-what-a-request-does). @@ -55,7 +219,7 @@ Three properties make it different from Flux, and each one has to survive into t Argo CD's own `SetAppOperation` refuses to overwrite one. A request that finds it occupied retries later; the dispatcher's backoff already covers this. -### 2.2 How I would build it +### 3.2 How I would build it 1. **API.** Add `applicationRef` (`meta.NamespacedObjectReference`) beside `kustomizationRef`, make both optional, and add a CEL rule on `GitTargetApplier` requiring exactly one. Document that @@ -65,13 +229,18 @@ Three properties make it different from Flux, and each one has to survive into t (`attempt` builds `NewFlux`; the controller calls `Flux.Validate`). Introduce: ```go + // Binding is the repository and branch the applier must apply from. + type Binding struct{ RepositoryURL, Branch string } + type Applier interface { - Request(ctx context.Context, commit string, reason v1alpha3.ApplyReason) (Outcome, error) - Validate(ctx context.Context, repositoryURL, branch string) Verdict + Request(ctx context.Context, want Binding, commit string, reason v1alpha3.ApplyReason) (Outcome, error) + Validate(ctx context.Context, want Binding) Verdict } ``` - and a constructor that picks the implementation from the `GitTargetApplier`. Add the kind to + `Request` takes the binding because it checks it before it writes anything (§2.2). An + `Application` whose source no longer matches is refused at delivery the same way a + `Kustomization` is. Then add a constructor that picks the implementation from the `GitTargetApplier`. Add the kind to `applierKey`, so a `Kustomization` and an `Application` with one name never supersede each other. Keep `applier_kind` on the metric (`kustomization`, `application`). 3. **`argocd.go`.** Read the `Application` (`argoproj.io/v1alpha1`, `applications`) through the @@ -96,7 +265,7 @@ Three properties make it different from Flux, and each one has to survive into t `argoproj.io` `applications`, `create` on `selfsubjectaccessreviews`), included in `config/rbac/kustomization.yaml` the way the Flux pair is, and Helm `applierAccess.argoCD` beside `applierAccess.flux`. -6. **e2e, the Argo CD corner.** The corner's specs currently prove Argo CD's own behaviour +6. **e2e, the Argo CD corner.** The corner's specs currently prove Argo CD's own behavior (`selfHeal: true` flaps, `selfHeal: false` keeps the edit). Keep the bookkeeping spec and the split-ownership spec, and replace the `selfHeal` phases with: - the race spec, as in the Flux corner, with `automated.enabled: false` and `applicationRef`; @@ -110,7 +279,7 @@ Three properties make it different from Flux, and each one has to survive into t on this branch's history is a good starting text. Update `configuration.md`'s applier section, the architecture ground rule, and the trigger design's build order. -### 2.3 Questions to settle before writing it +### 3.3 Questions to settle before writing it - **`HEAD` as `targetRevision`.** Many `Application`s track `HEAD`. Is that a match for a `GitTarget` on the remote's default branch? Resolving it needs the remote's HEAD, which the @@ -126,32 +295,25 @@ Three properties make it different from Flux, and each one has to survive into t Argo CD topology unsupported. It needs a way to say where the `Application` lives: the control cluster, or a kubeconfig. Decide this first, because it shapes the API. -## 3. The regression this branch ships until then - -`onRefusal: PushEmptyCommit` is gone, and `Reapply` needs a Flux applier. So for a folder Argo CD -applies, a refused edit now stays `OutOfSync` where the empty commit used to revert it. The same is -true for any applier GitOps Reverser cannot name: a hub Argo CD, or a tool with no applier kind. - -Two ways to close it: +## 4. The breaking change -- **Keep `PushEmptyCommit` as a third value**, documented as the fallback for an applier GitOps - Reverser cannot name, with the costs it always had (it wakes everything watching the branch, and - it can revert allowed edits still in a commit window). The machinery is in this branch's history - (`internal/git/refusal_touch.go` before the rename) and the guards carry over unchanged. -- **Land the Argo CD applier on this branch before it merges**, which is the plan, and note the gap - in [`UPGRADING.md`](../UPGRADING.md) only if it ships without it. +`onRefusal: PushEmptyCommit` is gone, and `Reapply` needs a Flux applier, so until §3 lands a +folder that Argo CD applies has no automatic revert. GitOps Reverser has no users yet, so there is +nothing to migrate and no reason to hold the merge for Argo CD. The branch merges once §2 is done, +and Argo CD can come in its own PR. Do not bring `PushEmptyCommit` back as a fallback. -My recommendation is the second, with the first as the fallback if the hub question in §2.3 turns -out to need its own release. +One fix in [`UPGRADING.md`](../UPGRADING.md): its entry says to set `onRefusal: Reapply` before +upgrading, which the old CRD rejects. The order is `Ignore`, then upgrade, then set +`spec.applier`, then `Reapply`. -## 4. The webhook applier +## 5. The webhook applier After Argo CD. `webhook: {url, secretRef}`, a POST of `{"repository", "branch", "commit", "reason"}` signed with `X-Signature: sha256=` (Flux's `generic-hmac` header), behind the same `Applier` interface. It gets every reason, `CatchUp` included, because nothing says the tool on the other end does not apply on its own. No in-process plugin, for the reason the trigger design gives. -## 5. Smaller loose ends +## 6. Smaller loose ends - **The ledger behind `status.applier` is never pruned** ([`dispatcher.go`](../../internal/applier/dispatcher.go)). It is keyed by namespace and name, so a @@ -165,11 +327,13 @@ other end does not apply on its own. No in-process plugin, for the reason the tr - **`ApplierConfigured` reads the source cluster on every reconcile**: two GETs and two access reviews. A target that is not converged reconciles every ten seconds, so a broken target costs that much. Cache the verdict per `(cluster, applier, generation)` for a minute, or check only on - the steady tick and on spec changes. + the steady tick and on spec changes. Do this together with the deadline in §2.7. - **`SourceDynamicClient` serves only declared source clusters.** The dispatcher retries until the GitTarget's watches have declared the cluster, which covers startup. A target whose declaration keeps failing therefore never gets its requests delivered, and only the retry log says why. Count it in `apply_requests_total{outcome="failed"}` with a reason, or surface it in `ApplierConfigured`. + This check answers whether the cluster has a client yet. It does not answer whether this target + may use that client, which is §2.1. - **The dispatcher's inbox is capped at 1024 and drops the oldest with a log line only.** That can only fill if the loop is wedged, but a drop there is a lost `ForeignCommit`. Count it. - **A catch-up when nothing was pushed since the last one.** The first observation after a restart @@ -179,7 +343,7 @@ other end does not apply on its own. No in-process plugin, for the reason the tr does (a new worker has a fresh `applyDebt`), and nothing tests it. - **The trigger design's open question 3**, path scoping for `ForeignCommit`, stays open. -## 6. What the tests do not reach yet +## 7. What the tests do not reach yet Ordered by how much I would worry about each. @@ -212,13 +376,13 @@ Ordered by how much I would worry about each. e2e has two folders on one branch applied by one `Kustomization`. 7. **A foreign commit while the remote is unreachable.** The rule is that the request waits with the pending writes (§3.4 of the design). Nothing tests that it does not go out during the - outage, or that it goes out once the push lands. + outage, or that it goes out once the push lands. The same test with a `Refusal` owed during the + outage, and the target set to `Ignore` before the outage ends, covers §2.3. 8. **A foreign commit closing a window a `CommitRequest` is attached to.** `closeWindowForForeignCommit` finalizes it early; the request should resolve `Committed` with the replayed commit. Untested. -9. **Upgrade with a stored `PushEmptyCommit`.** `EffectiveOnRefusal` reads it as `Ignore` (unit - level). The CRD rejecting an update until the field changes, and the CEL rule rejecting - `Reapply` without an applier, are not exercised by an envtest. +9. **The CEL rule rejecting `Reapply` without an applier** is not exercised by an envtest. A stored + `PushEmptyCommit` needs no test, because no user has one (§4). 10. **The Helm install.** Local e2e is `INSTALL_MODE=config-dir`, so the chart's receiver flag, `Service` and `applierAccess.flux` role are covered by `helm template` tests only. CI's quickstart leg installs the chart, but with neither value set. From 994d92bb4bd265b30c36a1f628074f87aec8cc21 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 19:07:59 +0000 Subject: [PATCH 11/36] fix(applier): check every apply request again at the patch The worker decided a request once, and the dispatcher attempted it for up to four minutes against whatever the cluster, the GitTarget and the branch had become. Every attempt now reads each GitTarget again and keeps only those that still name the applier on this branch and that their ClusterProvider admits; a Refusal needs consent at the attempt, and a target that withdrew it is asked a CatchUp. Flux.Request checks the GitRepository's repository and branch before it stamps anything. A request that cannot be delivered is retried for about one quiet period, then handed back to the branch worker, which owes it again and sends it on a quiet branch with its own backoff. A refusal owed while the branch was busy is checked for consent again before it is sent, and a collection that recovers cancels its own obligation. The applier check runs under a deadline, and the chart's Flux role drops list and watch. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../templates/applier-rbac.yaml | 4 - cmd/main.go | 2 + cmd/main_chart_receiver_test.go | 4 +- internal/applier/dispatcher.go | 358 +++++++++++++----- internal/applier/dispatcher_test.go | 275 ++++++++++++-- internal/applier/flux.go | 34 +- internal/applier/flux_test.go | 44 ++- internal/controller/gittarget_applier.go | 12 +- internal/controller/gittarget_applier_test.go | 33 ++ internal/git/apply_handback_test.go | 260 +++++++++++++ internal/git/apply_request.go | 82 ++++ internal/git/apply_trigger.go | 204 ++++++++-- internal/git/branch_worker.go | 14 +- internal/git/refusal_observation_test.go | 2 +- internal/git/refusal_reapply.go | 66 +++- internal/git/refusal_reapply_test.go | 8 +- internal/git/types.go | 3 + 17 files changed, 1226 insertions(+), 179 deletions(-) create mode 100644 internal/git/apply_handback_test.go diff --git a/charts/gitops-reverser/templates/applier-rbac.yaml b/charts/gitops-reverser/templates/applier-rbac.yaml index 9a0f1fe6..60f39f66 100644 --- a/charts/gitops-reverser/templates/applier-rbac.yaml +++ b/charts/gitops-reverser/templates/applier-rbac.yaml @@ -18,8 +18,6 @@ rules: - kustomizations verbs: - get - - list - - watch - patch - apiGroups: - source.toolkit.fluxcd.io @@ -27,8 +25,6 @@ rules: - gitrepositories verbs: - get - - list - - watch - patch - apiGroups: - authorization.k8s.io diff --git a/cmd/main.go b/cmd/main.go index d4863e15..a138d00d 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -186,6 +186,8 @@ func main() { applyDispatcher := applier.NewDispatcher( mgr.GetClient(), watchMgr.SourceDynamicClient, mgr.GetEventRecorderFor("gittarget"), ctrl.Log.WithName("applier")) + // A request the dispatcher cannot deliver goes back to its branch worker, which owes it again. + applyDispatcher.HandBack = workerManager fatalIfErr(mgr.Add(applyDispatcher), "unable to add apply request dispatcher to manager") workerManager.SetApplyRequester(applyDispatcher) diff --git a/cmd/main_chart_receiver_test.go b/cmd/main_chart_receiver_test.go index 15e45e5c..e83589cc 100644 --- a/cmd/main_chart_receiver_test.go +++ b/cmd/main_chart_receiver_test.go @@ -175,12 +175,12 @@ func TestChartApplierAccess_Flux(t *testing.T) { { APIGroups: []string{"kustomize.toolkit.fluxcd.io"}, Resources: []string{"kustomizations"}, - Verbs: []string{"get", "list", "watch", "patch"}, + Verbs: []string{"get", "patch"}, }, { APIGroups: []string{"source.toolkit.fluxcd.io"}, Resources: []string{"gitrepositories"}, - Verbs: []string{"get", "list", "watch", "patch"}, + Verbs: []string{"get", "patch"}, }, { APIGroups: []string{"authorization.k8s.io"}, diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index dd859867..007f0975 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -23,6 +23,7 @@ import ( "sigs.k8s.io/controller-runtime/pkg/manager" "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/authz" "github.com/ConfigButler/gitops-reverser/internal/git" "github.com/ConfigButler/gitops-reverser/internal/telemetry" itypes "github.com/ConfigButler/gitops-reverser/internal/types" @@ -32,7 +33,9 @@ import ( const ( // EventReasonApplyRequested is a request that reached the applier. EventReasonApplyRequested = "ApplyRequested" - // EventReasonApplyRequestFailed is a request given up on after its last attempt. + // EventReasonApplyRequestFailed is a request that could not be delivered: handed back to the + // branch worker after its retries, which asks again once the branch is quiet. Recorded once per + // failing request, not on every retry or on every time the worker asks again. EventReasonApplyRequestFailed = "ApplyRequestFailed" ) @@ -43,6 +46,7 @@ const ( outcomeSent = "sent" outcomeFailed = "failed" outcomeSuperseded = "superseded" + outcomeDropped = "dropped" ) // maxInbox bounds the requests waiting to be resolved. The loop drains the inbox the moment it is @@ -59,24 +63,20 @@ const shortCommitLength = 8 type SourceClients func(ctx context.Context, clusterID string) (dynamic.Interface, error) // Backoff is the retry schedule for one apply request: Initial after the first failure, doubling, -// capped at Max, and given up on after Attempts attempts in all. +// capped at Max. How long a request is retried is not the schedule's to say (see Dispatcher). type Backoff struct { - Initial time.Duration - Max time.Duration - Attempts int + Initial time.Duration + Max time.Duration } -// DefaultBackoff gives an applier about four minutes to answer before a request is given up on. -// That is long enough to ride out a source-cluster API server restart, and short enough that a -// misconfigured applier says so (ApplyRequestFailed) while somebody is still looking. Nothing is -// lost by giving up: the next request for the applier carries a commit at least as new. +// DefaultBackoff retries quickly: the retries are for a blip of a few seconds. func DefaultBackoff() Backoff { - return Backoff{Initial: defaultBackoffInitial, Max: time.Minute, Attempts: defaultBackoffAttempts} + return Backoff{Initial: defaultBackoffInitial, Max: defaultBackoffMax} } const ( - defaultBackoffInitial = 2 * time.Second - defaultBackoffAttempts = 8 + defaultBackoffInitial = 2 * time.Second + defaultBackoffMax = 15 * time.Second ) // delay is the wait after the given number of consecutive failures (at least one). @@ -105,7 +105,21 @@ const DefaultAttemptTimeout = 30 * time.Second // older. A CatchUp never weakens what it supersedes, though. A ForeignCommit or Refusal stamps the // Kustomization when the artifact already holds the commit, and a CatchUp does not, so a CatchUp // arriving behind a Refusal keeps the Refusal's reason with the newer commit; otherwise the revert -// the Refusal asked for would be lost whenever the artifact had already caught up. +// the Refusal asked for would be lost whenever the artifact had already caught up. Reasons are kept +// per GitTarget, because consent to a Refusal is per GitTarget. +// +// The worker decided each request once, and an attempt can come later, against whatever the +// cluster, the GitTarget and the branch have become. So every attempt, retries included, first +// reads each GitTarget again and keeps only those that still name this applier on this branch, +// that their ClusterProvider still admits, and, for a Refusal, that still consent; a GitTarget that +// no longer consents is asked a CatchUp instead. The request then checks that the Kustomization +// applies the GitProvider's repository and branch before it stamps anything. +// +// A failed request is retried only briefly: for about the shortest quiet period of its GitTargets, +// or not at all when the failure is permanent. A retry is the worker's decision replayed without +// its quiet check, and by the time a long retry lands, live edits may be waiting in an open window +// that the apply would revert. After that it is handed back to the branch worker +// (git.ApplyHandBack), which owes it again and sends it once the branch is quiet. type Dispatcher struct { reader client.Reader sourceClients SourceClients @@ -116,6 +130,9 @@ type Dispatcher struct { Backoff Backoff // AttemptTimeout bounds one delivery attempt. Set before Start. AttemptTimeout time.Duration + // HandBack takes back the requests that could not be delivered. Set before Start; nil gives + // them up (the CLI and tests). + HandBack git.ApplyHandBack // Now is the clock requestedAt stamps and the ledger read. Retries are scheduled on real time. Now func() time.Time @@ -251,52 +268,92 @@ func (k applierKey) ref() types.NamespacedName { // job is one applier's share of one or more requests. type job struct { - key applierKey - commit string - reason v1alpha3.ApplyReason - targets map[string]*v1alpha3.GitTarget - failures int - due time.Time + key applierKey + commit string + targets map[string]jobTarget + // handBacks is how many times these requests were handed back before (git.ApplyRequest). + handBacks int + // failures counts failed attempts, and firstFailure is when the first one settled. + failures int + firstFailure time.Time + due time.Time } -// absorb takes over an older job this one supersedes: its targets, and its reason when this one -// would weaken it. -func (j *job) absorb(older *job) { - j.reason = supersedingReason(older.reason, j.reason) - for key, target := range older.targets { - if _, ok := j.targets[key]; !ok { - j.targets[key] = target +// jobTarget is one GitTarget a job asks for, with its own reason. +type jobTarget struct { + ref itypes.ResourceReference + reason v1alpha3.ApplyReason + // quietPeriod is the GitTarget's spec.applier.quietPeriod when the request was resolved, which + // bounds how long the job is retried. + quietPeriod time.Duration +} + +// reason is what the applier is asked: the strongest of the job's GitTargets' reasons. +func (j *job) reason() v1alpha3.ApplyReason { + return strongestReason(j.targets) +} + +func strongestReason(targets map[string]jobTarget) v1alpha3.ApplyReason { + var reason v1alpha3.ApplyReason + for _, target := range targets { + reason = git.StrongerApplyReason(reason, target.reason) + } + return reason +} + +// retryBudget is how long a failing job is retried before it is handed back: the shortest quiet +// period among its GitTargets. A request a quiet period old is as stale as the worker allows a +// catch-up to be, and the worker is the one that can tell whether the branch is still quiet. +func (j *job) retryBudget() time.Duration { + budget := v1alpha3.DefaultApplierQuietPeriod + for i, target := range j.sortedTargets() { + if i == 0 || target.quietPeriod < budget { + budget = target.quietPeriod } } + return budget } -// supersedingReason is the reason a newer request carries when it replaces an older one: its own, -// except that a CatchUp never replaces a reason that asks for more (see Dispatcher). -func supersedingReason(older, newer v1alpha3.ApplyReason) v1alpha3.ApplyReason { - if newer == v1alpha3.ApplyReasonCatchUp && older != v1alpha3.ApplyReasonCatchUp { - return older +// absorb takes over an older job this one supersedes: its GitTargets, each keeping the stronger of +// its two reasons (git.StrongerApplyReason), and its hand-back count. +func (j *job) absorb(older *job) { + for key, target := range older.targets { + if mine, ok := j.targets[key]; ok { + target.reason = git.StrongerApplyReason(target.reason, mine.reason) + target.quietPeriod = mine.quietPeriod + } + j.targets[key] = target } - return newer + j.handBacks = max(j.handBacks, older.handBacks) } // sortedTargets returns the job's GitTargets in a stable order. -func (j *job) sortedTargets() []*v1alpha3.GitTarget { +func (j *job) sortedTargets() []jobTarget { keys := make([]string, 0, len(j.targets)) for key := range j.targets { keys = append(keys, key) } sort.Strings(keys) - out := make([]*v1alpha3.GitTarget, 0, len(keys)) + out := make([]jobTarget, 0, len(keys)) for _, key := range keys { out = append(out, j.targets[key]) } return out } +// eligibleTarget is a GitTarget an attempt found may still ask the applier, as read for it. +type eligibleTarget struct { + target *v1alpha3.GitTarget + reason v1alpha3.ApplyReason +} + +// result is what one attempt did. eligible is empty when no GitTarget may still ask the applier, +// in which case nothing was sent and err is nil. type result struct { - job *job - outcome Outcome - err error + job *job + eligible []eligibleTarget + outcome Outcome + err error } // Start runs the delivery loop until ctx ends. Attempts still running then are cancelled with it. @@ -348,7 +405,8 @@ func (d *Dispatcher) takeInbox(ctx context.Context) { // resolve turns a request into one job per applier its targets name. A target that is gone, or // declares no applier, is skipped: the worker decided on the targets it knew, and either can have -// changed since. The read is the manager's cache, so it costs the loop no round trip. +// changed since. Everything else about a target is checked at every attempt (recheck). The read is +// the manager's cache, so it costs the loop no round trip. func (d *Dispatcher) resolve(ctx context.Context, req git.ApplyRequest) []*job { jobs := map[applierKey]*job{} var order []applierKey @@ -364,24 +422,16 @@ func (d *Dispatcher) resolve(ctx context.Context, req git.ApplyRequest) []*job { if target.Spec.Applier == nil { continue } - kustomization := target.Spec.Applier.KustomizationRef - namespace := kustomization.Namespace - if namespace == "" { - namespace = target.Namespace - } - key := applierKey{ - branch: req.Branch, - cluster: target.SourceCluster(), - namespace: namespace, - name: kustomization.Name, - } + key := applierKeyFor(req.Branch, target) j := jobs[key] if j == nil { - j = &job{key: key, commit: req.Commit, reason: req.Reason, targets: map[string]*v1alpha3.GitTarget{}} + j = &job{key: key, commit: req.Commit, handBacks: req.HandBacks, targets: map[string]jobTarget{}} jobs[key] = j order = append(order, key) } - j.targets[itypes.NewResourceReference(target.Name, target.Namespace).Key()] = target + j.targets[ref.Key()] = jobTarget{ + ref: ref, reason: req.Reason, quietPeriod: target.Spec.Applier.EffectiveQuietPeriod(), + } } out := make([]*job, 0, len(order)) for _, key := range order { @@ -390,11 +440,22 @@ func (d *Dispatcher) resolve(ctx context.Context, req git.ApplyRequest) []*job { return out } +// applierKeyFor is the applier a GitTarget names, for branch. Its kustomizationRef namespace +// defaults to the GitTarget's own. +func applierKeyFor(branch git.BranchKey, target *v1alpha3.GitTarget) applierKey { + kustomization := target.Spec.Applier.KustomizationRef + namespace := kustomization.Namespace + if namespace == "" { + namespace = target.Namespace + } + return applierKey{branch: branch, cluster: target.SourceCluster(), namespace: namespace, name: kustomization.Name} +} + // enqueue queues a job, superseding the applier's job that is waiting to be sent, if any. A job // already being attempted is not superseded: it was sent. The new job waits behind it. func (d *Dispatcher) enqueue(ctx context.Context, j *job) { if older, ok := d.pending[j.key]; ok { - d.count(ctx, older.reason, outcomeSuperseded) + d.count(ctx, older.reason(), outcomeSuperseded) j.absorb(older) } d.pending[j.key] = j @@ -412,9 +473,9 @@ func (d *Dispatcher) startDue(ctx context.Context, attempts *sync.WaitGroup) { attempts.Add(1) go func() { defer attempts.Done() - outcome, err := d.attempt(ctx, j) + res := d.attempt(ctx, j) select { - case d.results <- result{job: j, outcome: outcome, err: err}: + case d.results <- res: case <-ctx.Done(): } }() @@ -436,89 +497,216 @@ func (d *Dispatcher) nextDue() (time.Time, bool) { return next, found } -// attempt makes one delivery attempt. -func (d *Dispatcher) attempt(ctx context.Context, j *job) (Outcome, error) { +// attempt makes one delivery attempt. It runs on its own goroutine and reads j without writing +// it: the loop leaves a job alone while it is in flight. +func (d *Dispatcher) attempt(ctx context.Context, j *job) result { ctx, cancel := context.WithTimeout(ctx, d.AttemptTimeout) defer cancel() + eligible, want, err := d.recheck(ctx, j) + if err != nil || len(eligible) == 0 { + return result{job: j, err: err} + } + reason := strongestEligible(eligible) sourceClient, err := d.sourceClients(ctx, j.key.cluster) if err != nil { - return "", fmt.Errorf("source cluster %q: %w", j.key.cluster, err) + return result{job: j, eligible: eligible, err: fmt.Errorf("source cluster %q: %w", j.key.cluster, err)} } flux := NewFlux(sourceClient) flux.Now = d.Now - return flux.Request(ctx, j.key.ref(), j.commit, j.reason) + outcome, err := flux.Request(ctx, j.key.ref(), want, j.commit, reason) + return result{job: j, eligible: eligible, outcome: outcome, err: err} +} + +// recheck reads each of the job's GitTargets again and returns those that may still ask its +// applier, each with the reason it may ask for, and the binding the applier must apply. A read +// that fails is an error, so the attempt is retried rather than a GitTarget dropped on a blip. +func (d *Dispatcher) recheck(ctx context.Context, j *job) ([]eligibleTarget, Binding, error) { + provider := &v1alpha3.GitProvider{} + providerKey := client.ObjectKey{Namespace: j.key.branch.RepoNamespace, Name: j.key.branch.RepoName} + if err := d.reader.Get(ctx, providerKey, provider); err != nil { + if apierrors.IsNotFound(err) { + return nil, Binding{}, nil + } + return nil, Binding{}, fmt.Errorf("read GitProvider %s: %w", providerKey, err) + } + want := Binding{RepositoryURL: provider.Spec.URL, Branch: j.key.branch.Branch} + + var eligible []eligibleTarget + for _, candidate := range j.sortedTargets() { + target, reason, err := d.recheckTarget(ctx, j.key, candidate) + if err != nil { + return nil, Binding{}, err + } + if target != nil { + eligible = append(eligible, eligibleTarget{target: target, reason: reason}) + } + } + return eligible, want, nil } -// settle records the result of an attempt: delivered, superseded by a request that arrived while -// it ran, retried later, or given up on. +// recheckTarget reads one GitTarget again and returns it, with the reason it may ask for, when it +// may still ask the applier, or nil when it may not. +// +// A ClusterProvider's source client is shared by every GitTarget that declared the cluster, so a +// client being available is no permission to use it: a GitTarget its ClusterProvider does not +// admit would otherwise drive the applier on the strength of another GitTarget's admission. +func (d *Dispatcher) recheckTarget( + ctx context.Context, key applierKey, candidate jobTarget, +) (*v1alpha3.GitTarget, v1alpha3.ApplyReason, error) { + log := d.log.WithValues("gitTarget", candidate.ref.String(), "applier", key.ref().String()) + target := &v1alpha3.GitTarget{} + objKey := client.ObjectKey{Namespace: candidate.ref.Namespace, Name: candidate.ref.Name} + if err := d.reader.Get(ctx, objKey, target); err != nil { + if apierrors.IsNotFound(err) { + return nil, "", nil + } + return nil, "", fmt.Errorf("read GitTarget %s: %w", candidate.ref, err) + } + if !target.DeletionTimestamp.IsZero() || target.Spec.Applier == nil || + target.Namespace != key.branch.RepoNamespace || target.Spec.GitProviderRef.Name != key.branch.RepoName || + target.Spec.Branch != key.branch.Branch || applierKeyFor(key.branch, target) != key { + log.V(1).Info("GitTarget no longer names this applier on this branch; not asking for it") + return nil, "", nil + } + decision, err := authz.GitTargetAdmitted(ctx, d.reader, target) + if err != nil { + return nil, "", fmt.Errorf("check GitTarget %s against its ClusterProvider: %w", candidate.ref, err) + } + if !decision.Allowed { + log.Info("GitTarget is not admitted by its ClusterProvider; not asking its applier", + "reason", decision.Reason) + return nil, "", nil + } + reason := candidate.reason + if reason == v1alpha3.ApplyReasonRefusal && !git.ConsentsToReapply(target) { + // Consent was withdrawn after the worker decided. The target's applier may still need to + // catch up, which needs no consent. + reason = v1alpha3.ApplyReasonCatchUp + } + return target, reason, nil +} + +// settle records the result of an attempt: delivered, dropped, superseded by a request that +// arrived while it ran, retried later, or handed back. func (d *Dispatcher) settle(ctx context.Context, res result) { j := res.job delete(d.inflight, j.key) log := d.log.WithValues("applier", j.key.ref().String(), "cluster", j.key.cluster, - "branch", j.key.branch.String(), "reason", j.reason, "commit", j.commit) + "branch", j.key.branch.String(), "reason", j.reason(), "commit", j.commit) - if res.err == nil { - d.count(ctx, j.reason, outcomeSent) + switch { + case res.err == nil && len(res.eligible) == 0: + d.count(ctx, j.reason(), outcomeDropped) + log.Info("no GitTarget may still ask this applier; dropping the request") + return + case res.err == nil: + reason := strongestEligible(res.eligible) + d.count(ctx, reason, outcomeSent) if res.outcome == OutcomeCurrent { log.V(1).Info("applier already holds the commit; nothing to request") return } log.Info("apply requested", "outcome", res.outcome) - d.delivered(j) + d.delivered(j, res.eligible) return } j.failures++ + now := time.Now() + if j.firstFailure.IsZero() { + j.firstFailure = now + } if newer, ok := d.pending[j.key]; ok { // A newer request is waiting, so this one is not retried: the newer one is what to send. - d.count(ctx, j.reason, outcomeSuperseded) + d.count(ctx, j.reason(), outcomeSuperseded) newer.absorb(j) log.Info("apply request failed and is superseded by a newer one", "error", res.err.Error()) return } - if isPermanent(res.err) || j.failures >= d.Backoff.Attempts { - d.count(ctx, j.reason, outcomeFailed) - log.Error(res.err, "apply request failed; giving up", "attempts", j.failures) - d.failed(j, res.err) + if isPermanent(res.err) || now.Sub(j.firstFailure) >= j.retryBudget() { + d.giveBack(ctx, j, res.err, log) return } delay := d.Backoff.delay(j.failures) - j.due = time.Now().Add(delay) + j.due = now.Add(delay) d.pending[j.key] = j log.Info("apply request failed; retrying", "attempt", j.failures, "retryIn", delay, "error", res.err.Error()) } +func strongestEligible(eligible []eligibleTarget) v1alpha3.ApplyReason { + var reason v1alpha3.ApplyReason + for _, target := range eligible { + reason = git.StrongerApplyReason(reason, target.reason) + } + return reason +} + +// giveBack stops retrying a job: it reports the failure, if this is the first time these requests +// failed, and hands every GitTarget back to the branch worker with its own reason. Every one is +// handed back, not only those the last attempt found eligible: the worker owes them again, and +// the attempt that sends them checks them again. +func (d *Dispatcher) giveBack(ctx context.Context, j *job, err error, log logr.Logger) { + d.count(ctx, j.reason(), outcomeFailed) + if j.handBacks == 0 { + d.failed(ctx, j, err) + } + if d.HandBack == nil { + log.Error(err, "apply request failed; giving up", "attempts", j.failures) + return + } + log.Error(err, "apply request failed; handing it back to the branch worker", + "attempts", j.failures, "handBacks", j.handBacks+1) + byReason := map[v1alpha3.ApplyReason][]itypes.ResourceReference{} + var reasons []v1alpha3.ApplyReason + for _, target := range j.sortedTargets() { + if _, seen := byReason[target.reason]; !seen { + reasons = append(reasons, target.reason) + } + byReason[target.reason] = append(byReason[target.reason], target.ref) + } + for _, reason := range reasons { + d.HandBack.HandBackApply(git.ApplyRequest{ + Branch: j.key.branch, Reason: reason, Commit: j.commit, + Targets: byReason[reason], HandBacks: j.handBacks + 1, + }) + } +} + // delivered records a delivered request in the ledger and as an ApplyRequested Event on every -// GitTarget it was for. -func (d *Dispatcher) delivered(j *job) { +// GitTarget it was for, each with its own reason. +func (d *Dispatcher) delivered(j *job, eligible []eligibleTarget) { at := metav1.NewTime(d.Now()) - status := v1alpha3.GitTargetApplierStatus{RequestedCommit: j.commit, RequestedAt: &at, Reason: j.reason} - targets := j.sortedTargets() - d.ledgerMu.Lock() - for _, target := range targets { - d.ledger[itypes.NewResourceReference(target.Name, target.Namespace).Key()] = *status.DeepCopy() + for _, e := range eligible { + d.ledger[itypes.NewResourceReference(e.target.Name, e.target.Namespace).Key()] = + v1alpha3.GitTargetApplierStatus{RequestedCommit: j.commit, RequestedAt: at.DeepCopy(), Reason: e.reason} } d.ledgerMu.Unlock() if d.recorder == nil { return } - for _, target := range targets { - d.recorder.Eventf(target, corev1.EventTypeNormal, EventReasonApplyRequested, - "asked %s to apply %s (%s)", j.key.ref(), describeCommit(j.commit), j.reason) + for _, e := range eligible { + d.recorder.Eventf(e.target, corev1.EventTypeNormal, EventReasonApplyRequested, + "asked %s to apply %s (%s)", j.key.ref(), describeCommit(j.commit), e.reason) } } -// failed records a request given up on as an ApplyRequestFailed Event on every GitTarget it was for. -func (d *Dispatcher) failed(j *job, err error) { +// failed records a request that could not be delivered as an ApplyRequestFailed Event on every +// GitTarget it was for. The Event needs the object, so a GitTarget that cannot be read is skipped. +func (d *Dispatcher) failed(ctx context.Context, j *job, err error) { if d.recorder == nil { return } - for _, target := range j.sortedTargets() { + for _, candidate := range j.sortedTargets() { + target := &v1alpha3.GitTarget{} + objKey := client.ObjectKey{Namespace: candidate.ref.Namespace, Name: candidate.ref.Name} + if d.reader.Get(ctx, objKey, target) != nil { + continue + } d.recorder.Eventf(target, corev1.EventTypeWarning, EventReasonApplyRequestFailed, - "could not ask %s to apply %s (%s) after %d attempts: %v", - j.key.ref(), describeCommit(j.commit), j.reason, j.failures, err) + "could not ask %s to apply %s (%s) after %d attempts, and will ask again once the branch is quiet: %v", + j.key.ref(), describeCommit(j.commit), candidate.reason, j.failures, err) } } diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index 417b7999..38f1037b 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -37,11 +37,15 @@ const ( var testBranchKey = git.BranchKey{RepoNamespace: targetNS, RepoName: "platform", Branch: testBranch} -// gitTarget builds a GitTarget; a nil kustomizationRef declares no applier. +// gitTarget builds a GitTarget on testBranchKey; a nil kustomizationRef declares no applier. func gitTarget(name string, kustomizationRef *fluxmeta.NamespacedObjectReference, cluster string) *v1alpha3.GitTarget { target := &v1alpha3.GitTarget{ ObjectMeta: metav1.ObjectMeta{Namespace: targetNS, Name: name}, - Spec: v1alpha3.GitTargetSpec{Branch: testBranch, Path: name}, + Spec: v1alpha3.GitTargetSpec{ + GitProviderRef: fluxmeta.LocalObjectReference{Name: testBranchKey.RepoName}, + Branch: testBranch, + Path: name, + }, } if kustomizationRef != nil { target.Spec.Applier = &v1alpha3.GitTargetApplier{KustomizationRef: *kustomizationRef} @@ -52,6 +56,39 @@ func gitTarget(name string, kustomizationRef *fluxmeta.NamespacedObjectReference return target } +// reapplying makes a GitTarget consent to a Refusal request. +func reapplying(target *v1alpha3.GitTarget) *v1alpha3.GitTarget { + target.Spec.OnRefusal = v1alpha3.RefusalActionReapply + return target +} + +// quietFor sets a GitTarget's quiet period, which is also how long its requests are retried. +func quietFor(target *v1alpha3.GitTarget, quietPeriod time.Duration) *v1alpha3.GitTarget { + target.Spec.Applier.QuietPeriod = &metav1.Duration{Duration: quietPeriod} + return target +} + +// clusterProvider admits the GitTargets in the named namespaces. +func clusterProvider(name string, admits ...string) *v1alpha3.ClusterProvider { + return &v1alpha3.ClusterProvider{ + ObjectMeta: metav1.ObjectMeta{Name: name}, + Spec: v1alpha3.ClusterProviderSpec{AccessFrom: &v1alpha3.NamespaceMatcher{Names: admits}}, + } +} + +// controlPlane is what every dispatcher test reads besides its GitTargets: the GitProvider of +// testBranchKey, and the ClusterProviders the tests name, admitting targetNS. +func controlPlane() []client.Object { + return []client.Object{ + &v1alpha3.GitProvider{ + ObjectMeta: metav1.ObjectMeta{Namespace: testBranchKey.RepoNamespace, Name: testBranchKey.RepoName}, + Spec: v1alpha3.GitProviderSpec{URL: testRepoURL}, + }, + clusterProvider(v1alpha3.DefaultClusterProviderName, targetNS), + clusterProvider("prod-eu-1", targetNS), + } +} + func ksRef(namespace, name string) *fluxmeta.NamespacedObjectReference { return &fluxmeta.NamespacedObjectReference{Namespace: namespace, Name: name} } @@ -87,9 +124,29 @@ func (s *sourceStub) callCount() int { type dispatcherFixture struct { dispatcher *Dispatcher + control client.Client source *sourceStub recorder *record.FakeRecorder reader *sdkmetric.ManualReader + handBacks *handBackRecorder +} + +// handBackRecorder records the requests the dispatcher hands back. +type handBackRecorder struct { + mu sync.Mutex + requests []git.ApplyRequest +} + +func (h *handBackRecorder) HandBackApply(req git.ApplyRequest) { + h.mu.Lock() + defer h.mu.Unlock() + h.requests = append(h.requests, req) +} + +func (h *handBackRecorder) got() []git.ApplyRequest { + h.mu.Lock() + defer h.mu.Unlock() + return append([]git.ApplyRequest(nil), h.requests...) } func newDispatcherFixture(t *testing.T, targets []client.Object, sourceObjs ...runtime.Object) *dispatcherFixture { @@ -100,14 +157,19 @@ func newDispatcherFixture(t *testing.T, targets []client.Object, sourceObjs ...r scheme := runtime.NewScheme() require.NoError(t, clientgoscheme.AddToScheme(scheme)) require.NoError(t, v1alpha3.AddToScheme(scheme)) - controlClient := fake.NewClientBuilder().WithScheme(scheme).WithObjects(targets...).Build() + controlClient := fake.NewClientBuilder().WithScheme(scheme). + WithObjects(append(controlPlane(), targets...)...).Build() source := &sourceStub{client: newSourceClient(sourceObjs...)} recorder := record.NewFakeRecorder(64) + handBacks := &handBackRecorder{} d := NewDispatcher(controlClient, source.get, recorder, logr.Discard()) - d.Backoff = Backoff{Initial: time.Millisecond, Max: 4 * time.Millisecond, Attempts: 4} + d.Backoff = Backoff{Initial: time.Millisecond, Max: 4 * time.Millisecond} d.Now = func() time.Time { return stampTime } - return &dispatcherFixture{dispatcher: d, source: source, recorder: recorder, reader: reader} + d.HandBack = handBacks + return &dispatcherFixture{ + dispatcher: d, control: controlClient, source: source, recorder: recorder, reader: reader, handBacks: handBacks, + } } // start runs the dispatcher until the test ends. @@ -217,7 +279,7 @@ func TestDispatcher_AsksEachApplierOnce(t *testing.T) { // delivery counts. func TestDispatcher_RetriesThenDelivers(t *testing.T) { f := newDispatcherFixture(t, - []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + []client.Object{reapplying(gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""))}, kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), ) @@ -239,32 +301,50 @@ func TestDispatcher_RetriesThenDelivers(t *testing.T) { assert.Equal(t, v1alpha3.ApplyReasonRefusal, got.Reason) } -// TestDispatcher_GivesUp: a request that keeps failing is given up on after its last attempt, with a -// Warning Event, and is not recorded as made. -func TestDispatcher_GivesUp(t *testing.T) { - f := newDispatcherFixture(t, []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}) +// TestDispatcher_HandsBackAfterItsRetryBudget is the review's finding 4: a request that kept +// failing was given up on, while the worker had already cleared its debt, so nothing asked again +// until an unrelated commit. It is now retried for about one quiet period and handed back to the +// worker, and the failure is reported once, not again when the worker's next request fails too. +func TestDispatcher_HandsBackAfterItsRetryBudget(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{ + quietFor(gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""), 20*time.Millisecond), + gitTarget("b", ksRef(fluxSystemNS, "checkout"), ""), + }) f.source.failFirst = -1 f.start(t) f.dispatcher.RequestApply(git.ApplyRequest{ Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, + Targets: []itypes.ResourceReference{ref("a"), ref("b")}, }) f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) - assert.Equal(t, f.dispatcher.Backoff.Attempts, f.source.callCount()) + assert.Greater(t, f.source.callCount(), 1, "retried within the budget before handing back") + require.Eventually(t, func() bool { return len(f.handBacks.got()) == 1 }, 5*time.Second, time.Millisecond) + assert.Equal(t, git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a"), ref("b")}, HandBacks: 1, + }, f.handBacks.got()[0], "every GitTarget goes back, with its reason and one more hand-back") _, ok := f.dispatcher.LastRequest(ref("a")) - assert.False(t, ok) + assert.False(t, ok, "nothing was delivered") events := f.events() - require.Len(t, events, 1) - const want = "Warning ApplyRequestFailed could not ask flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)" + - " after 4 attempts" + require.Len(t, events, 2, "one Warning per GitTarget") + const want = "Warning ApplyRequestFailed could not ask flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)" assert.True(t, strings.HasPrefix(events[0], want), events[0]) + + // The worker sends it again, and it fails again. + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a"), ref("b")}, HandBacks: 1, + }) + require.Eventually(t, func() bool { return len(f.handBacks.got()) == 2 }, 5*time.Second, time.Millisecond) + assert.Equal(t, 2, f.handBacks.got()[1].HandBacks) + assert.Empty(t, f.events(), "a failure already reported is not reported again") } -// TestDispatcher_PermanentFailureIsNotRetried: a Kustomization a request cannot work with fails -// once rather than retrying to the limit. -func TestDispatcher_PermanentFailureIsNotRetried(t *testing.T) { +// TestDispatcher_PermanentFailureIsHandedBackAtOnce: a Kustomization a request cannot work with is +// not retried; the worker asks again later, in case it was fixed. +func TestDispatcher_PermanentFailureIsHandedBackAtOnce(t *testing.T) { f := newDispatcherFixture(t, []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, kustomizationObj(fluxSystemNS, "checkout", map[string]any{"kind": "Bucket", "name": "platform"}), @@ -277,6 +357,151 @@ func TestDispatcher_PermanentFailureIsNotRetried(t *testing.T) { }) f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) assert.Equal(t, 1, f.source.callCount()) + require.Eventually(t, func() bool { return len(f.handBacks.got()) == 1 }, 5*time.Second, time.Millisecond) +} + +// TestDispatcher_ASourceMismatchIsNeverStamped is the review's finding 2: ApplierConfigured said +// SourceMismatch, and a foreign commit stamped the GitRepository of the other branch anyway. +func TestDispatcher_ASourceMismatchIsNeverStamped(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef("release"), "release@sha1:"+olderCommit), + ) + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) + assert.Empty(t, f.patches(), "a source that follows another branch is never stamped") + assert.Equal(t, 1, f.source.callCount(), "and a mismatch is not retried") +} + +// TestDispatcher_ATargetItsClusterProviderDeniesAsksNothing is the review's finding 1: the source +// client is shared by every GitTarget that declared the cluster, so a GitTarget its ClusterProvider +// does not admit drove the applier on another GitTarget's admission. +func TestDispatcher_ATargetItsClusterProviderDeniesAsksNothing(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{ + gitTarget("admitted", ksRef(fluxSystemNS, "checkout"), ""), + gitTarget("denied", ksRef(fluxSystemNS, "restricted"), "restricted"), + clusterProvider("restricted", "some-other-namespace"), + }, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + kustomizationObj(fluxSystemNS, "restricted", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + ) + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("admitted"), ref("denied")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeSent, 1) + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeDropped, 1) + + assert.Equal(t, []string{v1alpha3.DefaultClusterProviderName}, f.source.clusters, + "the denied GitTarget's cluster client is never even asked for") + _, ok := f.dispatcher.LastRequest(ref("denied")) + assert.False(t, ok) +} + +// TestDispatcher_EveryRetryChecksAgain: a GitTarget admitted when its request was decided, and +// denied before a retry, is not asked for on the retry. The same holds for a GitTarget that moved +// to another branch. +func TestDispatcher_EveryRetryChecksAgain(t *testing.T) { + for name, change := range map[string]func(ctx context.Context, c client.Client) error{ + "its ClusterProvider stops admitting it": func(ctx context.Context, c client.Client) error { + provider := &v1alpha3.ClusterProvider{} + if err := c.Get(ctx, client.ObjectKey{Name: v1alpha3.DefaultClusterProviderName}, provider); err != nil { + return err + } + provider.Spec.AccessFrom.Names = []string{"some-other-namespace"} + return c.Update(ctx, provider) + }, + "it moves to another branch": func(ctx context.Context, c client.Client) error { + target := &v1alpha3.GitTarget{} + if err := c.Get(ctx, client.ObjectKey{Namespace: targetNS, Name: "a"}, target); err != nil { + return err + } + target.Spec.Branch = "release" + return c.Update(ctx, target) + }, + } { + t.Run(name, func(t *testing.T) { + f := newDispatcherFixture( + t, + []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj( + fluxSystemNS, + "platform", + testRepoURL, + branchRef(testBranch), + "main@sha1:"+olderCommit, + ), + ) + f.dispatcher.Backoff = Backoff{Initial: 50 * time.Millisecond, Max: 50 * time.Millisecond} + f.source.failFirst = 1 + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + require.Eventually(t, func() bool { return f.source.callCount() == 1 }, 5*time.Second, time.Millisecond) + require.NoError(t, change(context.Background(), f.control)) + + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeDropped, 1) + assert.Empty(t, f.patches()) + }) + } +} + +// TestDispatcher_ARefusalNeedsConsentAtEveryAttempt is the dispatcher's half of the review's +// finding 3: consent is per GitTarget, so a GitTarget suspended after its Refusal was decided is +// asked only a CatchUp, while a sibling on the same applier that still consents keeps its Refusal. +func TestDispatcher_ARefusalNeedsConsentAtEveryAttempt(t *testing.T) { + suspended := reapplying(gitTarget("suspended", ksRef(fluxSystemNS, "checkout"), "")) + suspended.Spec.Suspend = true + f := newDispatcherFixture(t, + []client.Object{ + suspended, + reapplying(gitTarget("consenting", ksRef(fluxSystemNS, "checkout"), "")), + reapplying(gitTarget("alone", ksRef(fluxSystemNS, "alone"), "")), + }, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + kustomizationObj(fluxSystemNS, "alone", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + ) + f.start(t) + + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("suspended"), ref("consenting")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonRefusal, outcomeSent, 1) + consenting, ok := f.dispatcher.LastRequest(ref("consenting")) + require.True(t, ok) + assert.Equal(t, v1alpha3.ApplyReasonRefusal, consenting.Reason) + suspendedStatus, ok := f.dispatcher.LastRequest(ref("suspended")) + require.True(t, ok, "the shared stamp reached its applier too") + assert.Equal(t, v1alpha3.ApplyReasonCatchUp, suspendedStatus.Reason, "recorded as what it may ask for") + + // A Refusal whose only GitTarget withdrew consent is a CatchUp, which an applier already at the + // commit satisfies: the only stamp is the one the first request made. + alone := &v1alpha3.GitTarget{} + require.NoError(t, f.control.Get(context.Background(), client.ObjectKey{Namespace: targetNS, Name: "alone"}, alone)) + alone.Spec.OnRefusal = v1alpha3.RefusalActionIgnore + require.NoError(t, f.control.Update(context.Background(), alone)) + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("alone")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonCatchUp, outcomeSent, 1) + assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches()) } // TestDispatcher_NewerRequestSupersedes: a request waiting on its retry is replaced by a newer one @@ -285,11 +510,11 @@ func TestDispatcher_PermanentFailureIsNotRetried(t *testing.T) { // commit, and a CatchUp would then stamp nothing and drop the revert. func TestDispatcher_NewerRequestSupersedes(t *testing.T) { f := newDispatcherFixture(t, - []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + []client.Object{reapplying(gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""))}, kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), ) - f.dispatcher.Backoff = Backoff{Initial: time.Hour, Max: time.Hour, Attempts: 4} + f.dispatcher.Backoff = Backoff{Initial: time.Hour, Max: time.Hour} f.source.failFirst = 1 f.start(t) @@ -369,7 +594,7 @@ func TestDispatcher_RequestApplyNeverBlocks(t *testing.T) { } func TestBackoff_Delay(t *testing.T) { - b := Backoff{Initial: 2 * time.Second, Max: time.Minute, Attempts: 8} + b := Backoff{Initial: 2 * time.Second, Max: time.Minute} for failures, want := range map[int]time.Duration{ 1: 2 * time.Second, 2: 4 * time.Second, 3: 8 * time.Second, 5: 32 * time.Second, 6: time.Minute, 20: time.Minute, } { @@ -377,16 +602,18 @@ func TestBackoff_Delay(t *testing.T) { } } -func TestSupersedingReason(t *testing.T) { +func TestStrongerApplyReason(t *testing.T) { tests := []struct{ older, newer, want v1alpha3.ApplyReason }{ {v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonRefusal}, {v1alpha3.ApplyReasonForeignCommit, v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonForeignCommit}, {v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonCatchUp}, {v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonRefusal}, {v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonForeignCommit, v1alpha3.ApplyReasonForeignCommit}, + {v1alpha3.ApplyReasonForeignCommit, v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonForeignCommit}, + {"", v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonCatchUp}, } for _, tt := range tests { - assert.Equal(t, tt.want, supersedingReason(tt.older, tt.newer), "%s then %s", tt.older, tt.newer) + assert.Equal(t, tt.want, git.StrongerApplyReason(tt.older, tt.newer), "%s then %s", tt.older, tt.newer) } } diff --git a/internal/applier/flux.go b/internal/applier/flux.go index c1e7cb91..0ceeed86 100644 --- a/internal/applier/flux.go +++ b/internal/applier/flux.go @@ -109,6 +109,14 @@ type Verdict struct { Err error } +// Binding is the repository and branch an applier must apply from: the GitProvider's spec.url and +// the GitTarget's spec.branch. Request checks it before it writes anything, not only Validate, +// because a published ApplierConfigured verdict can be older than the applier it describes. +type Binding struct { + RepositoryURL string + Branch string +} + // AccessCheck reports whether the source-cluster identity may perform attrs. type AccessCheck func(ctx context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) @@ -130,8 +138,9 @@ func NewFlux(client dynamic.Interface) *Flux { } // permanentError marks a request no retry can make succeed: the Kustomization is configured in a -// way a request cannot work with. It fails loudly once rather than retrying to the limit; a newer -// request still tries again, because the configuration may have been fixed in between. +// way a request cannot work with, or applies another repository or branch. The dispatcher hands it +// back to the branch worker at once rather than retrying it, and the worker asks again later, +// because the configuration may have been fixed in between. type permanentError struct{ err error } func (e permanentError) Error() string { return e.err.Error() } @@ -146,6 +155,11 @@ func isPermanent(err error) bool { // Request asks the Kustomization ref to apply commit, the way `flux reconcile kustomization // --with-source` does, minus the waiting: a request asks and never waits to see the apply. // +// It first checks that the Kustomization applies want, and refuses with a permanent error when it +// does not: stamping a GitRepository that follows another branch would deploy that branch and +// leave this one unapplied. The check reads the same objects the stamp is decided from, so no +// change to either can slip in between. +// // When the GitRepository's artifact does not hold commit, it stamps the GitRepository: the fetch // produces a new artifact and the new artifact wakes the Kustomization by itself, so a second stamp // would only be a second apply. When the artifact already holds it, a source stamp would wake @@ -155,6 +169,7 @@ func isPermanent(err error) bool { func (f *Flux) Request( ctx context.Context, ref types.NamespacedName, + want Binding, commit string, reason v1alpha3.ApplyReason, ) (Outcome, error) { @@ -171,6 +186,9 @@ func (f *Flux) Request( if err != nil { return "", fmt.Errorf("get %s %s: %w", kindGitRepository, sourceRef, err) } + if msg := sourceMismatch(source, sourceRef, want.RepositoryURL, want.Branch); msg != "" { + return "", permanentError{fmt.Errorf("%s %s: %s", kindKustomization, ref, msg)} + } if commit != "" && artifactHolds(source, commit) { if reason == v1alpha3.ApplyReasonCatchUp { @@ -267,13 +285,13 @@ func splitRevision(revision string) (string, string) { } // Validate answers ApplierConfigured for a Kustomization: that it exists in the source cluster and -// its kind is served there, that it applies a GitRepository fetching branch of repositoryURL (the -// GitProvider's spec.url, compared after normalization), and that this identity may patch both -// objects, since a request patches one or the other. +// its kind is served there, that it applies a GitRepository fetching want (the GitProvider's +// spec.url, compared after normalization, and the GitTarget's branch), and that this identity may +// patch both objects, since a request patches one or the other. // // It reads; it never watches. A cluster without Flux installed answers ApplierNotFound from one // failed GET and keeps no informer for kinds it does not serve. -func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, repositoryURL, branch string) Verdict { +func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, want Binding) Verdict { ks, err := f.client.Resource(kustomizationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) if err != nil { return readFailure(kindKustomization, kustomizationGVR(), ref, err) @@ -289,7 +307,7 @@ func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, repositor verdict.Message = fmt.Sprintf("%s (the sourceRef of %s %s)", verdict.Message, kindKustomization, ref) return verdict } - if msg := sourceMismatch(source, sourceRef, repositoryURL, branch); msg != "" { + if msg := sourceMismatch(source, sourceRef, want.RepositoryURL, want.Branch); msg != "" { return Verdict{Reason: ReasonSourceMismatch, Message: msg} } @@ -329,7 +347,7 @@ func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, repositor OK: true, Reason: ReasonSucceeded, Message: fmt.Sprintf("%s %s applies %s %s, which fetches branch %s", - kindKustomization, ref, kindGitRepository, sourceRef, branch), + kindKustomization, ref, kindGitRepository, sourceRef, want.Branch), } } diff --git a/internal/applier/flux_test.go b/internal/applier/flux_test.go index 55bc8771..a27fc2e5 100644 --- a/internal/applier/flux_test.go +++ b/internal/applier/flux_test.go @@ -35,6 +35,9 @@ const ( var stampTime = time.Date(2026, 10, 20, 10, 14, 3, 123456789, time.UTC) +// testBinding is the repository and branch the test appliers are expected to apply. +var testBinding = Binding{RepositoryURL: testRepoURL, Branch: testBranch} + func newSourceClient(objs ...runtime.Object) *dynamicfake.FakeDynamicClient { return dynamicfake.NewSimpleDynamicClientWithCustomListKinds(runtime.NewScheme(), map[schema.GroupVersionResource]string{ @@ -177,7 +180,7 @@ func TestFluxRequest_Outcomes(t *testing.T) { gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), tt.revision), ) got, err := pinnedFlux(client).Request(context.Background(), - types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, tt.commit, tt.reason) + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding, tt.commit, tt.reason) require.NoError(t, err) assert.Equal(t, tt.want, got) @@ -217,7 +220,7 @@ func TestFluxRequest_SourceRefNamespace(t *testing.T) { gitRepositoryObj(tt.wantNamespace, "platform", testRepoURL, branchRef(testBranch), ""), ) kustomization := types.NamespacedName{Namespace: "apps", Name: "checkout"} - got, err := pinnedFlux(client).Request(context.Background(), kustomization, testCommit, + got, err := pinnedFlux(client).Request(context.Background(), kustomization, testBinding, testCommit, v1alpha3.ApplyReasonForeignCommit) require.NoError(t, err) assert.Equal(t, OutcomeStampedSource, got) @@ -248,14 +251,42 @@ func TestFluxRequest_Failures(t *testing.T) { objs: []runtime.Object{kustomizationObj(fluxSystemNS, "checkout", nil)}, wantPermanent: true, }, + { + // The review's reproduction of finding 2: Validate said SourceMismatch, and the + // request stamped the release source anyway. + name: "source follows another branch", + objs: []runtime.Object{ + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj( + fluxSystemNS, + "platform", + testRepoURL, + branchRef("release"), + "release@sha1:"+olderCommit, + ), + }, + wantPermanent: true, + }, + { + name: "source fetches another repository", + objs: []runtime.Object{ + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", "https://github.com/acme/other", branchRef(testBranch), ""), + }, + wantPermanent: true, + }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - _, err := pinnedFlux(newSourceClient(tt.objs...)).Request(context.Background(), - types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testCommit, + client := newSourceClient(tt.objs...) + _, err := pinnedFlux(client).Request(context.Background(), + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding, testCommit, v1alpha3.ApplyReasonForeignCommit) require.Error(t, err) assert.Equal(t, tt.wantPermanent, isPermanent(err)) + for _, action := range client.Actions() { + assert.NotEqual(t, "patch", action.GetVerb(), "a failed request stamps nothing") + } }) } } @@ -414,7 +445,7 @@ func TestFluxValidate_Reasons(t *testing.T) { } got := flux.Validate(context.Background(), - types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testRepoURL, testBranch) + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding) assert.Equal(t, tt.wantReason, got.Reason, got.Message) assert.Equal(t, tt.wantReason == ReasonSucceeded, got.OK) assert.Contains(t, got.Message, tt.wantMessage) @@ -450,7 +481,8 @@ func TestFluxValidate_URLEquivalences(t *testing.T) { flux := pinnedFlux(client) flux.CanI = allowAll got := flux.Validate(context.Background(), - types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, tt.provider, testBranch) + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, + Binding{RepositoryURL: tt.provider, Branch: testBranch}) assert.Equal(t, tt.wantOK, got.OK, got.Message) }) } diff --git a/internal/controller/gittarget_applier.go b/internal/controller/gittarget_applier.go index c362abfc..29731c55 100644 --- a/internal/controller/gittarget_applier.go +++ b/internal/controller/gittarget_applier.go @@ -5,6 +5,7 @@ package controller import ( "context" "fmt" + "time" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" k8stypes "k8s.io/apimachinery/pkg/types" @@ -19,6 +20,12 @@ import ( // and may be patched. Present only while spec.applier is set. const GitTargetConditionApplierConfigured = "ApplierConfigured" +// applierCheckTimeout bounds the ApplierConfigured check: two GETs and two access reviews against +// the source cluster. Its client has no request timeout, because it also carries the watches, and +// this controller runs one reconcile worker, so a source API server that accepts the connection and +// never answers would otherwise stop every GitTarget's reconcile, not only this one's. +const applierCheckTimeout = 10 * time.Second + // ApplyRequestLedger is what the GitTarget controller reads to publish status.applier: the last // apply request delivered for a GitTarget. The applier dispatcher implements it. type ApplyRequestLedger interface { @@ -64,6 +71,8 @@ func (r *GitTargetReconciler) applierVerdict( gitProvider *configbutleraiv1alpha3.GitProvider, ) conditionValue { ref := applierReference(target) + ctx, cancel := context.WithTimeout(ctx, applierCheckTimeout) + defer cancel() if r.ApplierClients == nil || gitProvider == nil { return conditionValue{ Status: metav1.ConditionUnknown, @@ -80,7 +89,8 @@ func (r *GitTargetReconciler) applierVerdict( if newFlux == nil { newFlux = applier.NewFlux } - verdict := newFlux(client).Validate(ctx, ref, gitProvider.Spec.URL, target.Spec.Branch) + verdict := newFlux(client).Validate(ctx, ref, + applier.Binding{RepositoryURL: gitProvider.Spec.URL, Branch: target.Spec.Branch}) switch { case verdict.OK: return conditionValue{ diff --git a/internal/controller/gittarget_applier_test.go b/internal/controller/gittarget_applier_test.go index 4b2ce10c..ee542f9f 100644 --- a/internal/controller/gittarget_applier_test.go +++ b/internal/controller/gittarget_applier_test.go @@ -158,6 +158,39 @@ func TestObserveApplier_AnUnreachableSourceKeepsThePublishedVerdict(t *testing.T assert.Equal(t, metav1.ConditionTrue, value.Status, "the published verdict stands") } +// TestObserveApplier_AHangingSourceIsBounded is the review's finding 6: the source client has no +// request timeout, so a source API server that accepts the connection and never answers held the +// reconcile, and with it every GitTarget, for as long as it liked. The check now gives up, and a +// check that could not be made keeps the published verdict. +func TestObserveApplier_AHangingSourceIsBounded(t *testing.T) { + var sawDeadline bool + r := &GitTargetReconciler{ + ApplierClients: func(ctx context.Context, _ string) (dynamic.Interface, error) { + _, sawDeadline = ctx.Deadline() + return applierFakeSource(applierFluxObjects("main")...), nil + }, + newFlux: func(client dynamic.Interface) *applier.Flux { + flux := applier.NewFlux(client) + flux.CanI = func(ctx context.Context, _ authorizationv1.ResourceAttributes) (bool, error) { + deadline, ok := ctx.Deadline() + require.True(t, ok, "the access review must run under a deadline") + assert.LessOrEqual(t, time.Until(deadline), applierCheckTimeout) + return false, context.DeadlineExceeded // what a server that never answers ends in + } + return flux + }, + } + + target := applierTarget(true) + target.Status.Conditions = []metav1.Condition{{ + Type: GitTargetConditionApplierConfigured, Status: metav1.ConditionTrue, Reason: ReasonSucceeded, + }} + value, _ := r.observeApplier(context.Background(), beginStatus(nil, nil, target), target, applierProvider()) + + assert.True(t, sawDeadline, "the client lookup must run under a deadline") + assert.Equal(t, metav1.ConditionTrue, value.Status, "a timed-out check keeps the published verdict") +} + func TestObserveApplier_PublishesTheLastRequest(t *testing.T) { target := applierTarget(true) at := metav1.NewTime(time.Date(2026, 10, 20, 10, 14, 3, 0, time.UTC)) diff --git a/internal/git/apply_handback_test.go b/internal/git/apply_handback_test.go new file mode 100644 index 00000000..c3337209 --- /dev/null +++ b/internal/git/apply_handback_test.go @@ -0,0 +1,260 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +// What happens to an apply request after the worker decided it: a refusal owed while the branch +// was busy, and a request the requester could not deliver and handed back. Both wait, and both are +// checked again when they stop waiting. These are the worker's halves of the review findings 3 and +// 4 recorded in docs/design/apply-trigger-follow-ups.md §2.3 and §2.5. + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "sigs.k8s.io/controller-runtime/pkg/client" + + configv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// refusalLoop is an event loop over refusalTouchWorker's GitTarget team-a/editing, asking for +// Reapply, whose requests are recorded. +func refusalLoop(t *testing.T) (*branchWorkerEventLoop, *recordingRequester) { + t.Helper() + w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{OnRefusal: configv1alpha3.RefusalActionReapply}) + requester := &recordingRequester{} + w.applyRequester = requester + loop := newBranchWorkerEventLoop(w, time.Minute) + t.Cleanup(loop.stopTimers) + return loop, requester +} + +// updateEditing changes team-a/editing in the worker's fake client. +func updateEditing(t *testing.T, w *BranchWorker, mutate func(*configv1alpha3.GitTarget)) { + t.Helper() + target := &configv1alpha3.GitTarget{} + require.NoError(t, w.Client.Get(w.ctx, client.ObjectKey{Name: "editing", Namespace: "team-a"}, target)) + mutate(target) + require.NoError(t, w.Client.Update(w.ctx, target)) +} + +// TestServiceRefusals_AnOwedRefusalAsksForConsentAgain: a refusal owed while the branch was busy +// waited, behind an open window or a publication outage, for as long as those lasted, and was +// sent to a target suspended or set back to Ignore in the meantime. +func TestServiceRefusals_AnOwedRefusalAsksForConsentAgain(t *testing.T) { + for name, withdraw := range map[string]func(*configv1alpha3.GitTarget){ + "suspended": func(g *configv1alpha3.GitTarget) { g.Spec.Suspend = true }, + "set to Ignore": func(g *configv1alpha3.GitTarget) { g.Spec.OnRefusal = configv1alpha3.RefusalActionIgnore }, + } { + t.Run(name, func(t *testing.T) { + loop, requester := refusalLoop(t) + loop.openWindow = &openWindow{} + loop.oweReapply(editingKey(), "a base-owned field", "observation-1") + loop.serviceAppliers() + + updateEditing(t, loop.w, withdraw) + loop.openWindow = nil + loop.serviceAppliers() + + assert.Zero(t, requester.refusals()) + assert.Empty(t, loop.reapplyOwed, "the obligation is spent, not kept for later") + }) + } +} + +// TestRefusalRecovered_CancelsWhatIsOwedForThatCollectionOnly: a refusal whose collection recovered +// while its request waited for a quiet branch has nothing left to revert. A sibling collection still +// refused keeps its own; the whole-target evaluation speaks for every collection. +func TestRefusalRecovered_CancelsWhatIsOwedForThatCollectionOnly(t *testing.T) { + configMaps := refusalKeyFor(editingRef(), "configmaps") + + loop, requester := refusalLoop(t) + loop.openWindow = &openWindow{} + loop.oweReapply(editingKey(), "a base-owned field", "observation-1") + loop.oweReapply(configMaps, "another base-owned field", "observation-2") + + loop.refusalRecovered(editingRef(), editingKey().collection) + assert.Equal(t, map[refusalKey]string{configMaps: "another base-owned field"}, loop.reapplyOwed) + + loop.openWindow = nil + loop.serviceAppliers() + assert.Equal(t, 1, requester.refusals(), "the configmaps refusal still stands") + + loop2, requester2 := refusalLoop(t) + loop2.openWindow = &openWindow{} + loop2.oweReapply(editingKey(), "a base-owned field", "observation-1") + loop2.oweReapply(configMaps, "another base-owned field", "observation-2") + loop2.refusalRecovered(editingRef(), itypes.CollectionKey{}) + loop2.openWindow = nil + loop2.serviceAppliers() + assert.Zero(t, requester2.refusals()) +} + +// TestHandBack_IsSentAgainOnlyWhenDueAndOnAQuietBranch: a handed-back request waits out its +// backoff, then waits for a quiet branch like any other, and carries its hand-back count so the +// requester does not report the same failure twice. +func TestHandBack_IsSentAgainOnlyWhenDueAndOnAQuietBranch(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true), applierTestTarget("bravo", true)) + owesForeign(loop.w) + loop.serviceAppliers() + require.Len(t, requester.taken(), 1, "the ForeignCommit, which then failed") + + alpha := itypes.NewResourceReference("alpha", "team-a") + loop.w.debt.handBack(ApplyRequest{ + Reason: configv1alpha3.ApplyReasonForeignCommit, Commit: "bbb", + Targets: []itypes.ResourceReference{alpha}, HandBacks: 1, + }, time.Now()) + + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "not before its backoff") + require.NotNil(t, loop.applierTimer, "the backoff is armed") + + loop.w.debt.returnedDue = time.Now().Add(-time.Second) + loop.openWindow = &openWindow{} + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "not while the branch is busy") + + loop.openWindow = nil + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, ApplyRequest{ + Branch: BranchKey{RepoNamespace: "team-a", RepoName: "test-provider", Branch: "main"}, + Reason: configv1alpha3.ApplyReasonForeignCommit, Commit: "bbb", + Targets: []itypes.ResourceReference{alpha}, HandBacks: 1, + }, requests[0], "only the target handed back, at the tip, with its count") + assert.Nil(t, loop.applierTimer) + assert.Empty(t, loop.handBackCount, "the count went with the request") +} + +// TestHandBack_ARefusalIsOwedAgainOnlyWhileItStands: a Refusal handed back is owed again while the +// refusal has not recovered, and only with consent. One that recovered while its request was +// failing, and a handed-back CatchUp, are owed a catch-up, which waits for the quiet period. +func TestHandBack_ARefusalIsOwedAgainOnlyWhileItStands(t *testing.T) { + handBack := func(loop *branchWorkerEventLoop, reason configv1alpha3.ApplyReason) { + loop.w.debt.handBack(ApplyRequest{ + Reason: reason, Commit: "aaa", Targets: []itypes.ResourceReference{editingRef()}, HandBacks: 2, + }, time.Now()) + loop.w.debt.returnedDue = time.Time{} + } + + t.Run("standing", func(t *testing.T) { + loop, requester := refusalLoop(t) + loop.w.recordRefusalObservation(editingKey(), "observation-1") + handBack(loop, configv1alpha3.ApplyReasonRefusal) + + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, configv1alpha3.ApplyReasonRefusal, requests[0].Reason) + assert.Equal(t, 2, requests[0].HandBacks) + }) + + t.Run("standing, but consent withdrawn", func(t *testing.T) { + loop, requester := refusalLoop(t) + loop.w.recordRefusalObservation(editingKey(), "observation-1") + updateEditing(t, loop.w, func(g *configv1alpha3.GitTarget) { g.Spec.Suspend = true }) + handBack(loop, configv1alpha3.ApplyReasonRefusal) + + loop.serviceAppliers() + assert.Empty(t, requester.taken()) + }) + + for _, tc := range []struct { + name string + reason configv1alpha3.ApplyReason + }{ + {"recovered", configv1alpha3.ApplyReasonRefusal}, + {"a catch-up", configv1alpha3.ApplyReasonCatchUp}, + } { + t.Run(tc.name, func(t *testing.T) { + loop, requester := refusalLoop(t) + handBack(loop, tc.reason) + + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "a catch-up waits for the quiet period") + assert.Contains(t, loop.catchUpOwed, editingRef()) + + loop.quietSince = time.Now().Add(-time.Hour) + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, configv1alpha3.ApplyReasonCatchUp, requests[0].Reason) + assert.Equal(t, 2, requests[0].HandBacks) + }) + } +} + +// TestHandBack_ATargetThatNoLongerNamesAnApplierIsOwedNothing. +func TestHandBack_ATargetThatNoLongerNamesAnApplierIsOwedNothing(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("plain", false)) + loop.w.debt.handBack(ApplyRequest{ + Reason: configv1alpha3.ApplyReasonForeignCommit, Commit: "aaa", + Targets: []itypes.ResourceReference{itypes.NewResourceReference("plain", "team-a")}, HandBacks: 1, + }, time.Now()) + loop.w.debt.returnedDue = time.Time{} + + loop.serviceAppliers() + + assert.Empty(t, requester.taken()) + assert.Empty(t, loop.handBackCount) + assert.Nil(t, loop.applierTimer) +} + +// TestHandBackApply_RecordsTheDebtAndWakesTheLoop: the record is the debt; the wake only makes an +// idle loop look at it. +func TestHandBackApply_RecordsTheDebtAndWakesTheLoop(t *testing.T) { + w := newMetricsTestWorker() + req := ApplyRequest{ + Reason: configv1alpha3.ApplyReasonCatchUp, Commit: "aaa", + Targets: []itypes.ResourceReference{editingRef()}, HandBacks: 1, + } + w.handBackApply(req) + + require.Len(t, w.eventQueue, 1) + assert.True(t, (<-w.eventQueue).ApplyWake) + assert.Equal(t, map[itypes.ResourceReference]configv1alpha3.ApplyReason{ + editingRef(): configv1alpha3.ApplyReasonCatchUp, + }, w.debt.returned) + + for range cap(w.eventQueue) { + w.eventQueue <- WorkItem{} + } + req.Reason = configv1alpha3.ApplyReasonRefusal + w.handBackApply(req) + assert.Equal(t, configv1alpha3.ApplyReasonRefusal, w.debt.returned[editingRef()], + "a full queue drops the wake, never the request, and the stronger reason wins") +} + +func TestHandBackDelay_DoublesToACap(t *testing.T) { + for handBacks, want := range map[int]time.Duration{ + 0: 15 * time.Second, 1: 15 * time.Second, 2: 30 * time.Second, 3: time.Minute, + 5: 4 * time.Minute, 6: 5 * time.Minute, 50: 5 * time.Minute, + } { + assert.Equal(t, want, handBackDelay(handBacks), "after %d hand-backs", handBacks) + } +} + +func TestApplyDebt_HandBackWaitsForTheLatestBackoff(t *testing.T) { + var debt applyDebt + now := time.Now() + target := itypes.NewResourceReference("alpha", "team-a") + debt.handBack(ApplyRequest{Reason: configv1alpha3.ApplyReasonCatchUp, Targets: []itypes.ResourceReference{target}, + HandBacks: 3}, now) + debt.handBack(ApplyRequest{Reason: configv1alpha3.ApplyReasonCatchUp, Targets: []itypes.ResourceReference{target}, + HandBacks: 1}, now) + + returned, count, due := debt.takeReturned(now.Add(30 * time.Second)) + assert.Nil(t, returned) + assert.Zero(t, count) + assert.Equal(t, now.Add(time.Minute), due, "the longer backoff stands") + + returned, count, _ = debt.takeReturned(now.Add(time.Minute)) + assert.Len(t, returned, 1) + assert.Equal(t, 3, count) + returned, _, due = debt.takeReturned(now.Add(time.Hour)) + assert.Nil(t, returned) + assert.True(t, due.IsZero(), "taken once") +} diff --git a/internal/git/apply_request.go b/internal/git/apply_request.go index 4f9943dc..69638ff0 100644 --- a/internal/git/apply_request.go +++ b/internal/git/apply_request.go @@ -3,6 +3,9 @@ package git import ( + "slices" + "time" + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" itypes "github.com/ConfigButler/gitops-reverser/internal/types" ) @@ -21,6 +24,25 @@ type ApplyRequest struct { // Targets are the GitTargets whose appliers are asked. Several may name one applier; the // requester asks it once. Targets []itypes.ResourceReference + // HandBacks is how many times a request for these targets has already been handed back + // undelivered (ApplyHandBack). Zero for a request the worker decided fresh. The worker's + // backoff grows with it, and the requester reports a failure only for a request at zero, so a + // standing failure is reported once rather than on every retry. + HandBacks int +} + +// StrongerApplyReason is the reason a target owed two requests is owed. A CatchUp asks for the +// least: it stamps nothing when the artifact already holds the commit, where the others stamp the +// Kustomization, so it never replaces either. A ForeignCommit outranks a Refusal because it needs +// no consent. The empty reason ranks below all three. +func StrongerApplyReason(a, b v1alpha3.ApplyReason) v1alpha3.ApplyReason { + weakestFirst := []v1alpha3.ApplyReason{ + "", v1alpha3.ApplyReasonCatchUp, v1alpha3.ApplyReasonRefusal, v1alpha3.ApplyReasonForeignCommit, + } + if slices.Index(weakestFirst, b) > slices.Index(weakestFirst, a) { + return b + } + return a } // ApplyRequester delivers apply requests. RequestApply must not block: it is called on the branch @@ -28,3 +50,63 @@ type ApplyRequest struct { type ApplyRequester interface { RequestApply(req ApplyRequest) } + +// ApplyHandBack takes back a request the requester could not deliver, so the worker owes it again. +// +// The requester retries a failed request only briefly. A retry is the worker's decision replayed +// without the quiet check: the worker decided it on a quiet branch, and by the time a long retry +// lands, live edits may be waiting in an open window that the apply would revert. So the requester +// hands a request back once it has failed for about one quiet period, and the worker sends it +// again only when the branch is quiet once more. HandBackApply must not block. +type ApplyHandBack interface { + HandBackApply(req ApplyRequest) +} + +// handBackDelay is how long the worker waits before sending a request that has been handed back +// handBacks times: doubling from handBackInitialDelay, capped at handBackMaxDelay, so a permanent +// failure (a misconfigured Kustomization) is asked about every few minutes rather than in a loop. +func handBackDelay(handBacks int) time.Duration { + delay := handBackInitialDelay + for i := 1; i < handBacks && delay < handBackMaxDelay; i++ { + delay *= 2 + } + return min(delay, handBackMaxDelay) +} + +const ( + handBackInitialDelay = 15 * time.Second + handBackMaxDelay = 5 * time.Minute +) + +// HandBackApply hands a request back to the worker for its branch. A branch with no worker has +// nothing to owe it to: a worker created later starts by owing every applier a catch-up. +func (m *WorkerManager) HandBackApply(req ApplyRequest) { + worker, found := m.GetWorkerForTarget(req.Branch.RepoName, req.Branch.RepoNamespace, req.Branch.Branch) + if !found { + return + } + worker.handBackApply(req) +} + +// handBackApply records a handed-back request in the debt and wakes the loop, which sends it +// again once it is due and the branch is quiet (serviceAppliers). The record is the debt, not the +// wake: a wake dropped on a full queue costs nothing, because the work already queued wakes the +// loop, and every wake ends by servicing the appliers. +func (w *BranchWorker) handBackApply(req ApplyRequest) { + w.debt.handBack(req, time.Now()) + w.Log.Info("An apply request could not be delivered; owing it again", + "branch", w.Branch, "reason", string(req.Reason), "gitTargets", len(req.Targets), + "handBacks", req.HandBacks, "retryIn", handBackDelay(req.HandBacks)) + w.inflightItems.Add(1) + w.pendingResyncsMu.Lock() + defer w.pendingResyncsMu.Unlock() + if w.stoppingLocked() { + w.inflightItems.Add(-1) + return + } + select { + case w.eventQueue <- WorkItem{ApplyWake: true}: + default: + w.inflightItems.Add(-1) + } +} diff --git a/internal/git/apply_trigger.go b/internal/git/apply_trigger.go index de30d0ec..a026096a 100644 --- a/internal/git/apply_trigger.go +++ b/internal/git/apply_trigger.go @@ -46,6 +46,13 @@ type applyDebt struct { // foreign and catchUp are owed until the loop takes them on a quiet branch. foreign bool catchUp bool + // returned holds the requests the requester handed back undelivered, per GitTarget with the + // reason it was owed, until returnedDue. returnedCount is the most hand-backs any of them has + // been through. They live here rather than on the loop because the requester hands back from + // its own goroutine. + returned map[itypes.ResourceReference]v1alpha3.ApplyReason + returnedCount int + returnedDue time.Time } // observe classifies one observation. The first one is a baseline, not a move: nothing says what @@ -91,6 +98,55 @@ func (d *applyDebt) restore(foreign, catchUp bool) { d.catchUp = d.catchUp || catchUp } +// handBack records a request the requester could not deliver, due after its backoff. A target +// handed back twice keeps the stronger reason (StrongerApplyReason). +func (d *applyDebt) handBack(req ApplyRequest, now time.Time) { + d.mu.Lock() + defer d.mu.Unlock() + d.mergeReturnedLocked(req.Reason, req.Targets, req.HandBacks) + if due := now.Add(handBackDelay(req.HandBacks)); due.After(d.returnedDue) { + d.returnedDue = due + } +} + +func (d *applyDebt) mergeReturnedLocked( + reason v1alpha3.ApplyReason, targets []itypes.ResourceReference, handBacks int, +) { + if d.returned == nil { + d.returned = map[itypes.ResourceReference]v1alpha3.ApplyReason{} + } + for _, target := range targets { + d.returned[target] = StrongerApplyReason(d.returned[target], reason) + } + d.returnedCount = max(d.returnedCount, handBacks) +} + +// takeReturned returns the handed-back requests and clears them, once they are due. Before then it +// returns none, and when they come due. +func (d *applyDebt) takeReturned(now time.Time) (map[itypes.ResourceReference]v1alpha3.ApplyReason, int, time.Time) { + d.mu.Lock() + defer d.mu.Unlock() + if len(d.returned) == 0 { + return nil, 0, time.Time{} + } + if now.Before(d.returnedDue) { + return nil, 0, d.returnedDue + } + returned, count := d.returned, d.returnedCount + d.returned, d.returnedCount, d.returnedDue = nil, 0, time.Time{} + return returned, count, time.Time{} +} + +// restoreReturned puts back what takeReturned returned, due at once, for a caller that could not +// act on it. +func (d *applyDebt) restoreReturned(returned map[itypes.ResourceReference]v1alpha3.ApplyReason, count int) { + d.mu.Lock() + defer d.mu.Unlock() + for target, reason := range returned { + d.mergeReturnedLocked(reason, []itypes.ResourceReference{target}, count) + } +} + // foreignOwed reports whether a ForeignCommit is owed, without taking it. func (d *applyDebt) foreignOwed() bool { d.mu.Lock() @@ -179,37 +235,57 @@ func (l *branchWorkerEventLoop) serviceAppliers() { } tip, foreign, catchUp := w.debt.take() - if !foreign && !catchUp && len(l.reapplyOwed) == 0 && len(l.catchUpOwed) == 0 { + returned, returnedCount, returnedDue := w.debt.takeReturned(now) + if !foreign && !catchUp && len(returned) == 0 && len(l.reapplyOwed) == 0 && len(l.catchUpOwed) == 0 { + l.armApplierTimer(now, returnedDue) return } - declared, ok := l.declaredAppliers(foreign, catchUp) + declared, ok := l.declaredAppliers() if !ok { + w.debt.restore(foreign, catchUp) + w.debt.restoreReturned(returned, returnedCount) return } if catchUp { l.oweCatchUp(declared) } + l.sendOwed(tip, foreign, l.oweReturned(returned, returnedCount, declared), declared, now, returnedDue) +} + +// sendOwed sends what serviceAppliers found owed on a quiet branch, and arms the timer for what is +// not due yet. +func (l *branchWorkerEventLoop) sendOwed( + tip string, + foreign bool, + returnedForeign []itypes.ResourceReference, + declared map[itypes.ResourceReference]time.Duration, + now, returnedDue time.Time, +) { if foreign && len(declared) > 0 { // The tip is the foreign commit with every pending write replayed onto it: the branch is // quiet, so anything that was pending has been published on top. Every applier on the // branch is asked, which also answers whatever else they were owed. - l.requestApply(v1alpha3.ApplyReasonForeignCommit, tip, sortedTargets(declared)) + l.requestApply(v1alpha3.ApplyReasonForeignCommit, tip, sortedTargetKeys(declared)) clear(l.reapplyOwed) clear(l.catchUpOwed) + l.armApplierTimer(now, returnedDue) return } + if len(returnedForeign) > 0 { + for _, target := range returnedForeign { + delete(l.catchUpOwed, target) + } + l.requestApply(v1alpha3.ApplyReasonForeignCommit, tip, returnedForeign) + } l.serviceRefusals(tip, declared) - l.serviceCatchUp(tip, declared, now) + l.armApplierTimer(now, l.serviceCatchUp(tip, declared, now), returnedDue) } // declaredAppliers maps every GitTarget on this branch that names an applier to its quiet period. -// On a failure to list them, what was taken is put back and a retry is armed. -func (l *branchWorkerEventLoop) declaredAppliers( - foreign, catchUp bool, -) (map[itypes.ResourceReference]time.Duration, bool) { +// On a failure to list them a retry is armed, and the caller puts back what it took. +func (l *branchWorkerEventLoop) declaredAppliers() (map[itypes.ResourceReference]time.Duration, bool) { appliers, err := l.w.appliersOnBranch(l.w.ctx) if err != nil { - l.w.debt.restore(foreign, catchUp) l.w.Log.Error(err, "Cannot list the appliers on this branch; retrying", "branch", l.w.Branch) l.resetApplierTimer(applierRetryInterval) return nil, false @@ -231,31 +307,89 @@ func (l *branchWorkerEventLoop) oweCatchUp(declared map[itypes.ResourceReference } } -// serviceRefusals sends each owed Refusal request to its target's applier alone. +// oweReturned owes again what the requester handed back, now that it is due, and returns the +// targets owed a ForeignCommit, which the caller sends at once. +// +// A Refusal is owed again only while the refusal still stands: one that recovered while its +// request was failing has nothing left to revert, and its target is owed a catch-up instead, as is +// a handed-back CatchUp. Both go through the ordinary paths, so a Refusal is sent only with consent +// and a catch-up only after the quiet period. +func (l *branchWorkerEventLoop) oweReturned( + returned map[itypes.ResourceReference]v1alpha3.ApplyReason, + count int, + declared map[itypes.ResourceReference]time.Duration, +) []itypes.ResourceReference { + var foreign []itypes.ResourceReference + for _, target := range sortedTargetKeys(returned) { + quietPeriod, ok := declared[target] + if !ok { + continue // its applier was removed since; nothing is owed to it any more + } + if l.handBackCount == nil { + l.handBackCount = map[itypes.ResourceReference]int{} + } + l.handBackCount[target] = max(l.handBackCount[target], count) + switch reason := returned[target]; { + case reason == v1alpha3.ApplyReasonForeignCommit: + foreign = append(foreign, target) + case reason == v1alpha3.ApplyReasonRefusal && l.oweStandingRefusals(target): + default: + if l.catchUpOwed == nil { + l.catchUpOwed = map[itypes.ResourceReference]time.Duration{} + } + l.catchUpOwed[target] = quietPeriod + } + } + return foreign +} + +// oweStandingRefusals owes target's applier a Refusal request again for every refusal of the +// target that has not recovered, and reports whether there was any. +func (l *branchWorkerEventLoop) oweStandingRefusals(target itypes.ResourceReference) bool { + standing := l.w.standingRefusals(target) + if l.reapplyOwed == nil && len(standing) > 0 { + l.reapplyOwed = map[refusalKey]string{} + } + for _, key := range standing { + if _, owed := l.reapplyOwed[key]; !owed { + l.reapplyOwed[key] = "a Refusal request that could not be delivered" + } + } + return len(standing) > 0 +} + +// serviceRefusals sends each owed Refusal request to its target's applier alone, once per target +// however many of its collections are refused. +// +// Consent is read again here. A refusal owed while the branch was busy can wait behind an open +// window or a publication outage for as long as those last, and a target suspended or set back to +// Ignore in the meantime has withdrawn it. func (l *branchWorkerEventLoop) serviceRefusals( tip string, declared map[itypes.ResourceReference]time.Duration, ) { - refusals := make([]itypes.ResourceReference, 0, len(l.reapplyOwed)) - for target := range l.reapplyOwed { - refusals = append(refusals, target) + targets := map[itypes.ResourceReference]struct{}{} + for key := range l.reapplyOwed { + targets[key.target] = struct{}{} } - sortReferences(refusals) - for _, target := range refusals { - delete(l.reapplyOwed, target) + clear(l.reapplyOwed) + for _, target := range sortedTargetKeys(targets) { if _, ok := declared[target]; !ok { continue // its applier was removed since the refusal; consent went with it } + if !l.w.refusalConsent(l.w.ctx, target) { + continue + } // A reapply applies the tip, which is everything a catch-up would have asked for. delete(l.catchUpOwed, target) l.requestApply(v1alpha3.ApplyReasonRefusal, tip, []itypes.ResourceReference{target}) } } -// serviceCatchUp asks every applier whose quiet period has run out to catch up, and arms the timer -// for the earliest one still waiting. +// serviceCatchUp asks every applier whose quiet period has run out to catch up, and returns when +// the earliest one still waiting comes due (zero when none waits). func (l *branchWorkerEventLoop) serviceCatchUp( tip string, declared map[itypes.ResourceReference]time.Duration, now time.Time, -) { +) time.Time { var due []itypes.ResourceReference var earliest time.Time for target, quietPeriod := range l.catchUpOwed { @@ -277,6 +411,18 @@ func (l *branchWorkerEventLoop) serviceCatchUp( sortReferences(due) l.requestApply(v1alpha3.ApplyReasonCatchUp, tip, due) } + return earliest +} + +// armApplierTimer points the applier timer at the earliest of the given deadlines that is set, or +// stops it when none is. +func (l *branchWorkerEventLoop) armApplierTimer(now time.Time, deadlines ...time.Time) { + var earliest time.Time + for _, at := range deadlines { + if !at.IsZero() && (earliest.IsZero() || at.Before(earliest)) { + earliest = at + } + } if earliest.IsZero() { l.stopApplierTimer() return @@ -284,21 +430,29 @@ func (l *branchWorkerEventLoop) serviceCatchUp( l.resetApplierTimer(earliest.Sub(now)) } -// requestApply hands one request to the requester, which delivers it off this goroutine. +// requestApply hands one request to the requester, which delivers it off this goroutine. It +// carries, and consumes, the hand-back count of its targets. func (l *branchWorkerEventLoop) requestApply( reason v1alpha3.ApplyReason, commit string, targets []itypes.ResourceReference, ) { + handBacks := 0 + for _, target := range targets { + handBacks = max(handBacks, l.handBackCount[target]) + delete(l.handBackCount, target) + } l.w.Log.Info("Asking the appliers to apply", - "branch", l.w.Branch, "reason", string(reason), "commit", commit, "gitTargets", len(targets)) + "branch", l.w.Branch, "reason", string(reason), "commit", commit, "gitTargets", len(targets), + "handBacks", handBacks) l.w.applyRequester.RequestApply(ApplyRequest{ Branch: BranchKey{ RepoNamespace: l.w.GitProviderNamespace, RepoName: l.w.GitProviderRef, Branch: l.w.Branch, }, - Reason: reason, - Commit: commit, - Targets: targets, + Reason: reason, + Commit: commit, + Targets: targets, + HandBacks: handBacks, }) } @@ -321,7 +475,7 @@ func (l *branchWorkerEventLoop) stopApplierTimer() { } } -func sortedTargets(set map[itypes.ResourceReference]time.Duration) []itypes.ResourceReference { +func sortedTargetKeys[V any](set map[itypes.ResourceReference]V) []itypes.ResourceReference { targets := make([]itypes.ResourceReference, 0, len(set)) for target := range set { targets = append(targets, target) diff --git a/internal/git/branch_worker.go b/internal/git/branch_worker.go index a2048d7e..046c31c1 100644 --- a/internal/git/branch_worker.go +++ b/internal/git/branch_worker.go @@ -1153,9 +1153,15 @@ type branchWorkerEventLoop struct { // overwrite another's pending work: A queues, B replaces it, B is then suspended, and A's // authorized request is gone with nothing to report it. refusalPending map[refusalKey]pendingRefusalReapply - // reapplyOwed holds the GitTargets whose applier is owed a Refusal request, with the refusal it - // answers, until serviceAppliers sends it on a quiet branch. See apply_trigger.go. - reapplyOwed map[itypes.ResourceReference]string + // reapplyOwed holds the refusals whose target's applier is owed a Refusal request, with the + // refusal each answers, until serviceAppliers sends it on a quiet branch. It is keyed like + // refusalPending, by target AND collection, so a collection that recovers cancels its own + // obligation and no other (refusalRecovered). See apply_trigger.go. + reapplyOwed map[refusalKey]string + // handBackCount is, per GitTarget, how many times its last request was handed back + // undelivered. The next request for the target carries it (ApplyRequest.HandBacks) and + // consumes it. See apply_trigger.go. + handBackCount map[itypes.ResourceReference]int // catchUpOwed holds, per GitTarget with an applier on this branch, how long the branch must stay // quiet before that applier is asked to catch up with this worker's own pushes. quietSince is @@ -1316,6 +1322,8 @@ func (l *branchWorkerEventLoop) dispatch(item WorkItem) int64 { return charge case item.Refresh != nil: l.handleRefreshRequest(item.Refresh) + case item.ApplyWake: + // The wake is the point: endWake's serviceAppliers sends what was handed back. case item.Request == nil: case item.Request.CommitMode == CommitModeAtomic: targetName, targetNamespace := atomicRefusalTarget(item.Request) diff --git a/internal/git/refusal_observation_test.go b/internal/git/refusal_observation_test.go index d35fa843..a6269201 100644 --- a/internal/git/refusal_observation_test.go +++ b/internal/git/refusal_observation_test.go @@ -105,7 +105,7 @@ func refusalRequests(t *testing.T, f *ledgerFixture) int { t.Helper() requester, ok := f.worker.applyRequester.(*recordingRequester) require.True(t, ok, "the fixture records apply requests") - return requester.count(configv1alpha3.ApplyReasonRefusal) + return requester.refusals() } // TestRefusalReapply_AnUnchangedRefusalIsNotASecondTrigger is the regression a review reproduced: diff --git a/internal/git/refusal_reapply.go b/internal/git/refusal_reapply.go index adc42827..b284e506 100644 --- a/internal/git/refusal_reapply.go +++ b/internal/git/refusal_reapply.go @@ -83,16 +83,20 @@ func (w *BranchWorker) refusalConsent(ctx context.Context, target itypes.Resourc "gitTarget", target.String(), "error", err.Error()) return false } + return ConsentsToReapply(gitTarget) +} + +// ConsentsToReapply reports whether a GitTarget, as it is now, asks for its applier to reapply on a +// refusal. The worker asks before it owes a Refusal request and again before it sends one, and the +// applier dispatcher asks before every attempt to deliver one. +func ConsentsToReapply(gitTarget *v1alpha3.GitTarget) bool { if gitTarget.EffectiveOnRefusal() != v1alpha3.RefusalActionReapply || gitTarget.Spec.Applier == nil { return false } - if gitTarget.Spec.Suspend { - // A suspended target has stopped publishing the live edits a reapply would be weighed - // against. Driving its applier on behalf of a target the operator has deliberately parked - // would be acting on a mirror nobody is keeping. - return false - } - return true + // A suspended target has stopped publishing the live edits a reapply would be weighed against. + // Driving its applier on behalf of a target the operator has deliberately parked would be + // acting on a mirror nobody is keeping. + return !gitTarget.Spec.Suspend } // refusalRateLimited reports whether a request for this target is inside the rate-limit window, @@ -248,17 +252,20 @@ func (w *BranchWorker) refusalAlreadyCovered(key refusalKey, observation string) return w.coveredRefusal[newRefusalKey(key.target, key.collection)] == observation } -// recordRefusalObservation remembers what the request just owed covers. +// recordRefusalObservation remembers what the request just owed covers. An empty observation is +// recorded too, though it covers nothing (refusalAlreadyCovered): the entry is also the record that +// the refusal stands until it recovers (standingRefusals). func (w *BranchWorker) recordRefusalObservation(key refusalKey, observation string) { - if observation == "" { - return - } w.refusalMu.Lock() defer w.refusalMu.Unlock() if w.coveredRefusal == nil { w.coveredRefusal = map[refusalKey]string{} } - w.coveredRefusal[newRefusalKey(key.target, key.collection)] = observation + key = newRefusalKey(key.target, key.collection) + if _, standing := w.coveredRefusal[key]; standing && observation == "" { + return // an observation that covers nothing must not replace one that covers something + } + w.coveredRefusal[key] = observation } // refusalRecovered records that an evaluation for this target and collection was ACCEPTED, so whatever @@ -282,14 +289,40 @@ func (l *branchWorkerEventLoop) refusalRecovered(target itypes.ResourceReference } collection = refusalBoundary(collection) l.w.clearRefusalObservations(target, collection) + recovered := func(key refusalKey) bool { + return key.target == target && (collection == (itypes.CollectionKey{}) || key.collection == collection) + } for _, key := range sortedRefusalKeys(l.refusalPending) { - if key.target == target && (collection == (itypes.CollectionKey{}) || key.collection == collection) { + if recovered(key) { delete(l.refusalPending, key) } } + // An obligation already owed, waiting for a quiet branch, is cancelled the same way. Keyed by + // target alone it could not be told apart from a sibling collection's, so it outlived the + // recovery and asked for an apply for a refusal that no longer existed. + for key := range l.reapplyOwed { + if recovered(key) { + delete(l.reapplyOwed, key) + } + } l.rearmRefusalTimer() } +// standingRefusals returns the refusals of target a request was owed for and that have not +// recovered since, in a stable order. +func (w *BranchWorker) standingRefusals(target itypes.ResourceReference) []refusalKey { + w.refusalMu.Lock() + defer w.refusalMu.Unlock() + var keys []refusalKey + for key := range w.coveredRefusal { + if key.target == target { + keys = append(keys, key) + } + } + sort.Slice(keys, func(i, j int) bool { return keys[i].String() < keys[j].String() }) + return keys +} + // clearRefusalObservations forgets what this collection's last reapply request covered. The ZERO collection is a // whole-GitTarget evaluation, which does speak for every collection, so it clears all of them. func (w *BranchWorker) clearRefusalObservations(target itypes.ResourceReference, collection itypes.CollectionKey) { @@ -405,12 +438,13 @@ func sortedRefusalKeys(pending map[refusalKey]pendingRefusalReapply) []refusalKe func (l *branchWorkerEventLoop) oweReapply(key refusalKey, detail, observation string) { // Recorded as covered now, when the request is owed, rather than when it is delivered: the // request is the worker's decision, and a recheck of the same refusal while the branch is still - // busy must not owe a second one. Delivery failures are the requester's to retry. + // busy must not owe a second one. A request the requester cannot deliver comes back + // (ApplyHandBack) and is owed again while the refusal stands. l.w.recordRefusalObservation(key, observation) if l.reapplyOwed == nil { - l.reapplyOwed = map[itypes.ResourceReference]string{} + l.reapplyOwed = map[refusalKey]string{} } - l.reapplyOwed[key.target] = detail + l.reapplyOwed[key] = detail l.w.Log.Info("Refused edit owes its applier a reapply, sent once the branch is quiet", "gitTarget", key.target.String(), "branch", l.w.Branch, "refused", detail) } diff --git a/internal/git/refusal_reapply_test.go b/internal/git/refusal_reapply_test.go index ae8b3418..e9b3c055 100644 --- a/internal/git/refusal_reapply_test.go +++ b/internal/git/refusal_reapply_test.go @@ -485,13 +485,13 @@ func (r *recordingRequester) RequestApply(req ApplyRequest) { r.requests = append(r.requests, req) } -// count is how many requests with this reason were recorded, taken or not. -func (r *recordingRequester) count(reason configv1alpha3.ApplyReason) int { +// refusals is how many Refusal requests were recorded, taken or not. +func (r *recordingRequester) refusals() int { r.mu.Lock() defer r.mu.Unlock() - n := r.counted[reason] + n := r.counted[configv1alpha3.ApplyReasonRefusal] for _, req := range r.requests { - if req.Reason == reason { + if req.Reason == configv1alpha3.ApplyReasonRefusal { n++ } } diff --git a/internal/git/types.go b/internal/git/types.go index 2511cef5..8748dc43 100644 --- a/internal/git/types.go +++ b/internal/git/types.go @@ -424,6 +424,9 @@ type WorkItem struct { // Withdraw cancels a CommitRequest the worker has not acted on yet. It rides the same FIFO as // the attach, so it is always handled after every attach sent before it. Withdraw *AttachCommitRequest + // ApplyWake wakes the loop after an apply request was handed back (ApplyHandBack). It carries + // nothing: the request is in the worker's debt, and the wake's serviceAppliers sends it. + ApplyWake bool // charge is what a write or an attach was charged against the intake budget at enqueue, and is // released once the loop has handled it. A resync's is on its marker, because coalescing can From 3e130d5463a3f6c825e4803005fa53e51899bea8 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 19:07:59 +0000 Subject: [PATCH 12/36] docs: narrow the refusal promise and record the review fixes A Refusal asks one applier, but a GitRepository that has to fetch the tip wakes every Kustomization that reads it; say so in the five places that promised no other folder is woken. Document the per-attempt checks and the hand-back, the dropped outcome, the upgrade order the old CRD allows, and the access-review permission. Co-Authored-By: Claude Opus 5.5 (1M context) --- api/v1alpha3/gittarget_types.go | 6 +- .../crd/bases/configbutler.ai_gittargets.yaml | 6 +- docs/UPGRADING.md | 21 +- docs/architecture.md | 13 +- docs/bi-directional.md | 14 +- docs/configuration.md | 31 ++- docs/design/apply-trigger-follow-ups.md | 249 ++++++------------ ...push-notification-and-reconcile-trigger.md | 16 +- docs/interpreting-metrics.md | 5 +- 9 files changed, 151 insertions(+), 210 deletions(-) diff --git a/api/v1alpha3/gittarget_types.go b/api/v1alpha3/gittarget_types.go index 291b05ef..e542f354 100644 --- a/api/v1alpha3/gittarget_types.go +++ b/api/v1alpha3/gittarget_types.go @@ -110,8 +110,10 @@ type GitTargetSpec struct { // // "Reapply" asks this target's applier (spec.applier, which it requires) to apply the current // tip once the branch holds no open window and no pending writes, which reverts the refused - // edit and nothing that is still waiting to be published. No other folder on the branch is - // woken. + // edit and nothing that is still waiting to be published. Only this target's applier is asked; + // if its GitRepository has to fetch to reach the tip, every Kustomization on that GitRepository + // applies the new revision, as for any push. Give the folder its own GitRepository to keep the + // apply to it alone. // // It fires only for an edit to a document the folder ALREADY holds, in a write that removes no // document anywhere: re-applying would otherwise prune, or put back, an object somebody else is diff --git a/config/crd/bases/configbutler.ai_gittargets.yaml b/config/crd/bases/configbutler.ai_gittargets.yaml index b933278a..15b1b78e 100644 --- a/config/crd/bases/configbutler.ai_gittargets.yaml +++ b/config/crd/bases/configbutler.ai_gittargets.yaml @@ -365,8 +365,10 @@ spec: "Reapply" asks this target's applier (spec.applier, which it requires) to apply the current tip once the branch holds no open window and no pending writes, which reverts the refused - edit and nothing that is still waiting to be published. No other folder on the branch is - woken. + edit and nothing that is still waiting to be published. Only this target's applier is asked; + if its GitRepository has to fetch to reach the tip, every Kustomization on that GitRepository + applies the new revision, as for any push. Give the folder its own GitRepository to keep the + apply to it alone. It fires only for an edit to a document the folder ALREADY holds, in a write that removes no document anywhere: re-applying would otherwise prune, or put back, an object somebody else is diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index c8a3cdd4..a49549b7 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -13,22 +13,29 @@ We are pre-1.0, so breaking changes bump the **minor** version (release-please i GitOps Reverser no longer moves the branch with an empty commit when an edit is refused. With `Reapply` it asks the target's applier, the Flux `Kustomization` named in the new `spec.applier.kustomizationRef`, to apply the current tip once the branch holds no open commit window -and no pending writes. Only that applier is asked, so other folders on the branch are no longer woken, -and an allowed edit still in a commit window is published before the apply instead of being reverted +and no pending writes. Only that applier is asked, rather than every applier watching the branch, +though a `GitRepository` that has to fetch the tip still wakes every `Kustomization` that reads it. +An allowed edit still in a commit window is published before the apply instead of being reverted with the refused one. The API rejects `Reapply` without `spec.applier`. A stored `GitTarget` that still says `PushEmptyCommit` reads as `Ignore`, and any update to it is -rejected until the field changes. Before upgrading, set `onRefusal: Ignore` on those targets, or -`onRefusal: Reapply` together with `spec.applier`. See -[reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal). +rejected until the field changes. The old CRD rejects `Reapply`, so move in this order: + +1. Before upgrading, set `onRefusal: Ignore` on those targets. +2. Upgrade the CRDs, the controller and its permissions. +3. Set `spec.applier` on each target and wait for `ApplierConfigured=True`. +4. Set `onRefusal: Reapply` where you want it. + +See [reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal). `spec.applier` also changes what else applies the folder: GitOps Reverser asks the Kustomization to apply when somebody else pushes to the branch, and to catch up with its own pushes once the branch has been quiet for `spec.applier.quietPeriod`. Lengthen the `GitRepository` and `Kustomization` intervals and remove any Git-host webhook into a Flux `Receiver` for the repository, as the [bi-directional guide](bi-directional.md#configure-flux) describes, or Flux keeps applying on its own -schedule as well. The operator needs `get` and `patch` on `kustomizations` and `gitrepositories` in -the source cluster: Helm `applierAccess.flux: true` grants it for the operator's own cluster. +schedule as well. The operator needs `get` and `patch` on `kustomizations` and `gitrepositories`, +and `create` on `selfsubjectaccessreviews`, in the source cluster: Helm `applierAccess.flux: true` +grants them for the operator's own cluster. ## `git_queue_drops_total` is now `git_queue_refusals_total` diff --git a/docs/architecture.md b/docs/architecture.md index 6157b0c9..3cd1c734 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1533,9 +1533,9 @@ Controllers watch their dependencies so dependents reconcile quickly after spec Namespace label changes re-check provider-reference access. Rule changes re-resolve selected collections. It deliberately does **not** watch encryption Secrets, so their recovery is picked up by periodic reconciliation without retaining - every Secret value in the control-plane cache. Nor does it watch appliers: it reads the - `Kustomization` or `Application` on its tick to set `ApplierConfigured`, so a cluster without the - Flux or Argo CD CRDs needs no informer for them. + every Secret value in the control-plane cache. Nor does it watch appliers: it reads the Flux + `Kustomization` on its tick to set `ApplierConfigured`, so a cluster without the Flux CRDs needs no + informer for them. - `WatchRuleReconciler` / `ClusterWatchRuleReconciler` watch `GitTarget` and `GitProvider`, populate the RuleStore, and trigger the rule-change reconcile. - `GitProviderReconciler` validates reachability and manages the signing key lifecycle. @@ -1641,9 +1641,10 @@ as one funnel and its loss paths as one selector: **Apply requests.** `apply_requests_total{applier_kind,reason,outcome}` counts every request the worker decided to send (`applier_kind` is `kustomization`; `reason` is `ForeignCommit`, `Refusal` or `CatchUp`), with `outcome` separating -`sent` from `failed` and `superseded` (a newer request for the same applier replaced it before it -went out). `receiver_deliveries_total{type,outcome}` counts receiver deliveries: `notified`, -`unmatched`, `ignored` (a ping, a tag, another event type), `rejected` (bad signature) and `invalid`. +`sent` from `failed` (handed back to the branch worker), `superseded` (a newer request for the same +applier replaced it before it went out) and `dropped` (no target may still ask). +`receiver_deliveries_total{type,outcome}` counts receiver deliveries: `notified`, `unmatched`, +`ignored` (a ping, a tag, another event type), `rejected` (bad signature) and `invalid`. The receiver cannot tell a foreign push from an own one; the worker decides that from the advertisement, which is why a `notified` delivery on an otherwise quiet branch is followed by a `CatchUp` and only a genuinely foreign one by a `ForeignCommit`. See diff --git a/docs/bi-directional.md b/docs/bi-directional.md index c34ed1bd..aa70ac20 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -418,11 +418,15 @@ until something applies Git over it. | Policy | Response to a refused edit | Operator's trade | | --- | --- | --- | | `Ignore` (default) | Report the refusal | Leave recovery to a human or a later apply | -| `Reapply` | Ask the target's applier to apply the current tip, once the branch is quiet | The refused edit is reverted; nothing else on the branch is | - -`Reapply` needs a declared applier, and the request goes to that applier alone: other folders on -the branch are not woken. Waiting for a quiet branch means an allowed edit still in a commit window -is published first, so the apply reverts only what has no home in Git. +| `Reapply` | Ask the target's applier to apply the current tip, once the branch is quiet | The refused edit is reverted; nothing still waiting to be published is | + +`Reapply` needs a declared applier, and the request goes to that applier alone. When the applier's +`GitRepository` has to fetch to reach the tip, though, every `Kustomization` on that +`GitRepository` applies the new revision, exactly as it would for any push: moving the source is +the only way to apply the tip, and Flux wakes everything that reads it. Give the folder its own +`GitRepository` if the apply must reach that folder alone. Waiting for a quiet branch means an +allowed edit still in a commit window is published first, so the apply reverts only what has no +home in Git. Eligibility is narrow: the folder must be accepted, the refused edit must concern a document the folder already holds, and the write must remove no document, because re-applying would put back an diff --git a/docs/configuration.md b/docs/configuration.md index db2c3706..8474b602 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1139,9 +1139,23 @@ cluster already holds; an artifact left behind would revert edits that are alrea the artifact already holds the requested commit. A new artifact wakes the `Kustomization` by itself. A `CatchUp` whose artifact already holds the commit stamps nothing. -A request asks; it never claims the apply happened. It runs off the publication path, is retried -with backoff on its own, and a failed request never fails a publication that succeeded. Read the -`Kustomization`'s `status.lastAppliedRevision` for what it applied. +A request asks; it never claims the apply happened. It runs off the publication path, and a failed +request never fails a publication that succeeded. Read the `Kustomization`'s +`status.lastAppliedRevision` for what it applied. + +**Every attempt checks again.** A request is decided once, on a quiet branch, and can be attempted +later. So before every attempt, retries included, GitOps Reverser reads the `GitTarget` again and +asks nothing for it when it no longer names this applier on this branch or its `ClusterProvider` no +longer admits its namespace. A `Refusal` is asked only while the target still consents +(`onRefusal: Reapply`, not suspended); otherwise its applier is asked a `CatchUp`. And nothing is +stamped unless the `Kustomization`'s `GitRepository` fetches this repository and branch, whatever +`ApplierConfigured` last said. + +**A request that fails** is retried for about one `quietPeriod`, or not at all when the +`Kustomization` cannot take a request, and then goes back to the branch worker, which asks again +once the branch is quiet, waiting longer each time up to a few minutes. It is not retried for +longer than that because a retry skips the quiet check: Flux applies whatever the tip is when the +retry lands, and by then live edits may be waiting to be published. **What it reports.** @@ -1168,7 +1182,8 @@ status: It is checked on the target's reconcile tick rather than watched, so a cluster without the Flux CRDs installed costs nothing. Each delivered request also records an `ApplyRequested` Event on the -`GitTarget`, and a request that keeps failing records `ApplyRequestFailed`. +`GitTarget`, and a request that could not be delivered records `ApplyRequestFailed` once, not again +each time the worker asks again. ### Reverting a refused edit (`spec.onRefusal`) @@ -1190,9 +1205,11 @@ spec: | `Reapply` | Once the branch is quiet, this target's [applier](#who-applies-this-folder-specapplier) is asked to apply the current tip, with reason `Refusal`. The refused edit is reverted | `Reapply` requires `spec.applier`; a target without one is rejected at admission. The request goes -to this target's applier alone, so other folders on the branch are not woken, and it waits for a -quiet branch, so an allowed edit still in a commit window is published before the apply and is not -reverted with the refused one. The [bi-directional guide](bi-directional.md#choosing-speconrefusal) +to this target's applier alone, and it waits for a quiet branch, so an allowed edit still in a commit +window is published before the apply and is not reverted with the refused one. When the applier's +`GitRepository` has to fetch to reach the tip, every `Kustomization` on that `GitRepository` applies +the new revision, as it would for any push; give the folder its own `GitRepository` if the apply +must reach it alone. The [bi-directional guide](bi-directional.md#choosing-speconrefusal) covers when to choose it. **It only fires for an edit to a document the folder already holds, and never for a write that diff --git a/docs/design/apply-trigger-follow-ups.md b/docs/design/apply-trigger-follow-ups.md index f9671ecb..a0bfbab1 100644 --- a/docs/design/apply-trigger-follow-ups.md +++ b/docs/design/apply-trigger-follow-ups.md @@ -5,10 +5,9 @@ > The design this continues: [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md). > The user-facing model: [`../bi-directional.md`](../bi-directional.md). -The branch makes GitOps Reverser decide when a Flux `Kustomization` applies a folder. This page is -what comes next, in order: the fixes the 2026-10-16 review found, which block the merge; Argo CD; -the breaking change; the smaller loose ends; and the parts of the feature the tests do not reach -yet. +The branch makes GitOps Reverser decide when a Flux `Kustomization` applies a folder. This page +records how the 2026-10-16 review's findings were fixed, then what comes next: Argo CD, the +breaking change, the smaller loose ends, and the parts of the feature the tests do not reach yet. ## 1. Where the branch stands @@ -23,176 +22,77 @@ Built and tested: `Refusal` request ([`refusal_reapply.go`](../../internal/git/refusal_reapply.go)). - **Delivery**, in [`internal/applier`](../../internal/applier/): the Flux client (stamp the `GitRepository`, or the `Kustomization` when the artifact already holds the commit), the - `Validate` behind `ApplierConfigured`, and the dispatcher (per-applier superseding, retry with - backoff, Events, the ledger behind `status.applier`). + `Validate` behind `ApplierConfigured`, and the dispatcher (per-applier superseding, a check of + every target before every attempt, a short retry and then a hand-back to the worker, Events, the + ledger behind `status.applier`). - **The receiver**, `GitProvider.spec.receiver` on `--receiver-bind-address` ([`receiver.go`](../../internal/webhook/receiver.go)), which turns a verified delivery into a refresh that always reaches the remote. Not built: the Argo CD applier, the webhook applier. The API accepts `kustomizationRef` only. -## 2. Before merge: what the review found +## 2. The review's findings, fixed The review of `21df16be` (`docs/reviews/20261016-review.md`, kept out of the tree) found six -problems. Five hold as reported, and one holds only for its wording. They have one cause: the -worker decides a request once, and nothing after that decision checks it again. The dispatcher -attempts it for up to four minutes after the worker decided, against whatever the cluster, the -`GitTarget` and the branch have become in the meantime. GitOps Reverser now controls when -deployments happen, so each decision has to hold both when it is made and when a patch is sent. - -### 2.1 A target the ClusterProvider denies still drives its applier (finding 1) - -`appliersOnBranch` lists every `GitTarget` on the provider and branch that names an applier, and -`Dispatcher.resolve` checks nothing more. `SourceDynamicClient` hands out a cluster's client when -**any** target declared that cluster, so a target with `Validated=False/NamespaceNotAuthorized` -rides on another target's declaration. That target can be kept on a running worker by any -legitimate target in its namespace on the same branch. The reviewer reproduced the patch. - -The damage is bounded: the request is a reconcile annotation, and once §2.2 is fixed it reaches only -a `Kustomization` that applies this repository and branch. It is still an authorization bypass, -and the fix is cheap. The dispatcher calls `authz.GitTargetAdmitted` for every target in a job, -on every attempt (see §2.4), and drops a target it denies. A client being available is not -permission to use it. - -One more gap sits at the same fix site, and the review did not raise it: `kustomizationRef.namespace` -is unrestricted, so an admitted target can name a `Kustomization` in any namespace of the source -cluster, while the operator's `flux-applier` role grants `patch` cluster-wide. A WatchRule needs -`allowAnySourceNamespace` on its ClusterProvider to reach outside its own namespace, and the applier -reference should be held to the same rule. - -### 2.2 A `SourceMismatch` applier is still driven (finding 2) - -`Flux.Validate` checks the `GitRepository`'s URL and branch, but `Flux.Request` does not, and the -dispatcher never reads the verdict. A target that publishes `main` and names a `Kustomization` whose -source follows `release` reports `ApplierConfigured=False/SourceMismatch`, yet every foreign commit -on `main` still stamps that `release` source. The reviewer reproduced this as well. - -`Request` already reads both objects, so it runs `sourceMismatch` on them before it stamps and -returns a `permanentError` on a mismatch. Reading the published condition instead would leave the -race between the check and the patch. This means `Request` needs the expected repository URL and -branch, which the job does not carry today. The dispatcher reads the `GitProvider` when it resolves -the job. This also shapes the `Applier` interface in §3.2: the expected source binding is an input -to `Request`, not only to `Validate`. - -### 2.3 A delayed Refusal outlives its consent and its recovery (finding 3) - -There are two waiting places for a refusal, and only one of them checks again. `flushPendingReapply`, -which handles the rate-limit trailer, calls `refusalConsent`, as that function's comment promises. -`reapplyOwed` does not. A refusal owed while the branch is busy, behind an open window or a -publication outage, is sent by `serviceRefusals` once the branch is quiet, and the only check left -is whether the target still names an applier. So it goes out after the target is suspended or set to -`Ignore`. - -`refusalRecovered` cancels `refusalPending` but leaves `reapplyOwed` alone, because `reapplyOwed` is -keyed by target and has lost the collection it would have to match. A refusal that has recovered, -which means its edit was accepted and is now pending, still asks for an apply once the branch is -quiet. By then the tip carries the accepted edit, so the apply is harmless, but it is still a -request made for a refusal that no longer exists. - -Fix both in the worker: - -- Key `reapplyOwed` by `refusalKey`, and have `refusalRecovered` delete the matching entries the way - it already does for `refusalPending`. -- Have `serviceRefusals` call `refusalConsent` before it sends. - -The dispatcher has to check consent again on its own attempts (see §2.4). A job's reason is per -applier, but consent is per target: a `Refusal` job can have absorbed a `CatchUp` for a sibling -target (`supersedingReason`). The job therefore has to keep each target's own reason. When the only -target that consented drops out, the job falls back to `CatchUp` for the rest rather than being -dropped. - -### 2.4 One rule for all three: check again at the patch - -§2.1 to §2.3 need the same check, so it should be written once. Before each attempt, the dispatcher -reads each target in the job again from the cache and keeps only the targets that: - -- still name this applier, as `resolve` already checks once, -- `authz.GitTargetAdmitted` admits (§2.1), -- consent to a refusal, for a target that is in the job only for a `Refusal` (§2.3). - -Then `Request` checks the source binding (§2.2). A job with no target left is dropped and counted. -This is the last gate before the patch, so the retries go through it too, which is the point the -review makes about "including retries". - -### 2.5 A request given up on is lost (finding 4) - -After eight failed attempts the dispatcher emits `ApplyRequestFailed` and discards the job, while -the worker cleared its debt when it sent the request. The `DefaultBackoff` comment reads "Nothing is -lost by giving up", but that assumes another request comes, and none has to. A refresh that sees -the same tip owes nothing, so a `ForeignCommit` that failed while the source API was down waits for -the next commit or for Flux's long backstop interval. A failed `Refusal` is worse: `oweReapply` -marks the observation as covered when the request is owed, so the same standing refusal never owes -another request until it recovers. - -The review asks to "retain or reconstruct" the request, and I would reconstruct it in the worker -rather than make the dispatcher retry forever. The reason is one the review does not raise: **a -retry is a request sent without the quiet check.** The worker decided it on a quiet branch, and the -dispatcher replays that decision up to four minutes later. Flux cannot pin a commit, so the retry -applies whatever the tip is then, and live edits waiting in an open window by that time are -reverted. That is the hazard this whole design exists to close. Retrying for longer makes the -window longer. - -The shape I propose: - -- The dispatcher keeps its short retries for a blip of a few seconds, since a request decided a - moment ago is still sound. It hands a job back to the worker once the job has failed for longer - than about one `quietPeriod`, or permanently, instead of giving up. "Handing back" is a new call on - the requester's side of `git.ApplyRequester`, delivered as an event on the worker's loop. -- The worker puts the debt back (`applyDebt.restore`, or `reapplyOwed` for a refusal). It re-sends - only on a quiet branch, with its own backoff capped at a few minutes so that a permanent failure - does not produce a tight loop. -- `ApplyRequestFailed` stays, emitted once each time a request goes from delivering to failing, - not on every attempt. - -A `permanentError`, which is a misconfigured `Kustomization` and now also a source mismatch, then -stays owed and is delivered once the configuration is fixed, rather than being forgotten. A -restart loses the hand-back state, which is fine: the first observation owes a `CatchUp`, and that -stamps the source when its artifact is behind. - -### 2.6 A Refusal can wake folders that share the GitRepository (finding 5) - -This one is a contract problem, and no reason to enforce isolation. When the artifact -is behind the tip, `Request` stamps the `GitRepository`, the new revision wakes every -`Kustomization` on it, and those reapply as well. A `Kustomization` that stamped only itself would be -worse: it reconciles from the old artifact and reverts edits that are already in Git. So the source -has to move, and moving the source wakes everything that consumes it. That is how Flux works, not -something GitOps Reverser can avoid. - -The harm is small. The branch is quiet, because a Refusal is sent only then, so the woken siblings -apply a tip with nothing live left out of it. They would have applied the same tip at their own -`CatchUp` one `quietPeriod` later. A `Kustomization` that GitOps Reverser does not know about would -apply it at its next interval. What is wrong is the promise. Five places say a refusal wakes no -other folder: - -- the `onRefusal` doc comment in `gittarget_types.go`, -- [`bi-directional.md`](../bi-directional.md), -- [`configuration.md`](../configuration.md), -- [`UPGRADING.md`](../UPGRADING.md), -- §4.5 of the trigger design. - -The fix is to narrow all five: only this target's applier is asked, and if its `GitRepository` has -to fetch to reach the tip, every `Kustomization` on that `GitRepository` applies the new revision, as -it would for any push. Give the folder its own `GitRepository` for strict isolation. - -### 2.7 Validation can hold the GitTarget controller indefinitely (finding 6) - -`applierVerdict` calls `ApplierClients` and `Validate` with the reconcile context, which has no -deadline. The source client has no request timeout either, by design, because it carries the -watches. The GitTarget controller runs one reconcile worker, so an API server that accepts the -connection and never answers stops every GitTarget's reconcile, not only this target's. Wrap both -calls in a bounded context. About 10 seconds is enough: `Validate` makes two GETs and two access -reviews, and a timeout is already `ReasonCheckFailed`, so `keptOrUnknown` keeps the previous verdict. -This is the same fix site as the per-reconcile cost in §6. Do the caching there together with this. - -### 2.8 Validation the review ran - -The review ran `task lint`, `task test` and `task test-e2e-bi-directional`, and all three were -green. `task test-e2e` failed one spec, audit-route attribution, which passed on its own. That spec -is the known flaky one: it has failed twice in CI on comment-only commits, and both passed on a -rerun. It is not evidence against this branch. Each fix above gets a unit test: - -- the reviewer's reproductions for findings 1 to 4 and 6, kept as regression tests, -- a test that a hand-back is re-sent only on a quiet branch. +problems with one cause: the worker decided a request once, and nothing after that decision checked +it again. The dispatcher attempted it for up to four minutes later, against whatever the cluster, +the `GitTarget` and the branch had become. GitOps Reverser now controls when deployments happen, so +each decision has to hold both when it is made and when a patch is sent. All six are fixed on the +branch, each with the reviewer's reproduction kept as a regression test. + +### 2.1 Every attempt checks again (findings 1, 2, 3) + +Before each attempt, retries included, the dispatcher reads each `GitTarget` in the job again and +keeps only those that still name this applier on this branch and that `authz.GitTargetAdmitted` +admits (finding 1: a shared source client is no permission to use it). A `Refusal` needs +`git.ConsentsToReapply` at the attempt; a target that withdrew consent is asked a `CatchUp` instead. +For that, a job keeps each target's own reason, merged with `git.StrongerApplyReason`. A job with no +target left is dropped and counted `outcome="dropped"`. + +Then `Flux.Request` checks the source binding (`Binding{RepositoryURL, Branch}`, read from the +`GitProvider` at the attempt) against the objects it is about to stamp, and returns a permanent +error on a mismatch (finding 2). `Validate` takes the same `Binding`, which is the shape §3.2's +`Applier` interface needs. + +**Decided: no namespace gate on `kustomizationRef`.** The earlier draft held a `kustomizationRef` +in another namespace to the WatchRule rule (`allowAnySourceNamespace`). It is not gated: once the +source binding is checked at the patch, a request reaches only a `Kustomization` that applies this +`GitProvider`'s repository and branch. Whoever can create a `GitTarget` on that `GitProvider` can +already write what that `Kustomization` applies, so an early reconcile adds no power, and Flux's +usual layout (`Kustomization`s in `flux-system`) keeps working without widening WatchRule +delegation. + +### 2.2 A waiting refusal is checked when it stops waiting (finding 3) + +`reapplyOwed` is keyed by `refusalKey`, and `refusalRecovered` cancels the matching entries as it +does `refusalPending`. `serviceRefusals` reads consent again before it sends. + +### 2.3 A request that cannot be delivered goes back to the worker (finding 4) + +The dispatcher retries for about the shortest `quietPeriod` of the job's targets (immediately for a +permanent error), then hands every target back with its reason (`git.ApplyHandBack`, implemented by +the `WorkerManager`). A retry is the worker's decision replayed without the quiet check, so it is +kept short; the worker is what can tell whether the branch is quiet. + +The worker keeps hand-backs in its `applyDebt` (written from the dispatcher's goroutine, then a wake +on the queue), due after a backoff of 15 seconds doubling to 5 minutes. When due and the branch is +quiet: a `ForeignCommit` is sent to those targets; a `Refusal` is owed again only while the refusal +stands (an entry in `coveredRefusal`, which is now recorded even for an empty observation), and so +only with consent; anything else becomes a `CatchUp` and waits for the quiet period. + +`ApplyRequest.HandBacks` carries the count through the worker and back, so the backoff grows and +`ApplyRequestFailed` is recorded only for a request at zero: once per failing request. + +### 2.4 The refusal promise is narrowed (finding 5) + +The five places now say that only this target's applier is asked, that a `GitRepository` that has +to fetch wakes every `Kustomization` reading it, as for any push, and that a folder needing strict +isolation gets its own `GitRepository`. + +### 2.5 Validation is bounded (finding 6) + +`applierVerdict` runs the client lookup and `Validate` under a 10-second deadline. A timeout is +`ReasonCheckFailed`, so the published verdict is kept. The per-reconcile cost (§6) is still open. ## 3. Next: the Argo CD applier @@ -299,12 +199,11 @@ Three properties make it different from Flux, and each one has to survive into t `onRefusal: PushEmptyCommit` is gone, and `Reapply` needs a Flux applier, so until §3 lands a folder that Argo CD applies has no automatic revert. GitOps Reverser has no users yet, so there is -nothing to migrate and no reason to hold the merge for Argo CD. The branch merges once §2 is done, -and Argo CD can come in its own PR. Do not bring `PushEmptyCommit` back as a fallback. +nothing to migrate and no reason to hold the merge for Argo CD, which can come in its own PR. Do not +bring `PushEmptyCommit` back as a fallback. -One fix in [`UPGRADING.md`](../UPGRADING.md): its entry says to set `onRefusal: Reapply` before -upgrading, which the old CRD rejects. The order is `Ignore`, then upgrade, then set -`spec.applier`, then `Reapply`. +[`UPGRADING.md`](../UPGRADING.md) gives the order the old CRD allows: `Ignore`, then upgrade, then +set `spec.applier`, then `Reapply`. ## 5. The webhook applier @@ -327,13 +226,13 @@ other end does not apply on its own. No in-process plugin, for the reason the tr - **`ApplierConfigured` reads the source cluster on every reconcile**: two GETs and two access reviews. A target that is not converged reconciles every ten seconds, so a broken target costs that much. Cache the verdict per `(cluster, applier, generation)` for a minute, or check only on - the steady tick and on spec changes. Do this together with the deadline in §2.7. + the steady tick and on spec changes. The deadline is in place (§2.5). - **`SourceDynamicClient` serves only declared source clusters.** The dispatcher retries until the GitTarget's watches have declared the cluster, which covers startup. A target whose declaration keeps failing therefore never gets its requests delivered, and only the retry log says why. Count it in `apply_requests_total{outcome="failed"}` with a reason, or surface it in `ApplierConfigured`. - This check answers whether the cluster has a client yet. It does not answer whether this target - may use that client, which is §2.1. + This check answers whether the cluster has a client yet. Whether this target may use that client + is §2.1. - **The dispatcher's inbox is capped at 1024 and drops the oldest with a log line only.** That can only fill if the loop is wedged, but a drop there is a lost `ForeignCommit`. Count it. - **A catch-up when nothing was pushed since the last one.** The first observation after a restart @@ -377,7 +276,7 @@ Ordered by how much I would worry about each. 7. **A foreign commit while the remote is unreachable.** The rule is that the request waits with the pending writes (§3.4 of the design). Nothing tests that it does not go out during the outage, or that it goes out once the push lands. The same test with a `Refusal` owed during the - outage, and the target set to `Ignore` before the outage ends, covers §2.3. + outage, and the target set to `Ignore` before the outage ends, covers §2.2 end to end. 8. **A foreign commit closing a window a `CommitRequest` is attached to.** `closeWindowForForeignCommit` finalizes it early; the request should resolve `Committed` with the replayed commit. Untested. diff --git a/docs/design/push-notification-and-reconcile-trigger.md b/docs/design/push-notification-and-reconcile-trigger.md index 74b450ed..01db9456 100644 --- a/docs/design/push-notification-and-reconcile-trigger.md +++ b/docs/design/push-notification-and-reconcile-trigger.md @@ -303,10 +303,16 @@ in-process plugin, because it would run with every credential the operator holds - A request asks and never reports. Nothing GitOps Reverser publishes claims the apply happened; the applier's own status says that. - It runs off the worker's queue, in `internal/applier`, so a slow applier never holds the branch. -- It retries with backoff, and a newer request for the same applier supersedes one not yet sent. +- Every attempt reads its targets again: one that no longer names the applier on this branch, or + that its `ClusterProvider` no longer admits, is dropped, and a `Refusal` needs consent at the + attempt, not only when it was decided. The request checks the source binding before it stamps. +- It retries briefly, for about one `quietPeriod`, because a retry is the worker's decision + replayed without the quiet check. Then it is handed back to the worker, which owes it again and + sends it on a quiet branch, with a backoff of its own. A newer request for the same applier + supersedes one not yet sent. - A failed request never fails the publication that caused it. - Each request records an `ApplyRequested` Event and updates `status.applier`. A request that - keeps failing records `ApplyRequestFailed`. + could not be delivered records `ApplyRequestFailed` once. ### 3.7 Refusals: `onRefusal: Reapply` @@ -426,7 +432,7 @@ status: | Family | Labels | Reads as | | --- | --- | --- | -| `gitopsreverser_apply_requests_total` | `applier_kind`, `reason`, `outcome` | Requests decided. `outcome`: `sent`, `failed`, `superseded` | +| `gitopsreverser_apply_requests_total` | `applier_kind`, `reason`, `outcome` | Requests decided. `outcome`: `sent`, `failed`, `superseded`, `dropped` | | `gitopsreverser_receiver_deliveries_total` | `type`, `outcome` | Deliveries. `outcome`: `notified`, `unmatched`, `ignored`, `rejected`, `invalid`. Foreign or own is the worker's call, not the receiver's | | `gitopsreverser_git_fetches_total` | `reason` | Unchanged. A delivery that finds a foreign tip on an idle branch fetches under `refresh` | @@ -453,7 +459,9 @@ In the bi-directional corner, against real Flux and Argo CD configured as the gu 4. **Catch-up.** After `quietPeriod` the `GitRepository` artifact holds the tip, and a forced `Kustomization` reconcile changes nothing and produces no commit. 5. **Refusal.** An eligible refused edit is reverted by `Reapply`, only its own applier is asked, - and an allowed edit made just before it reaches Git. + and an allowed edit made just before it reaches Git. A `Kustomization` sharing that applier's + `GitRepository` applies too when the source has to fetch, which is Flux's behavior and not a + failure of the request. 6. **No receiver.** A foreign push with no delivery at all is still applied once the refresh finds it. This is the guarantee that a missing webhook degrades to latency, never to loss, and it must stay green for the life of the feature. diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index 8407fdfd..dc8ffd35 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -822,7 +822,7 @@ and the push receiver tells it that somebody else pushed. See | Metric | Type | Labels | Meaning | | --- | --- | --- | --- | -| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per applier per request the branch worker decided to send. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (attempts exhausted, or the `Kustomization` cannot take a request) or `superseded` (a newer request for the same `Kustomization` replaced it before it went out) | +| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per applier per request the branch worker decided to send. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not be delivered and handed back to the branch worker, which asks again once the branch is quiet; one per hand-back), `superseded` (a newer request for the same `Kustomization` replaced it before it went out) or `dropped` (no `GitTarget` may still ask: each was deleted, moved to another branch or applier, or is no longer admitted by its `ClusterProvider`) | | `receiver_deliveries_total` | counter | `type`, `outcome` | One per verified-or-refused delivery to `/git-push/`. `type` is the `GitProvider.spec.receiver.type`; `outcome` is `notified` (handed to the branch worker), `unmatched` (no worker for that branch), `ignored` (a ping, a tag, another branch's event type, or a `generic-hmac` body naming another repository), `rejected` (bad signature or an oversized body) or `invalid` (verified but unparseable) | A delivery counted `notified` is only a hint: the worker reads the advertisement, and fetches under @@ -831,7 +831,8 @@ follows when it did. So on a branch where only GitOps Reverser pushes, `notified followed by `CatchUp` requests and never by `ForeignCommit`. ```promql -# Requests that could not be delivered, by reason. +# Requests that could not be delivered, by reason. A failure that persists keeps counting, at the +# worker's backoff of up to a few minutes, until the applier is fixed. sum by (reason) (rate(gitopsreverser_apply_requests_total{outcome="failed"}[15m])) # A receiver a Git host is calling with the wrong secret. sum by (type) (rate(gitopsreverser_receiver_deliveries_total{outcome="rejected"}[15m])) > 0 From cf1485ca94a365e7266b9cc81633c4aa22d7ef8a Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 19:22:07 +0000 Subject: [PATCH 13/36] fix(applier): keep status.applier per GitTarget incarnation The ledger behind status.applier was keyed by namespace and name and never pruned, so a GitTarget deleted and recreated under the same name inherited its predecessor's last request. It now remembers the uid it recorded for, and the GitTarget controller forgets an entry when the object is gone. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/applier/dispatcher.go | 43 +++++++++++++---- internal/applier/dispatcher_test.go | 48 +++++++++++++++---- internal/controller/gittarget_applier.go | 9 ++-- internal/controller/gittarget_applier_test.go | 7 ++- internal/controller/gittarget_controller.go | 4 ++ 5 files changed, 87 insertions(+), 24 deletions(-) diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index 007f0975..3775cd75 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -141,7 +141,7 @@ type Dispatcher struct { wake chan struct{} ledgerMu sync.RWMutex - ledger map[string]v1alpha3.GitTargetApplierStatus + ledger map[string]ledgerEntry // Owned by the Start loop. pending map[applierKey]*job @@ -172,7 +172,7 @@ func NewDispatcher( AttemptTimeout: DefaultAttemptTimeout, Now: time.Now, wake: make(chan struct{}, 1), - ledger: map[string]v1alpha3.GitTargetApplierStatus{}, + ledger: map[string]ledgerEntry{}, pending: map[applierKey]*job{}, inflight: map[applierKey]*job{}, results: make(chan result), @@ -239,17 +239,34 @@ func sameScope(a, b git.ApplyRequest) bool { return slices.Equal(keys(a.Targets), keys(b.Targets)) } +// ledgerEntry is the last request delivered for one GitTarget, and which incarnation of it the +// request was for. +type ledgerEntry struct { + uid types.UID + status v1alpha3.GitTargetApplierStatus +} + // LastRequest is the last apply request delivered for a GitTarget's applier, for status.applier. -// It is false until one was delivered. A CatchUp that found the applier already current is not a -// request made, and leaves it unchanged. -func (d *Dispatcher) LastRequest(ref itypes.ResourceReference) (v1alpha3.GitTargetApplierStatus, bool) { +// It is false until one was delivered for this uid: a GitTarget deleted and recreated under the +// same name does not inherit its predecessor's. A CatchUp that found the applier already current is +// not a request made, and leaves it unchanged. +func (d *Dispatcher) LastRequest( + ref itypes.ResourceReference, uid types.UID, +) (v1alpha3.GitTargetApplierStatus, bool) { d.ledgerMu.RLock() defer d.ledgerMu.RUnlock() - status, ok := d.ledger[ref.Key()] - if !ok { + entry, ok := d.ledger[ref.Key()] + if !ok || entry.uid != uid { return v1alpha3.GitTargetApplierStatus{}, false } - return *status.DeepCopy(), true + return *entry.status.DeepCopy(), true +} + +// Forget drops what the ledger holds for a GitTarget that is gone. +func (d *Dispatcher) Forget(ref itypes.ResourceReference) { + d.ledgerMu.Lock() + defer d.ledgerMu.Unlock() + delete(d.ledger, ref.Key()) } // applierKey identifies one applier for one branch. The branch is part of it so that two branches @@ -678,8 +695,14 @@ func (d *Dispatcher) delivered(j *job, eligible []eligibleTarget) { at := metav1.NewTime(d.Now()) d.ledgerMu.Lock() for _, e := range eligible { - d.ledger[itypes.NewResourceReference(e.target.Name, e.target.Namespace).Key()] = - v1alpha3.GitTargetApplierStatus{RequestedCommit: j.commit, RequestedAt: at.DeepCopy(), Reason: e.reason} + d.ledger[itypes.NewResourceReference(e.target.Name, e.target.Namespace).Key()] = ledgerEntry{ + uid: e.target.UID, + status: v1alpha3.GitTargetApplierStatus{ + RequestedCommit: j.commit, + RequestedAt: at.DeepCopy(), + Reason: e.reason, + }, + } } d.ledgerMu.Unlock() diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index 38f1037b..b27a1ac3 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -17,6 +17,7 @@ import ( sdkmetric "go.opentelemetry.io/otel/sdk/metric" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/dynamic" dynamicfake "k8s.io/client-go/dynamic/fake" clientgoscheme "k8s.io/client-go/kubernetes/scheme" @@ -40,7 +41,7 @@ var testBranchKey = git.BranchKey{RepoNamespace: targetNS, RepoName: "platform", // gitTarget builds a GitTarget on testBranchKey; a nil kustomizationRef declares no applier. func gitTarget(name string, kustomizationRef *fluxmeta.NamespacedObjectReference, cluster string) *v1alpha3.GitTarget { target := &v1alpha3.GitTarget{ - ObjectMeta: metav1.ObjectMeta{Namespace: targetNS, Name: name}, + ObjectMeta: metav1.ObjectMeta{Namespace: targetNS, Name: name, UID: uid(name)}, Spec: v1alpha3.GitTargetSpec{ GitProviderRef: fluxmeta.LocalObjectReference{Name: testBranchKey.RepoName}, Branch: testBranch, @@ -95,6 +96,9 @@ func ksRef(namespace, name string) *fluxmeta.NamespacedObjectReference { func ref(name string) itypes.ResourceReference { return itypes.NewResourceReference(name, targetNS) } +// uid is the uid gitTarget gives the GitTarget of that name. +func uid(name string) types.UID { return types.UID("uid-" + name) } + // sourceStub hands out one fake source cluster's client, failing the first failFirst calls (or // every call, with failFirst < 0), and remembers which cluster ids it was asked for. type sourceStub struct { @@ -259,13 +263,13 @@ func TestDispatcher_AsksEachApplierOnce(t *testing.T) { Reason: v1alpha3.ApplyReasonForeignCommit, } for _, name := range []string{"a", "b", "c"} { - got, ok := f.dispatcher.LastRequest(ref(name)) + got, ok := f.dispatcher.LastRequest(ref(name), uid(name)) require.True(t, ok, name) assert.True(t, got.RequestedAt.Equal(want.RequestedAt), name) assert.Equal(t, want.RequestedCommit, got.RequestedCommit, name) assert.Equal(t, want.Reason, got.Reason, name) } - _, ok := f.dispatcher.LastRequest(ref("no-applier")) + _, ok := f.dispatcher.LastRequest(ref("no-applier"), uid("no-applier")) assert.False(t, ok, "a target without an applier records no request") assert.ElementsMatch(t, []string{ @@ -296,11 +300,35 @@ func TestDispatcher_RetriesThenDelivers(t *testing.T) { assert.Equal(t, int64(0), f.count(v1alpha3.ApplyReasonRefusal, outcomeFailed)) assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches(), "a refusal at the artifact's commit stamps the kustomization") - got, ok := f.dispatcher.LastRequest(ref("a")) + got, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) require.True(t, ok) assert.Equal(t, v1alpha3.ApplyReasonRefusal, got.Reason) } +// TestDispatcher_TheLedgerIsPerIncarnation: a GitTarget deleted and recreated under the same name +// does not inherit its predecessor's last request, and a forgotten one holds nothing. +func TestDispatcher_TheLedgerIsPerIncarnation(t *testing.T) { + f := newDispatcherFixture(t, + []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit), + ) + f.start(t) + f.dispatcher.RequestApply(git.ApplyRequest{ + Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, + Targets: []itypes.ResourceReference{ref("a")}, + }) + f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeSent, 1) + + _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) + assert.True(t, ok) + _, ok = f.dispatcher.LastRequest(ref("a"), "the-recreated-one") + assert.False(t, ok, "a recreated GitTarget does not inherit the last request") + f.dispatcher.Forget(ref("a")) + _, ok = f.dispatcher.LastRequest(ref("a"), uid("a")) + assert.False(t, ok) +} + // TestDispatcher_HandsBackAfterItsRetryBudget is the review's finding 4: a request that kept // failing was given up on, while the worker had already cleared its debt, so nothing asked again // until an unrelated commit. It is now retried for about one quiet period and handed back to the @@ -325,7 +353,7 @@ func TestDispatcher_HandsBackAfterItsRetryBudget(t *testing.T) { Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, Targets: []itypes.ResourceReference{ref("a"), ref("b")}, HandBacks: 1, }, f.handBacks.got()[0], "every GitTarget goes back, with its reason and one more hand-back") - _, ok := f.dispatcher.LastRequest(ref("a")) + _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) assert.False(t, ok, "nothing was delivered") events := f.events() require.Len(t, events, 2, "one Warning per GitTarget") @@ -404,7 +432,7 @@ func TestDispatcher_ATargetItsClusterProviderDeniesAsksNothing(t *testing.T) { assert.Equal(t, []string{v1alpha3.DefaultClusterProviderName}, f.source.clusters, "the denied GitTarget's cluster client is never even asked for") - _, ok := f.dispatcher.LastRequest(ref("denied")) + _, ok := f.dispatcher.LastRequest(ref("denied"), uid("denied")) assert.False(t, ok) } @@ -483,10 +511,10 @@ func TestDispatcher_ARefusalNeedsConsentAtEveryAttempt(t *testing.T) { Targets: []itypes.ResourceReference{ref("suspended"), ref("consenting")}, }) f.waitForCount(t, v1alpha3.ApplyReasonRefusal, outcomeSent, 1) - consenting, ok := f.dispatcher.LastRequest(ref("consenting")) + consenting, ok := f.dispatcher.LastRequest(ref("consenting"), uid("consenting")) require.True(t, ok) assert.Equal(t, v1alpha3.ApplyReasonRefusal, consenting.Reason) - suspendedStatus, ok := f.dispatcher.LastRequest(ref("suspended")) + suspendedStatus, ok := f.dispatcher.LastRequest(ref("suspended"), uid("suspended")) require.True(t, ok, "the shared stamp reached its applier too") assert.Equal(t, v1alpha3.ApplyReasonCatchUp, suspendedStatus.Reason, "recorded as what it may ask for") @@ -534,7 +562,7 @@ func TestDispatcher_NewerRequestSupersedes(t *testing.T) { "the failed Refusal at the older commit was superseded, not retried") assert.Equal(t, int64(0), f.count(v1alpha3.ApplyReasonCatchUp, outcomeSent)) assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches()) - got, ok := f.dispatcher.LastRequest(ref("a")) + got, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) require.True(t, ok) assert.Equal(t, testCommit, got.RequestedCommit, "the newer commit is the one sent") assert.Equal(t, v1alpha3.ApplyReasonRefusal, got.Reason) @@ -558,7 +586,7 @@ func TestDispatcher_CatchUpAlreadyCurrent(t *testing.T) { assert.Empty(t, f.patches()) assert.Empty(t, f.events()) - _, ok := f.dispatcher.LastRequest(ref("a")) + _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) assert.False(t, ok) } diff --git a/internal/controller/gittarget_applier.go b/internal/controller/gittarget_applier.go index 29731c55..8fedba22 100644 --- a/internal/controller/gittarget_applier.go +++ b/internal/controller/gittarget_applier.go @@ -27,9 +27,11 @@ const GitTargetConditionApplierConfigured = "ApplierConfigured" const applierCheckTimeout = 10 * time.Second // ApplyRequestLedger is what the GitTarget controller reads to publish status.applier: the last -// apply request delivered for a GitTarget. The applier dispatcher implements it. +// apply request delivered for a GitTarget, for the incarnation with this uid. Forget drops a +// GitTarget that is gone. The applier dispatcher implements it. type ApplyRequestLedger interface { - LastRequest(ref types.ResourceReference) (configbutleraiv1alpha3.GitTargetApplierStatus, bool) + LastRequest(ref types.ResourceReference, uid k8stypes.UID) (configbutleraiv1alpha3.GitTargetApplierStatus, bool) + Forget(ref types.ResourceReference) } // observeApplier publishes status.applier and the ApplierConfigured condition, and returns the @@ -52,7 +54,8 @@ func (r *GitTargetReconciler) observeApplier( } if r.ApplyRequests != nil { - if last, ok := r.ApplyRequests.LastRequest(types.NewResourceReference(target.Name, target.Namespace)); ok { + ref := types.NewResourceReference(target.Name, target.Namespace) + if last, ok := r.ApplyRequests.LastRequest(ref, target.UID); ok { target.Status.Applier = &last } } diff --git a/internal/controller/gittarget_applier_test.go b/internal/controller/gittarget_applier_test.go index ee542f9f..da836d59 100644 --- a/internal/controller/gittarget_applier_test.go +++ b/internal/controller/gittarget_applier_test.go @@ -16,6 +16,7 @@ import ( "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" + k8stypes "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/dynamic" dynamicfake "k8s.io/client-go/dynamic/fake" @@ -59,11 +60,15 @@ func applierFakeSource(objs ...runtime.Object) *dynamicfake.FakeDynamicClient { type fixedLedger map[types.ResourceReference]configbutleraiv1alpha3.GitTargetApplierStatus -func (l fixedLedger) LastRequest(ref types.ResourceReference) (configbutleraiv1alpha3.GitTargetApplierStatus, bool) { +func (l fixedLedger) LastRequest( + ref types.ResourceReference, _ k8stypes.UID, +) (configbutleraiv1alpha3.GitTargetApplierStatus, bool) { status, ok := l[ref] return status, ok } +func (l fixedLedger) Forget(ref types.ResourceReference) { delete(l, ref) } + func applierTarget(applierSet bool) *configbutleraiv1alpha3.GitTarget { target := &configbutleraiv1alpha3.GitTarget{ ObjectMeta: metav1.ObjectMeta{Name: "checkout", Namespace: "team-a"}, diff --git a/internal/controller/gittarget_controller.go b/internal/controller/gittarget_controller.go index 91cd6642..35497478 100644 --- a/internal/controller/gittarget_controller.go +++ b/internal/controller/gittarget_controller.go @@ -1486,6 +1486,10 @@ func (r *GitTargetReconciler) cleanupDeletedGitTarget( // said, and an entry that outlives the object is a rate limit held against a name that may be // recreated tomorrow with nothing published. r.remotePublications.forget(gitDest) + // Same terms: the last apply request delivered for it. + if r.ApplyRequests != nil { + r.ApplyRequests.Forget(gitDest) + } telemetry.ForgetResourceConditions(conditionKindGitTarget, namespacedName.Namespace, namespacedName.Name) // Same terms, same reason: a join series that outlives its GitTarget keeps attributing a live // branch's push failures to an object that no longer exists. From afd7e834b79de2d9ba065fe1cba886d299e7d926 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 19:22:08 +0000 Subject: [PATCH 14/36] test(e2e): prove the Flux applier's source check, hand-back and receiver A Kustomization whose GitRepository follows another branch is reported SourceMismatch, a ForeignCommit for it never stamps that source, and the request is delivered once the source is fixed. A real Gitea push webhook to the operator's receiver Service makes Flux apply a Git-side push with nothing else on the branch. A unit test covers a second write inside the quiet period. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/apply-trigger-follow-ups.md | 59 ++--- internal/git/apply_trigger_test.go | 36 +++ test/e2e/flux_applier_e2e_test.go | 305 ++++++++++++++++++++++++ test/e2e/refusal_reapply_e2e_test.go | 6 +- 4 files changed, 367 insertions(+), 39 deletions(-) create mode 100644 test/e2e/flux_applier_e2e_test.go diff --git a/docs/design/apply-trigger-follow-ups.md b/docs/design/apply-trigger-follow-ups.md index a0bfbab1..2a8030c2 100644 --- a/docs/design/apply-trigger-follow-ups.md +++ b/docs/design/apply-trigger-follow-ups.md @@ -24,7 +24,7 @@ Built and tested: `GitRepository`, or the `Kustomization` when the artifact already holds the commit), the `Validate` behind `ApplierConfigured`, and the dispatcher (per-applier superseding, a check of every target before every attempt, a short retry and then a hand-back to the worker, Events, the - ledger behind `status.applier`). + ledger behind `status.applier`, kept per GitTarget uid and forgotten on deletion). - **The receiver**, `GitProvider.spec.receiver` on `--receiver-bind-address` ([`receiver.go`](../../internal/webhook/receiver.go)), which turns a verified delivery into a refresh that always reaches the remote. @@ -214,11 +214,6 @@ other end does not apply on its own. No in-process plugin, for the reason the tr ## 6. Smaller loose ends -- **The ledger behind `status.applier` is never pruned** - ([`dispatcher.go`](../../internal/applier/dispatcher.go)). It is keyed by namespace and name, so a - `GitTarget` deleted and recreated under the same name inherits its predecessor's last request. - Key it by UID, or forget an entry from the GitTarget controller's deleted-object path, the way - `remotePublications` does. - **The receiver listens on the elected replica only.** A standby refuses the connection, so a `Service` across replicas loses deliveries to it. With one replica (the default) this is invisible. Either bind on every replica and answer `503` until elected, which a Git host retries @@ -244,48 +239,44 @@ other end does not apply on its own. No in-process plugin, for the reason the tr ## 7. What the tests do not reach yet -Ordered by how much I would worry about each. +Ordered by how much I would worry about each. Two items left this list on 2026-10-16: +`Manager Flux Applier` (`test/e2e/flux_applier_e2e_test.go`) proves the receiver end to end (Gitea +delivers a signed push event to the operator's receiver Service, and Flux applies the push with +nothing else on the branch), and that a request for a `SourceMismatch` applier never stamps the +other branch's source and is delivered once the source is fixed. -1. **The receiver end to end.** The config-dir install runs the listener and a `Service`, but no - e2e spec sends a delivery. Discovery of a foreign push by a delivery is unit-tested over a real - Git server (`TestApplyTrigger_APushNoticeFindsAForeignCommit`) and the handler is unit-tested, - but nothing proves the two are wired together in a deployed operator. A spec should sign a - `generic-hmac` body with the route's token, POST it to the `Service` from inside the cluster, - and assert Flux applies a Git-side push within seconds, with no API edit to make it surface. -2. **A foreign push found by the refresher.** The Flux spec finds the foreign push through a +1. **A foreign push found by the refresher.** The Flux spec finds the foreign push through a rejected push, because the e2e refresh rides the five-minute reconcile tick. An idle branch with no receiver depends entirely on the refresh path, and nothing exercises it end to end. A reconcile-request annotation would not do (it forces a resync, the hazard of §2.2 of the design); the test needs a way to fire the refresh, or a shorter tick under test. -3. **The race spec's margin.** It edits twice a few seconds apart and asserts no extra commit for - 30 seconds. It caught the one-wake bug, which is the reason it exists, but it depends on the - corner's commit window and push cooldown. Write the unit equivalent with an injected clock: a - window that opens, closes, commits and pushes in one wake, followed by a second write inside - `quietPeriod`, asserting no request in between. Part of this is - `TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod`; the second-write - half is not. -4. **A remote source cluster.** Every test of the applier uses the operator's own cluster. A +2. **The race spec's margin.** It edits twice a few seconds apart and asserts no extra commit for + 30 seconds, which depends on the corner's commit window and push cooldown. The unit equivalent + now exists in two halves: + `TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod` and + `TestServiceAppliers_ASecondWriteInsideTheQuietPeriodRestartsIt`. Both move `quietSince` by hand + rather than injecting a clock. +3. **A remote source cluster.** Every test of the applier uses the operator's own cluster. A `ClusterProvider` with a kubeconfig, a `Kustomization` in that cluster, and the `Forbidden` path when the kubeconfig's identity lacks the grant, are all untested. -5. **`ApplierConfigured` against real RBAC.** e2e covers `True` only. `Forbidden` (drop the - flux-applier binding), `SourceMismatch` (point the `Kustomization` at another branch) and - `ApplierNotFound` (name one that does not exist, then create it, and watch the target recover) - are unit-tested with a fake access check. -6. **Several GitTargets naming one `Kustomization`.** The dispatcher's dedupe is unit-tested; no +4. **`ApplierConfigured` against real RBAC.** `ApplierNotFound` and `SourceMismatch` are covered + end to end by `Manager Flux Applier`. `Forbidden` (drop the flux-applier binding) is unit-tested + with a fake access check only. +5. **Several GitTargets naming one `Kustomization`.** The dispatcher's dedupe is unit-tested; no e2e has two folders on one branch applied by one `Kustomization`. -7. **A foreign commit while the remote is unreachable.** The rule is that the request waits with +6. **A foreign commit while the remote is unreachable.** The rule is that the request waits with the pending writes (§3.4 of the design). Nothing tests that it does not go out during the outage, or that it goes out once the push lands. The same test with a `Refusal` owed during the outage, and the target set to `Ignore` before the outage ends, covers §2.2 end to end. -8. **A foreign commit closing a window a `CommitRequest` is attached to.** +7. **A foreign commit closing a window a `CommitRequest` is attached to.** `closeWindowForForeignCommit` finalizes it early; the request should resolve `Committed` with the replayed commit. Untested. -9. **The CEL rule rejecting `Reapply` without an applier** is not exercised by an envtest. A stored +8. **The CEL rule rejecting `Reapply` without an applier** is not exercised by an envtest. A stored `PushEmptyCommit` needs no test, because no user has one (§4). -10. **The Helm install.** Local e2e is `INSTALL_MODE=config-dir`, so the chart's receiver flag, - `Service` and `applierAccess.flux` role are covered by `helm template` tests only. CI's - quickstart leg installs the chart, but with neither value set. -11. **The race detector in CI.** `applyDebt` is fed from `publishObservation`, which can run off +9. **The Helm install.** Local e2e is `INSTALL_MODE=config-dir`, so the chart's receiver flag, + `Service` and `applierAccess.flux` role are covered by `helm template` tests only. CI's + quickstart leg installs the chart, but with neither value set. +10. **The race detector in CI.** `applyDebt` is fed from `publishObservation`, which can run off the loop goroutine (the first clone), and the dispatcher delivers on goroutines of its own. Both are lock-guarded, and their tests pass under `-race -count=3` locally (2026-10-16), but CI does not run `-race`, so a later change can regress this unnoticed. diff --git a/internal/git/apply_trigger_test.go b/internal/git/apply_trigger_test.go index ce4d6b51..a29cd653 100644 --- a/internal/git/apply_trigger_test.go +++ b/internal/git/apply_trigger_test.go @@ -239,6 +239,42 @@ func TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod(t assert.NotNil(t, loop.applierTimer) } +// TestServiceAppliers_ASecondWriteInsideTheQuietPeriodRestartsIt is the other half of the race the +// bi-directional e2e spec covers with real timers: two API edits a few seconds apart, each decided +// and pushed in one wake. The catch-up owed for the first push must not go out between them, and +// when it goes out it names the second push. +func TestServiceAppliers_ASecondWriteInsideTheQuietPeriodRestartsIt(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true, func(g *configv1alpha3.GitTarget) { + g.Spec.Applier.QuietPeriod = &metav1.Duration{Duration: time.Minute} + })) + loop.w.debt.observe(RemoteObservation{Commit: "base", By: ObservedByFetch}) + loop.w.debt.take() + + pushInOneWake := func(commit string) { + loop.decide(PendingWrite{}) + loop.pendingWrites, loop.pendingWritesBytes = nil, 0 + loop.w.debt.observe(RemoteObservation{Commit: commit, By: ObservedByPush}) + loop.serviceAppliers() + } + + pushInOneWake("first") + loop.quietSince = time.Now().Add(-50 * time.Second) // ten seconds short of the quiet period + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "the first push's catch-up is not due yet") + + pushInOneWake("second") + loop.quietSince = time.Now().Add(-50 * time.Second) // past the first push's period, not the second's + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "the second write restarted the quiet period") + + loop.quietSince = time.Now().Add(-time.Minute) + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, configv1alpha3.ApplyReasonCatchUp, requests[0].Reason) + assert.Equal(t, "second", requests[0].Commit, "the catch-up names the later push, never the earlier") +} + func TestServiceAppliers_AForeignCommitAnswersEverythingElseOwed(t *testing.T) { loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) loop.oweReapply(refusalKeyFor(itypes.NewResourceReference("alpha", "team-a"), "deployments"), "refused", "obs") diff --git a/test/e2e/flux_applier_e2e_test.go b/test/e2e/flux_applier_e2e_test.go new file mode 100644 index 00000000..b588b996 --- /dev/null +++ b/test/e2e/flux_applier_e2e_test.go @@ -0,0 +1,305 @@ +// SPDX-License-Identifier: Apache-2.0 + +package e2e + +import ( + "crypto/rand" + "encoding/hex" + "fmt" + "os" + "path/filepath" + "strings" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" +) + +// GitTarget spec.applier against a real Flux, from what a request is allowed to do to how GitOps +// Reverser learns that it has to make one. See docs/design/apply-trigger-follow-ups.md. +// +// The Kustomization applies its own folder (fluxFolder) into a namespace the WatchRule does not +// watch, so nothing Flux applies is mirrored back and every commit on the branch is either this +// spec's API edit or this spec's Git-side push. Its GitRepository and Kustomization sit on a 24h +// interval and no Flux Receiver is wired to the repository, so any fetch or apply inside the spec +// is one GitOps Reverser asked for. +var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Ordered, func() { + const ( + providerName = "flux-applier-provider" + destName = "flux-applier-dest" + ruleName = "flux-applier-rule" + gitPath = "apps/applier" + fluxFolder = "flux/applier" + ) + var ( + testNs, appliedNs string + repo *RepoArtifacts + fluxName, fluxSecretName string + receiverSecretName string + receiverHookID int64 + ) + + BeforeAll(func() { + testNs = testNamespaceFor("manager-flux-applier") + appliedNs = testNs + "-applied" + fluxName = "flux-applier-" + testNs + fluxSecretName = fluxName + "-git" + receiverSecretName = providerName + "-receiver" + for _, ns := range []string{testNs, appliedNs} { + _, _ = kubectlRun("create", "namespace", ns) + } + + By("setting up the Gitea repo, its credentials and the GitProvider") + repo = SetupRepo(resolveE2EContext(), testNs, fmt.Sprintf("e2e-flux-applier-%d", GinkgoRandomSeed())) + _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) + createGitProviderWithURLInNamespace(providerName, testNs, repo.GitSecretHTTP, repo.RepoURLHTTP) + verifyResourceStatus("gitprovider", providerName, testNs, "True", "Succeeded", "") + configureRepoOriginWithCredentials(repo, testNs) + + By("seeding the folder Flux applies, on main and on a release branch") + pushAppliedColorFromOutside(repo, fluxFolder, appliedNs, "blue") + Expect(runGitIn(repo.CheckoutDir, "push", "origin", "HEAD:refs/heads/release")).To(Succeed()) + }) + + AfterAll(func() { + if receiverHookID != 0 { + gitea := giteaTestInstance() + ctx, cancel := gitea.Context() + _ = gitea.Client().DeleteRepoHook(ctx, gitea.Org, repo.RepoName, receiverHookID) + cancel() + } + cleanupWatchRule(ruleName, testNs) + cleanupGitTarget(destName, testNs) + _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", providerName, "--ignore-not-found=true") + cleanupNamespacedResource(testNs, "secret", receiverSecretName) + cleanupNamespacedResource("flux-system", "kustomization", fluxName) + cleanupNamespacedResource("flux-system", "gitrepository", fluxName) + cleanupNamespacedResource("flux-system", "secret", fluxSecretName) + cleanupNamespace(appliedNs) + cleanupNamespace(testNs) + }) + + // The review's findings 2 and 4 together, against the real thing: a Kustomization whose source + // follows another branch is reported, and a request for it never stamps that source, however + // it is triggered; and the request is not lost for it, but delivered once the source is fixed. + It("reports the applier from the source cluster, and never stamps a source on another branch", func() { + By("naming a Kustomization that does not exist yet") + createGitTarget(destName, testNs, providerName, gitPath, "main") + _, err := kubectlRunInNamespace(testNs, "patch", "gittarget", destName, "--type=merge", "-p", + fmt.Sprintf(`{"spec":{"applier":{"kustomizationRef":{"namespace":"flux-system","name":%q}}}}`, fluxName)) + Expect(err).NotTo(HaveOccurred(), "failed to set spec.applier") + verifyResourceCondition("gittarget", destName, testNs, "ApplierConfigured", "False", "ApplierNotFound", + "not found in the source cluster") + + By("creating it, with a GitRepository that follows the release branch") + applyFluxApplier(testNs, repo, fluxName, fluxSecretName, "release", "./"+fluxFolder) + verifyResourceCondition("gittarget", destName, testNs, "ApplierConfigured", "False", "SourceMismatch", + `fetches branch "release", not main`) + Expect(fluxRequestStamps(fluxName)).To(Equal("|"), "nothing has asked Flux for anything yet") + + By("mirroring a first API edit, so the branch has a tip GitOps Reverser pushed") + Expect(applyFromTemplate("test/e2e/templates/manager/watchrule-resources.tmpl", struct { + Name, Namespace, DestinationName, Resources string + }{Name: ruleName, Namespace: testNs, DestinationName: destName, Resources: `"configmaps"`}, + testNs)).To(Succeed(), "failed to apply the WatchRule") + applyConfigMap(testNs, "applier-first-edit") + Eventually(func(g Gomega) { + g.Expect(runGitIn(repo.CheckoutDir, "fetch", "origin", "main")).To(Succeed()) + g.Expect(gitShowFiles(repo.CheckoutDir)).To(ContainSubstring("applier-first-edit")) + }, 2*time.Minute, 2*time.Second).Should(Succeed()) + + // GitOps Reverser finds the Git-side push the hard way: its next push is rejected, and the + // fetch that follows is a ForeignCommit, which asks every applier on the branch. + By("pushing a Git-side change, then an API edit that finds it") + pushAppliedColorFromOutside(repo, fluxFolder, appliedNs, "green") + applyConfigMap(testNs, "applier-second-edit") + + By("the ForeignCommit is refused at the patch, and the source on release is never stamped") + Eventually(func(g Gomega) { + failed := gitTargetEventMessages(g, testNs, destName, "ApplyRequestFailed") + g.Expect(failed).To(ContainSubstring("(ForeignCommit)")) + g.Expect(failed).To(ContainSubstring(`fetches branch "release", not main`)) + }, 3*time.Minute, 2*time.Second).Should(Succeed()) + Consistently(func(g Gomega) { + g.Expect(fluxRequestStamps(fluxName)).To(Equal("|"), + "a GitRepository that follows another branch must never be stamped for this one") + }, 20*time.Second, 2*time.Second).Should(Succeed()) + + // A request that cannot be delivered goes back to the branch worker, which asks again on a + // quiet branch. Before, it was given up on, and nothing asked again until the next commit. + By("pointing the GitRepository at main, after which the handed-back request is delivered") + _, err = kubectlRunInNamespace("flux-system", "patch", "gitrepository", fluxName, "--type=merge", "-p", + `{"spec":{"ref":{"branch":"main"}}}`) + Expect(err).NotTo(HaveOccurred(), "failed to point the GitRepository at main") + verifyResourceCondition("gittarget", destName, testNs, "ApplierConfigured", "True", "Succeeded", "") + Eventually(func(g Gomega) { + g.Expect(gitTargetEventMessages(g, testNs, destName, "ApplyRequested")). + To(ContainSubstring("(ForeignCommit)"), "the ForeignCommit must not be lost to the failure") + g.Expect(fluxRequestStamps(fluxName)).NotTo(Equal("|")) + }, 5*time.Minute, 5*time.Second).Should(Succeed()) + Eventually(func(g Gomega) { + g.Expect(appliedColor(g, appliedNs)).To(Equal("green")) + }, 2*time.Minute, 2*time.Second).Should(Succeed()) + }) + + // The receiver's handler and the worker's discovery of a foreign push were each unit-tested, and + // nothing proved them wired together in a deployed operator. Here Gitea itself delivers the push + // event, signed, to the operator's receiver Service, and nothing else happens on the branch: no + // API edit surfaces the push, and the refresher's tick is minutes away. + It("applies a Git-side push the Git host reports, with nothing else to surface it", func() { + By("configuring the GitProvider's receiver, and registering it as a Gitea webhook") + token := randomToken() + _, err := kubectlRunInNamespace(testNs, "create", "secret", "generic", receiverSecretName, + "--from-literal=token="+token) + Expect(err).NotTo(HaveOccurred(), "failed to create the receiver Secret") + _, err = kubectlRunInNamespace(testNs, "patch", "gitprovider", providerName, "--type=merge", "-p", + fmt.Sprintf(`{"spec":{"receiver":{"type":"gitea","secretRef":{"name":%q}}}}`, receiverSecretName)) + Expect(err).NotTo(HaveOccurred(), "failed to set spec.receiver") + var webhookPath string + Eventually(func(g Gomega) { + out, err := kubectlRunInNamespace(testNs, "get", "gitprovider", providerName, + "-o", "jsonpath={.status.webhookPath}") + g.Expect(err).NotTo(HaveOccurred()) + webhookPath = strings.TrimSpace(out) + g.Expect(webhookPath).NotTo(BeEmpty()) + }, time.Minute, 2*time.Second).Should(Succeed()) + gitea := giteaTestInstance() + ctx, cancel := gitea.Context() + hook, err := gitea.Client().CreateGiteaWebhook(ctx, gitea.Org, repo.RepoName, + fmt.Sprintf("http://gitops-reverser-receiver.%s.svc.cluster.local%s", resolveE2ENamespace(), webhookPath), + token, []string{"push"}) + cancel() + Expect(err).NotTo(HaveOccurred(), "failed to register the receiver as a Gitea webhook") + receiverHookID = hook.ID + + Eventually(func(g Gomega) { + g.Expect(appliedColor(g, appliedNs)).To(Equal("green")) + }, 2*time.Minute, 2*time.Second).Should(Succeed(), "the previous spec left Flux at the tip") + foreignBefore := strings.Count( + gitTargetEventMessages(Default, testNs, destName, "ApplyRequested"), "(ForeignCommit)") + + By("pushing a Git-side change, and nothing else") + head := pushAppliedColorFromOutside(repo, fluxFolder, appliedNs, "purple") + + By("Flux applies it, because GitOps Reverser was told and asked") + Eventually(func(g Gomega) { + g.Expect(fluxLastAppliedRevision(g, fluxName)).To(ContainSubstring(head)) + g.Expect(appliedColor(g, appliedNs)).To(Equal("purple")) + }, 90*time.Second, 2*time.Second).Should(Succeed()) + Expect(strings.Count(gitTargetEventMessages(Default, testNs, destName, "ApplyRequested"), + "(ForeignCommit)")).To(BeNumerically(">", foreignBefore), + "the apply must come from a ForeignCommit request, not from Flux's own schedule") + }) +}) + +// applyFluxApplier creates a GitRepository following branch and a Kustomization applying path from +// it, both on a 24h interval, and waits for the Kustomization's first apply. +func applyFluxApplier(testNs string, repo *RepoArtifacts, name, secretName, branch, path string) { + GinkgoHelper() + // The in-cluster Gitea test credential, read from the cluster at runtime. + checkout := gitCheckout{secretNamespace: testNs, gitSecretName: repo.GitSecretHTTP} + username, password := checkout.readGitCredentialSecretDataBase64() + Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-gitrepository-http.tmpl", struct { + Namespace, SecretName, Name, RepoURL, Branch, Interval, Username, Password string + }{ + Namespace: "flux-system", SecretName: secretName, Name: name, RepoURL: repo.RepoURLHTTP, + Branch: branch, Interval: "24h", Username: username, + Password: password, + }, "flux-system")).To(Succeed(), "failed to apply the Flux GitRepository") + Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-kustomization.tmpl", struct { + Namespace, Name, Path, SourceName, Interval, DependsOn string + Prune, Wait bool + TargetNS string + HasTargetNS bool + DecryptionProvider, DecryptionSecretName string + }{ + Namespace: "flux-system", Name: name, Path: path, SourceName: name, Interval: "24h", + }, "flux-system")).To(Succeed(), "failed to apply the Flux Kustomization") + _, err := kubectlRunInNamespace("flux-system", "wait", "--for=condition=Ready", + "kustomization/"+name, "--timeout=3m") + Expect(err).NotTo(HaveOccurred(), "the Kustomization must apply its folder once") +} + +// pushAppliedColorFromOutside writes the ConfigMap the Kustomization applies, with the given +// color, and pushes it to main the way another writer would, leaving it checked out, and returns +// the commit it pushed. It retries, because GitOps Reverser may move the tip at any moment and losing that race says nothing +// about the behavior under test. +func pushAppliedColorFromOutside(repo *RepoArtifacts, folder, appliedNs, color string) string { + GinkgoHelper() + manifest := fmt.Sprintf(`apiVersion: v1 +kind: ConfigMap +metadata: + name: flux-applied + namespace: %s +data: + color: %s +`, appliedNs, color) + var head string + Eventually(func() error { + // The first push seeds an empty repository, which has no main to build on. + start := [][]string{{"checkout", "--orphan", "main"}, {"rm", "-rf", "--ignore-unmatch", "."}} + if _, err := gitRun(repo.CheckoutDir, "fetch", "origin", "main"); err == nil { + start = [][]string{{"checkout", "-B", "main", "origin/main"}, {"reset", "--hard", "origin/main"}} + } + for _, args := range start { + if out, err := gitRun(repo.CheckoutDir, args...); err != nil { + return fmt.Errorf("git %v: %w: %s", args, err, out) + } + } + dest := filepath.Join(repo.CheckoutDir, folder) + if err := os.MkdirAll(dest, 0o750); err != nil { + return err + } + if err := os.WriteFile(filepath.Join(dest, "configmap.yaml"), []byte(manifest), 0o600); err != nil { + return err + } + for _, args := range [][]string{ + {"add", folder}, + {"commit", "-m", "e2e: another writer sets the applied color to " + color}, + {"push", "origin", "HEAD:main"}, + } { + if out, err := gitRun(repo.CheckoutDir, args...); err != nil { + return fmt.Errorf("git %v: %w: %s", args, err, out) + } + } + out, err := gitRun(repo.CheckoutDir, "rev-parse", "HEAD") + head = strings.TrimSpace(out) + return err + }, seedPushTimeout, seedPushInterval).Should(Succeed(), "the outside writer kept losing the push race") + return head +} + +// appliedColor is the color of the ConfigMap the Kustomization applies. +func appliedColor(g Gomega, appliedNs string) string { + out, err := kubectlRunInNamespace(appliedNs, "get", "configmap", "flux-applied", "-o", "jsonpath={.data.color}") + g.Expect(err).NotTo(HaveOccurred()) + return strings.TrimSpace(out) +} + +// fluxLastAppliedRevision is the revision the Kustomization last applied. +func fluxLastAppliedRevision(g Gomega, name string) string { + out, err := kubectlRunInNamespace("flux-system", "get", "kustomization", name, + "-o", "jsonpath={.status.lastAppliedRevision}") + g.Expect(err).NotTo(HaveOccurred()) + return strings.TrimSpace(out) +} + +// gitTargetEventMessages is every message of the Events with reason recorded on a GitTarget. +func gitTargetEventMessages(g Gomega, testNs, gitTarget, reason string) string { + out, err := kubectlRunInNamespace(testNs, "get", "events", "--field-selector", + "involvedObject.kind=GitTarget,involvedObject.name="+gitTarget+",reason="+reason, + "-o", "jsonpath={.items[*].message}") + g.Expect(err).NotTo(HaveOccurred()) + return out +} + +// randomToken is a receiver secret for one spec. +func randomToken() string { + GinkgoHelper() + buf := make([]byte, 24) + _, err := rand.Read(buf) + Expect(err).NotTo(HaveOccurred()) + return hex.EncodeToString(buf) +} diff --git a/test/e2e/refusal_reapply_e2e_test.go b/test/e2e/refusal_reapply_e2e_test.go index edfe8655..450f341c 100644 --- a/test/e2e/refusal_reapply_e2e_test.go +++ b/test/e2e/refusal_reapply_e2e_test.go @@ -200,11 +200,7 @@ func applyRefusalFluxApplier(testNs string, repo *RepoArtifacts, name, secretNam // applyRequestedMessages is every ApplyRequested Event message recorded on a GitTarget. func applyRequestedMessages(g Gomega, testNs, gitTarget string) string { - out, err := kubectlRunInNamespace(testNs, "get", "events", "--field-selector", - "involvedObject.kind=GitTarget,involvedObject.name="+gitTarget+",reason=ApplyRequested", - "-o", "jsonpath={.items[*].message}") - g.Expect(err).NotTo(HaveOccurred()) - return out + return gitTargetEventMessages(g, testNs, gitTarget, "ApplyRequested") } // checkoutLogLevel reads the LOG_LEVEL env var of the live checkout Deployment. From e3c8df47c18bf05799af35b56250d9e5c345996c Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 19:51:17 +0000 Subject: [PATCH 15/36] test(e2e): prove the refresher alone gets a Git-side push applied With no receiver and no API edit, an idle GitTarget's steady-tick refresh finds an outside push, and Flux applies it from a ForeignCommit request: a missing webhook costs latency, never an apply. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/apply-trigger-follow-ups.md | 40 +++++++++----------- test/e2e/flux_applier_e2e_test.go | 49 +++++++++++++++++++------ 2 files changed, 56 insertions(+), 33 deletions(-) diff --git a/docs/design/apply-trigger-follow-ups.md b/docs/design/apply-trigger-follow-ups.md index 2a8030c2..c5c1c636 100644 --- a/docs/design/apply-trigger-follow-ups.md +++ b/docs/design/apply-trigger-follow-ups.md @@ -239,44 +239,40 @@ other end does not apply on its own. No in-process plugin, for the reason the tr ## 7. What the tests do not reach yet -Ordered by how much I would worry about each. Two items left this list on 2026-10-16: -`Manager Flux Applier` (`test/e2e/flux_applier_e2e_test.go`) proves the receiver end to end (Gitea +Ordered by how much I would worry about each. Three items left this list on 2026-10-16, all proved +by `Manager Flux Applier` (`test/e2e/flux_applier_e2e_test.go`): the receiver end to end (Gitea delivers a signed push event to the operator's receiver Service, and Flux applies the push with -nothing else on the branch), and that a request for a `SourceMismatch` applier never stamps the -other branch's source and is delivered once the source is fixed. - -1. **A foreign push found by the refresher.** The Flux spec finds the foreign push through a - rejected push, because the e2e refresh rides the five-minute reconcile tick. An idle branch with - no receiver depends entirely on the refresh path, and nothing exercises it end to end. A - reconcile-request annotation would not do (it forces a resync, the hazard of §2.2 of the design); - the test needs a way to fire the refresh, or a shorter tick under test. -2. **The race spec's margin.** It edits twice a few seconds apart and asserts no extra commit for +nothing else on the branch); a foreign push found by the refresher alone, with no receiver and no +API edit, on the steady tick of about five minutes; and a request for a `SourceMismatch` applier, +which never stamps the other branch's source and is delivered once the source is fixed. + +1. **The race spec's margin.** It edits twice a few seconds apart and asserts no extra commit for 30 seconds, which depends on the corner's commit window and push cooldown. The unit equivalent now exists in two halves: `TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod` and `TestServiceAppliers_ASecondWriteInsideTheQuietPeriodRestartsIt`. Both move `quietSince` by hand rather than injecting a clock. -3. **A remote source cluster.** Every test of the applier uses the operator's own cluster. A +2. **A remote source cluster.** Every test of the applier uses the operator's own cluster. A `ClusterProvider` with a kubeconfig, a `Kustomization` in that cluster, and the `Forbidden` path when the kubeconfig's identity lacks the grant, are all untested. -4. **`ApplierConfigured` against real RBAC.** `ApplierNotFound` and `SourceMismatch` are covered +3. **`ApplierConfigured` against real RBAC.** `ApplierNotFound` and `SourceMismatch` are covered end to end by `Manager Flux Applier`. `Forbidden` (drop the flux-applier binding) is unit-tested with a fake access check only. -5. **Several GitTargets naming one `Kustomization`.** The dispatcher's dedupe is unit-tested; no +4. **Several GitTargets naming one `Kustomization`.** The dispatcher's dedupe is unit-tested; no e2e has two folders on one branch applied by one `Kustomization`. -6. **A foreign commit while the remote is unreachable.** The rule is that the request waits with +5. **A foreign commit while the remote is unreachable.** The rule is that the request waits with the pending writes (§3.4 of the design). Nothing tests that it does not go out during the outage, or that it goes out once the push lands. The same test with a `Refusal` owed during the outage, and the target set to `Ignore` before the outage ends, covers §2.2 end to end. -7. **A foreign commit closing a window a `CommitRequest` is attached to.** +6. **A foreign commit closing a window a `CommitRequest` is attached to.** `closeWindowForForeignCommit` finalizes it early; the request should resolve `Committed` with the replayed commit. Untested. -8. **The CEL rule rejecting `Reapply` without an applier** is not exercised by an envtest. A stored +7. **The CEL rule rejecting `Reapply` without an applier** is not exercised by an envtest. A stored `PushEmptyCommit` needs no test, because no user has one (§4). -9. **The Helm install.** Local e2e is `INSTALL_MODE=config-dir`, so the chart's receiver flag, +8. **The Helm install.** Local e2e is `INSTALL_MODE=config-dir`, so the chart's receiver flag, `Service` and `applierAccess.flux` role are covered by `helm template` tests only. CI's quickstart leg installs the chart, but with neither value set. -10. **The race detector in CI.** `applyDebt` is fed from `publishObservation`, which can run off - the loop goroutine (the first clone), and the dispatcher delivers on goroutines of its own. Both - are lock-guarded, and their tests pass under `-race -count=3` locally (2026-10-16), but CI does - not run `-race`, so a later change can regress this unnoticed. +9. **The race detector in CI.** `applyDebt` is fed from `publishObservation`, which can run off + the loop goroutine (the first clone), and the dispatcher delivers on goroutines of its own. Both + are lock-guarded, and their tests pass under `-race -count=3` locally (2026-10-16), but CI does + not run `-race`, so a later change can regress this unnoticed. diff --git a/test/e2e/flux_applier_e2e_test.go b/test/e2e/flux_applier_e2e_test.go index b588b996..4de58a70 100644 --- a/test/e2e/flux_applier_e2e_test.go +++ b/test/e2e/flux_applier_e2e_test.go @@ -18,8 +18,8 @@ import ( // GitTarget spec.applier against a real Flux, from what a request is allowed to do to how GitOps // Reverser learns that it has to make one. See docs/design/apply-trigger-follow-ups.md. // -// The Kustomization applies its own folder (fluxFolder) into a namespace the WatchRule does not -// watch, so nothing Flux applies is mirrored back and every commit on the branch is either this +// The Kustomization applies its own folder (fluxApplierFolder) into a namespace the WatchRule does +// not watch, so nothing Flux applies is mirrored back and every commit on the branch is either this // spec's API edit or this spec's Git-side push. Its GitRepository and Kustomization sit on a 24h // interval and no Flux Receiver is wired to the repository, so any fetch or apply inside the spec // is one GitOps Reverser asked for. @@ -29,7 +29,6 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order destName = "flux-applier-dest" ruleName = "flux-applier-rule" gitPath = "apps/applier" - fluxFolder = "flux/applier" ) var ( testNs, appliedNs string @@ -59,7 +58,7 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order configureRepoOriginWithCredentials(repo, testNs) By("seeding the folder Flux applies, on main and on a release branch") - pushAppliedColorFromOutside(repo, fluxFolder, appliedNs, "blue") + pushAppliedColorFromOutside(repo, appliedNs, "blue") Expect(runGitIn(repo.CheckoutDir, "push", "origin", "HEAD:refs/heads/release")).To(Succeed()) }) @@ -94,7 +93,7 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order "not found in the source cluster") By("creating it, with a GitRepository that follows the release branch") - applyFluxApplier(testNs, repo, fluxName, fluxSecretName, "release", "./"+fluxFolder) + applyFluxApplier(testNs, repo, fluxName, fluxSecretName, "release", "./"+fluxApplierFolder) verifyResourceCondition("gittarget", destName, testNs, "ApplierConfigured", "False", "SourceMismatch", `fetches branch "release", not main`) Expect(fluxRequestStamps(fluxName)).To(Equal("|"), "nothing has asked Flux for anything yet") @@ -113,7 +112,7 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order // GitOps Reverser finds the Git-side push the hard way: its next push is rejected, and the // fetch that follows is a ForeignCommit, which asks every applier on the branch. By("pushing a Git-side change, then an API edit that finds it") - pushAppliedColorFromOutside(repo, fluxFolder, appliedNs, "green") + pushAppliedColorFromOutside(repo, appliedNs, "green") applyConfigMap(testNs, "applier-second-edit") By("the ForeignCommit is refused at the patch, and the source on release is never stamped") @@ -144,6 +143,31 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order }, 2*time.Minute, 2*time.Second).Should(Succeed()) }) + // With no receiver and no API edit, the only thing that can find a Git-side push is the + // refresher on the GitTarget's steady tick. This is the guarantee that a missing webhook costs + // latency and never an apply (docs/design/push-notification-and-reconcile-trigger.md §4.5 item 6). + // The e2e install refreshes after 30s, so the wait is the control plane's reconcile cadence of + // about five minutes, as in the refresh-remote spec. + It("applies a Git-side push the refresher finds, with no receiver and no API edit", func() { + Eventually(func(g Gomega) { + g.Expect(appliedColor(g, appliedNs)).To(Equal("green")) + }, 2*time.Minute, 2*time.Second).Should(Succeed(), "the previous spec left Flux at the tip") + foreignBefore := strings.Count( + gitTargetEventMessages(Default, testNs, destName, "ApplyRequested"), "(ForeignCommit)") + + By("pushing a Git-side change, and nothing else") + head := pushAppliedColorFromOutside(repo, appliedNs, "orange") + + By("the steady tick's refresh finds it, and Flux applies it") + Eventually(func(g Gomega) { + g.Expect(fluxLastAppliedRevision(g, fluxName)).To(ContainSubstring(head)) + g.Expect(appliedColor(g, appliedNs)).To(Equal("orange")) + }, 7*time.Minute, 5*time.Second).Should(Succeed()) + Expect(strings.Count(gitTargetEventMessages(Default, testNs, destName, "ApplyRequested"), + "(ForeignCommit)")).To(BeNumerically(">", foreignBefore), + "the apply must come from a ForeignCommit request, not from Flux's own schedule") + }) + // The receiver's handler and the worker's discovery of a foreign push were each unit-tested, and // nothing proved them wired together in a deployed operator. Here Gitea itself delivers the push // event, signed, to the operator's receiver Service, and nothing else happens on the branch: no @@ -175,13 +199,13 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order receiverHookID = hook.ID Eventually(func(g Gomega) { - g.Expect(appliedColor(g, appliedNs)).To(Equal("green")) + g.Expect(appliedColor(g, appliedNs)).To(Equal("orange")) }, 2*time.Minute, 2*time.Second).Should(Succeed(), "the previous spec left Flux at the tip") foreignBefore := strings.Count( gitTargetEventMessages(Default, testNs, destName, "ApplyRequested"), "(ForeignCommit)") By("pushing a Git-side change, and nothing else") - head := pushAppliedColorFromOutside(repo, fluxFolder, appliedNs, "purple") + head := pushAppliedColorFromOutside(repo, appliedNs, "purple") By("Flux applies it, because GitOps Reverser was told and asked") Eventually(func(g Gomega) { @@ -194,6 +218,9 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order }) }) +// fluxApplierFolder is the folder the Manager Flux Applier spec's Kustomization applies. +const fluxApplierFolder = "flux/applier" + // applyFluxApplier creates a GitRepository following branch and a Kustomization applying path from // it, both on a 24h interval, and waits for the Kustomization's first apply. func applyFluxApplier(testNs string, repo *RepoArtifacts, name, secretName, branch, path string) { @@ -226,7 +253,7 @@ func applyFluxApplier(testNs string, repo *RepoArtifacts, name, secretName, bran // color, and pushes it to main the way another writer would, leaving it checked out, and returns // the commit it pushed. It retries, because GitOps Reverser may move the tip at any moment and losing that race says nothing // about the behavior under test. -func pushAppliedColorFromOutside(repo *RepoArtifacts, folder, appliedNs, color string) string { +func pushAppliedColorFromOutside(repo *RepoArtifacts, appliedNs, color string) string { GinkgoHelper() manifest := fmt.Sprintf(`apiVersion: v1 kind: ConfigMap @@ -248,7 +275,7 @@ data: return fmt.Errorf("git %v: %w: %s", args, err, out) } } - dest := filepath.Join(repo.CheckoutDir, folder) + dest := filepath.Join(repo.CheckoutDir, fluxApplierFolder) if err := os.MkdirAll(dest, 0o750); err != nil { return err } @@ -256,7 +283,7 @@ data: return err } for _, args := range [][]string{ - {"add", folder}, + {"add", fluxApplierFolder}, {"commit", "-m", "e2e: another writer sets the applied color to " + color}, {"push", "origin", "HEAD:main"}, } { From ffe95035f115e8ea5f1353384929994f3d291fae Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 20:49:31 +0000 Subject: [PATCH 16/36] docs: add the Flux user guide, with its example held to the code docs/flux.md is the complete Flux setup: why the GitRepository must not fetch on its own, a worked example, the rules ApplierConfigured checks, permissions, the Git host's webhook, what a request does and that it is asynchronous, how to check it works, troubleshooting, and which test proves each claim. A unit test reads the guide's complete example and checks it against the applier: the objects decode with no unknown field, ApplierConfigured comes out True, a request stamps the GitRepository, and the Flux objects follow the page's own advice. The bi-directional guide's Flux section now points to it, and the chart's applierAccess comment matches the role. Co-Authored-By: Claude Opus 5.5 (1M context) --- .docs-lint-scope | 1 + charts/gitops-reverser/values.yaml | 6 +- docs/README.md | 1 + docs/bi-directional.md | 39 +--- docs/configuration.md | 3 +- docs/flux.md | 330 ++++++++++++++++++++++++++++ internal/applier/flux_guide_test.go | 151 +++++++++++++ 7 files changed, 497 insertions(+), 34 deletions(-) create mode 100644 docs/flux.md create mode 100644 internal/applier/flux_guide_test.go diff --git a/.docs-lint-scope b/.docs-lint-scope index 2d6ff11b..0c1fd9ba 100644 --- a/.docs-lint-scope +++ b/.docs-lint-scope @@ -25,6 +25,7 @@ README.md docs/architecture.md docs/configuration.md +docs/flux.md docs/spec/attribution.md docs/components.md docs/quickstart.md diff --git a/charts/gitops-reverser/values.yaml b/charts/gitops-reverser/values.yaml index f546f1e8..86291072 100644 --- a/charts/gitops-reverser/values.yaml +++ b/charts/gitops-reverser/values.yaml @@ -129,9 +129,9 @@ receiver: # somebody else pushed to a branch a GitTarget writes. Off unless asked for, since most installs run # one applier or none. applierAccess: - # Grant get/list/watch/patch on kustomize.toolkit.fluxcd.io Kustomizations and - # source.toolkit.fluxcd.io GitRepositories, plus create on SelfSubjectAccessReviews (to check that - # access before relying on it). Cluster-wide, because a GitTarget's applier can live in any + # Grant get/patch on kustomize.toolkit.fluxcd.io Kustomizations and source.toolkit.fluxcd.io + # GitRepositories, plus create on SelfSubjectAccessReviews (to check that access before relying + # on it). Cluster-wide, because a GitTarget's applier can live in any # namespace. flux: false diff --git a/docs/README.md b/docs/README.md index 8aef271f..de359711 100644 --- a/docs/README.md +++ b/docs/README.md @@ -29,6 +29,7 @@ If you only want the supported product docs, start with the files below. credentials Secret shape - [`rbac.md`](rbac.md): the two ClusterRoles, and how to stop the reverser enumerating Secrets - [`bi-directional.md`](bi-directional.md): safe shared-path and handoff patterns +- [`flux.md`](flux.md): the complete Flux setup when GitOps Reverser decides when Flux applies - [`alternatives.md`](alternatives.md): nearby tools and when another approach fits better - [`UPGRADING.md`](UPGRADING.md): breaking changes and migration steps, newest first - [`../CONTRIBUTING.md`](../CONTRIBUTING.md): contributor workflow and validation commands diff --git a/docs/bi-directional.md b/docs/bi-directional.md index aa70ac20..82bd7a3c 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -193,36 +193,14 @@ field from GitOps Reverser's whole-object replay. ## Configure Flux -Flux has no refresh-only mode: kustomize-controller applies whenever it reconciles, and -`suspend: true` also blocks the requests GitOps Reverser makes. So the configuration keeps Flux's -timers as a long backstop and removes every short path: - -```yaml -# GitRepository: long. GitOps Reverser requests the fetches. -spec: - interval: 24h ---- -# Kustomization: long, and unsuspended. -spec: - interval: 24h - retryInterval: 1m - suspend: false -``` - -Remove any Git-host webhook into a Flux `Receiver` for this repository. GitOps Reverser's request is -the one `flux reconcile kustomization --with-source` makes: it stamps -`reconcile.fluxcd.io/requestedAt` on the `GitRepository`, and stamps the `Kustomization` too when -the artifact already holds the requested commit. - -Flux needs one more request than the cluster's own state suggests. Its timers and a controller restart apply whatever artifact -the `GitRepository` holds, and with a long interval that artifact falls behind GitOps Reverser's own -pushes. Applying a stale artifact would revert published edits, not only pending ones. So after its -own pushes, once the branch has been quiet for `spec.applier.quietPeriod` (`30s` by default), -GitOps Reverser asks the `GitRepository` to fetch, with reason `CatchUp`. The new artifact wakes the -`Kustomization`, which applies values the cluster already holds. - -`quietPeriod` is a trade. A short one keeps the artifact current; a long one makes it less likely -that an edit arrives while that catch-up apply is running. Every new edit restarts it. +[Using GitOps Reverser with Flux](flux.md) is the complete setup, with a worked example a unit test +holds to the code. In short: Flux has no fetch-only mode (a new `GitRepository` artifact makes every +`Kustomization` reading it apply, and `suspend: true` also blocks GitOps Reverser's requests), so +set both intervals to a long backstop such as `24h`, keep the `Kustomization` unsuspended, and +remove any Git-host webhook into a Flux `Receiver` for the repository. GitOps Reverser then asks the +`GitRepository` to fetch: after a foreign push, after a refusal, and, once the branch has been +quiet for `spec.applier.quietPeriod` (`30s` by default), to catch up with its own pushes, so a +Flux restart never applies a stale artifact. Flux's equivalent of split ownership is `Kustomization.spec.ignore`, added in kustomize-controller `v1.9.0`, with the same trade as Argo CD's. The @@ -471,6 +449,7 @@ concurrent writers. | A Git-side push with an API edit on top lands both, and Flux applies the replayed commit without being reconciled by hand | Flux | | A new `GitRepository` revision is enough for the `Kustomization` to apply and revert drift | Flux | | A refused edit with `onRefusal: Reapply` is reverted by its `Kustomization`, with no commit, and settles instead of repeating | [Refusal reapply](../test/e2e/refusal_reapply_e2e_test.go) | +| A `SourceMismatch` applier is never stamped, a push the Git host reports or the refresh finds is applied, without the corner | [Manager Flux Applier](../test/e2e/flux_applier_e2e_test.go), and [the Flux guide](flux.md#what-the-tests-prove) | | A Git to cluster to Git round trip settles without a commit loop | Flux | | A SOPS-encrypted Secret round-trips without a reverse commit | Flux | | A Git revert removes the live object and the committed file | Flux | diff --git a/docs/configuration.md b/docs/configuration.md index 8474b602..e26cb969 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1110,7 +1110,8 @@ spec: | `quietPeriod` | `30s` | How long the branch must stay quiet after GitOps Reverser's own pushes before a `CatchUp` request. At most `1h` | Argo CD (`applicationRef`) and a webhook for any other tool are the next appliers; until they -exist, an Argo CD `Application` keeps its own triggers. +exist, an Argo CD `Application` keeps its own triggers. [Using GitOps Reverser with Flux](flux.md) +walks through the whole Flux setup, from the `GitRepository` to the Git host's webhook. **Where the reference is looked up.** In the source cluster, the cluster the `Kustomization` applies to, with the identity the [`ClusterProvider`](#clusterprovider) already uses. That identity needs `get` diff --git a/docs/flux.md b/docs/flux.md new file mode 100644 index 00000000..479ca261 --- /dev/null +++ b/docs/flux.md @@ -0,0 +1,330 @@ +# Using GitOps Reverser with Flux + +When GitOps Reverser publishes a folder that Flux applies, GitOps Reverser decides when Flux +applies it. Flux still fetches, renders, applies, prunes and reports health; it applies when +GitOps Reverser asks, instead of on its own timers or on a Git-host webhook. This page is the +complete Flux setup: what to change on the `GitRepository` and the `Kustomization`, what to declare +on the `GitTarget`, where the Git host's webhook goes, how to check it works, and which tests prove +each part. + +The [bi-directional guide](bi-directional.md) explains the model behind this for any applier, and +the [configuration reference](configuration.md#who-applies-this-folder-specapplier) has every field +and condition reason. + +## What changes + +| Before | With GitOps Reverser | +| --- | --- | +| The `GitRepository` fetches every minute or so | It fetches when GitOps Reverser asks; its `interval` is a 24h backstop | +| The `Kustomization` applies on its interval | It applies when the new artifact arrives, or when GitOps Reverser asks; its `interval` is a 24h backstop | +| The Git host's push webhook goes to a Flux `Receiver` | It goes to GitOps Reverser's receiver; the Flux `Receiver` for this repository is removed | +| Nothing ties a folder to its `Kustomization` | The `GitTarget` names it in `spec.applier.kustomizationRef` | + +## Why Flux must not fetch on its own + +**A new `GitRepository` artifact makes every `Kustomization` that reads it apply, at once.** Flux +has no mode that fetches without applying, and `suspend: true` on the `Kustomization` also blocks +the requests GitOps Reverser makes. So a source that fetches on its own, on a short interval or on a +webhook into a Flux `Receiver`, is a `Kustomization` that applies on its own. That brings back the +two races GitOps Reverser exists to close: + +- **Its own pushes.** Two API edits a few seconds apart produce two commits. A fetch between them + applies the first commit over the second edit, and GitOps Reverser then publishes the revert as a + third commit. +- **Somebody else's push.** While an API edit is still in an open commit window, a Git-side push + arrives. Applied straight away, the push's tree reverts the edit, because the edit is not in Git + yet. GitOps Reverser instead closes the window, replays the edit onto the push, pushes the + result, and only then asks Flux to apply. + +Both apply something correct; they apply it at the wrong moment. Only GitOps Reverser can see the +edits that are live and not yet pushed, so only it can tell when an apply is safe. It sends a +request only on a **quiet branch**: no open commit window and no pending writes. + +The source still stays current. After GitOps Reverser's own pushes, once the branch has been quiet +for `spec.applier.quietPeriod`, it asks the `GitRepository` to fetch if its artifact is behind +(reason `CatchUp`). That matters because Flux still applies on a controller restart and on its long +interval, from whatever artifact it holds, and an artifact left behind would revert edits that are +already in Git. + +## A complete example + +One folder, `apps/checkout` on branch `main` of `github.com/acme/platform`, published by GitOps +Reverser and applied by the `Kustomization` `flux-system/checkout`. Every object below is checked by +a unit test against the code that validates and drives the applier (see +[what the tests prove](#what-the-tests-prove)), so this example cannot drift from what is accepted. + +The Flux side. The `GitRepository` follows exactly the branch the `GitTarget` writes, and both +intervals are long: + +```yaml +apiVersion: source.toolkit.fluxcd.io/v1 +kind: GitRepository +metadata: + name: platform + namespace: flux-system +spec: + url: https://github.com/acme/platform + ref: + branch: main + interval: 24h + secretRef: + name: platform-git +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: checkout + namespace: flux-system +spec: + sourceRef: + kind: GitRepository + name: platform + path: ./apps/checkout + prune: true + interval: 24h + retryInterval: 1m +``` + +The GitOps Reverser side. The `GitProvider` points at the same repository (any URL form of it) and +serves a push receiver; the `GitTarget` names the `Kustomization` as its applier: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-receiver + namespace: team-a +stringData: + token: replace-with-a-long-random-string +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitProvider +metadata: + name: platform + namespace: team-a +spec: + url: git@github.com:acme/platform.git + allowedBranches: ["main"] + secretRef: + name: platform-git + receiver: + type: github + secretRef: + name: platform-receiver +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitTarget +metadata: + name: checkout + namespace: team-a +spec: + gitProviderRef: + name: platform + branch: main + path: apps/checkout + applier: + kustomizationRef: + namespace: flux-system + name: checkout + quietPeriod: 30s + onRefusal: Reapply +``` + +No Flux `Receiver` for `acme/platform` appears, on purpose: remove yours if you have one. + +## The `GitRepository` + +`ApplierConfigured` is `True` only when the `Kustomization`'s source fetches the branch the +`GitTarget` writes. Each rule below is checked when the target reconciles and again before every +request, so a source changed after the check is still never stamped for the wrong branch. + +| Rule | Reported when broken | +| --- | --- | +| `spec.url` is the `GitProvider`'s repository. The forms compare as one repository: `https://` and `ssh://`, `git@host:org/repo` (SCP style), a trailing `.git` or `/`, letter case, and the default port | `SourceMismatch`: `fetches , not ` | +| `spec.ref.branch` is exactly the `GitTarget`'s `spec.branch` | `SourceMismatch`: `fetches branch "release", not main` | +| `spec.ref` pins nothing stronger: no `commit`, `name`, `semver` or `tag`, even alongside the branch. Flux resolves those before the branch, so the artifact would never follow it | `SourceMismatch`: `pins spec.ref.tag "v1.0.0", so it never follows branch main` | +| `spec.ref` is set | `SourceMismatch`: `names no spec.ref.branch` | + +Set `spec.interval` long, `24h` in the example. It is Flux's backstop for the case where GitOps +Reverser is down; a short one is a source that fetches, and so applies, on its own. + +## The `Kustomization` + +- `spec.sourceRef` names a `GitRepository`. A `Kustomization` that applies from an `OCIRepository` + or a `Bucket` reports `SourceMismatch`: GitOps Reverser writes Git, and can only ask a Git source + to fetch. +- `spec.interval` is long, for the same reason as the source's. +- `spec.suspend` is not `true`. A suspended `Kustomization` ignores the requests. +- `spec.retryInterval` is how soon Flux retries an apply that **failed**, on its own. Keep it, since + a failed apply otherwise waits out the 24h interval, and know that this one retry is not gated on + a quiet branch. +- It may live in any namespace (`flux-system` is the usual one). The reference's namespace defaults + to the `GitTarget`'s own. A request reaches only a `Kustomization` whose source fetches this + `GitProvider`'s repository and branch, so naming one elsewhere grants nothing that write access to + the repository does not already give. +- Several `GitTarget`s may name one `Kustomization`, for example one whose `path` covers several + folders. It is asked once per request, however many of them share it. + +## The `GitTarget` + +```yaml +spec: + applier: + kustomizationRef: + namespace: flux-system # default: this GitTarget's namespace + name: checkout + quietPeriod: 30s # default 30s, at most 1h +``` + +`quietPeriod` is how long the branch must stay quiet after GitOps Reverser's own pushes before Flux +is asked to catch up with them. Every new write restarts it. A short one keeps Flux's artifact +current; a long one makes it less likely that an edit arrives while that catch-up apply runs. + +The target reports `ApplierConfigured`, and `Ready` includes it: + +| Reason | Means | Readiness | +| --- | --- | --- | +| `Succeeded` | The `Kustomization` exists, applies this repository and branch, and may be patched | ready | +| `ApplierNotFound` | No such `Kustomization` in the source cluster, or Flux is not installed there | progressing: Flux objects are often created after the `GitTarget` | +| `SourceMismatch` | One of the [`GitRepository` rules](#the-gitrepository) is broken, or the source is not a `GitRepository` | stalled: needs a human | +| `Forbidden` | The source-cluster identity may not patch the `Kustomization` or its `GitRepository` | stalled: needs a human | + +## Permissions + +GitOps Reverser looks the `Kustomization` up in the **source cluster** (the cluster it applies to) +with the identity of the target's `ClusterProvider`. That identity needs: + +| API group | Resource | Verbs | +| --- | --- | --- | +| `kustomize.toolkit.fluxcd.io` | `kustomizations` | `get`, `patch` | +| `source.toolkit.fluxcd.io` | `gitrepositories` | `get`, `patch` | +| `authorization.k8s.io` | `selfsubjectaccessreviews` | `create`, to check the two above | + +For the operator's own cluster, install the chart with `applierAccess.flux: true`, or apply +[`config/rbac/flux-applier-role.yaml`](../config/rbac/flux-applier-role.yaml) and its binding. For +a remote source cluster, grant the same to the identity in the `ClusterProvider`'s kubeconfig. + +## The Git host's webhook + +Point the Git host's push webhook at GitOps Reverser, so a push reaches the cluster in seconds, +after GitOps Reverser has replayed any pending edits onto it. + +1. Serve the receiver: set the chart's `receiver.bindAddress` (for example `":9292"`). The chart + adds a `-receiver` `Service`. The listener is plain HTTP; expose it to your Git host + with an Ingress or Gateway route that terminates TLS, the way Flux's `webhook-receiver` is + exposed. +2. Create a Secret with the webhook secret under the key `token`, in the `GitProvider`'s namespace. + It is the key a Flux `Receiver` Secret uses, so an existing one can be copied over. +3. Set `GitProvider.spec.receiver` with that Secret and the host's `type`: `github`, `gitlab`, + `gitea`, or `generic-hmac` for a relay of your own. The values are Flux `Receiver` types. +4. Read the path from `status.webhookPath` and register `https://` as a push + webhook with the same secret: + + ```bash + kubectl get gitprovider platform -n team-a -o jsonpath='{.status.webhookPath}' + # /git-push/team-a/platform + ``` + +5. Delete the Flux `Receiver` for this repository and its webhook at the Git host. + +A delivery is only a hint. GitOps Reverser reads where the branch is from the remote and +decides from that, so a delivery for one of its own pushes, a duplicate, or a late one asks Flux for +nothing it should not. Without a receiver everything still works, later: a foreign push is found by +the next rejected push or by the periodic refresh (`git.refreshInterval`, `10m`, on the 5-minute +reconcile tick). + +## What GitOps Reverser asks, and when + +| Reason | When | Asked | +| --- | --- | --- | +| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Pending edits are replayed onto it and pushed first | Every `Kustomization` declared on the branch | +| `Refusal` | An edit was refused and the target has `onRefusal: Reapply` | The refusing target's `Kustomization` | +| `CatchUp` | The branch has been quiet for `quietPeriod` after GitOps Reverser's own pushes, or since the operator started | Every `Kustomization` declared on the branch | + +A request is what `flux reconcile kustomization --with-source` does, minus the waiting. GitOps +Reverser merge-patches `reconcile.fluxcd.io/requestedAt` on the `GitRepository` when its artifact +does not hold the commit; the new artifact then wakes the `Kustomization`. When the artifact +already holds it, it stamps the `Kustomization` instead, and a `CatchUp` stamps nothing at all. + +**It is asynchronous.** The patch returns as soon as the API server accepts it, and nothing waits +for Flux to fetch or apply. It runs off the branch worker, so a slow source cluster never holds a +publication, and a failed request never fails one. + +Before every attempt, retries included, GitOps Reverser reads the `GitTarget` again: it must still +name this `Kustomization` on this branch, its `ClusterProvider` must still admit it, and a +`Refusal` must still have consent. A request that fails is retried for about one `quietPeriod`, +because a retry skips the quiet check, and is then handed back to the branch worker, which asks +again once the branch is quiet, waiting longer each time up to five minutes. A failure that cannot +be retried, such as a `SourceMismatch`, is handed back at once and delivered once it is fixed. + +## Reverting refused edits + +With `onRefusal: Reapply`, an edit GitOps Reverser cannot write (one that has no home in the folder, +such as a field a kustomize base owns) is reverted by asking the target's `Kustomization` to apply +the tip, once the branch is quiet. Only that `Kustomization` is asked. If its `GitRepository` has to +fetch to reach the tip, though, every `Kustomization` on that `GitRepository` applies the new +revision, as it would for any push. Give the folder its own `GitRepository` if the revert must reach +that folder alone. + +[Reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal) has the +eligibility rules. + +## Checking that it works + +```bash +# The applier is found, applies this repository and branch, and may be patched. +kubectl get gittarget checkout -n team-a \ + -o jsonpath='{.status.conditions[?(@.type=="ApplierConfigured")]}{"\n"}' + +# What GitOps Reverser last asked Flux for, and why. +kubectl get gittarget checkout -n team-a -o jsonpath='{.status.applier}{"\n"}' +# {"reason":"ForeignCommit","requestedAt":"2026-10-20T10:14:03Z","requestedCommit":"4f2c1ab9..."} + +# Each request, and each request that could not be delivered. +kubectl get events -n team-a --field-selector involvedObject.name=checkout,involvedObject.kind=GitTarget + +# What Flux actually applied: a request asks, it never reports. +kubectl get kustomization checkout -n flux-system -o jsonpath='{.status.lastAppliedRevision}{"\n"}' +``` + +`gitopsreverser_apply_requests_total{reason,outcome}` counts the requests; +[interpreting metrics](interpreting-metrics.md#apply-requests-and-the-push-receiver) reads it. + +## Troubleshooting + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `ApplierConfigured=False`, `ApplierNotFound` | The `Kustomization` does not exist in the source cluster under that namespace and name, or Flux is not installed there | Create it, or fix `kustomizationRef`; an omitted namespace is the `GitTarget`'s | +| `ApplierConfigured=False`, `SourceMismatch` | The message names the broken [`GitRepository` rule](#the-gitrepository) | Fix the `GitRepository`; the request it was holding is delivered afterward | +| `ApplierConfigured=False`, `Forbidden` | The identity lacks one of the [permissions](#permissions) | Grant it, or set `applierAccess.flux: true` | +| A Warning `ApplyRequestFailed` Event | A request could not be delivered; the message says why | GitOps Reverser keeps asking on a quiet branch, at most every five minutes. It is recorded once per failing request | +| Flux applies while edits are in flight | Something else still makes the source fetch: a short `interval`, a Flux `Receiver`, or someone running `flux reconcile` | Lengthen the intervals and remove the Flux `Receiver` | +| A Git-side push takes minutes to reach the cluster | No receiver, so the refresh finds it | Configure [the webhook](#the-git-hosts-webhook) | +| A refused edit is not reverted | The target is suspended, `onRefusal` is not `Reapply`, or the refusal is not eligible | See [reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal) | + +## While GitOps Reverser is down + +GitOps Reverser becomes the only fast path to a deployment. While it is down, a Git-side push +reaches the cluster through `flux reconcile kustomization --with-source`, which is safe then +because nothing is publishing, or through the 24h backstop. When it starts again, its first look at +the remote owes every `Kustomization` a `CatchUp`, never a `ForeignCommit`: an apply on startup +could revert edits made while it was down, before its watches have caught them. + +## What the tests prove + +| Covered | Where | +| --- | --- | +| The [complete example](#a-complete-example) decodes strictly, reports `ApplierConfigured=True`, and its request stamps the `GitRepository`; the Flux objects follow this page's advice | [`flux_guide_test.go`](../internal/applier/flux_guide_test.go) | +| `ApplierNotFound`, then `SourceMismatch` for a source on another branch, reported from a running source cluster | [Manager Flux Applier](../test/e2e/flux_applier_e2e_test.go) | +| A `ForeignCommit` for a `SourceMismatch` applier never stamps the other branch's source, records `ApplyRequestFailed`, and is delivered once the source is fixed | Manager Flux Applier | +| A Gitea push webhook to the operator's receiver gets a Git-side push applied, with nothing else on the branch | Manager Flux Applier | +| With no receiver and no API edit, the refresh finds a Git-side push and Flux applies it | Manager Flux Applier | +| Two API edits seconds apart end at the later value with no extra commit, and Flux catches up by itself | [Bi-directional (Flux)](../test/e2e/flux_bi_directional_e2e_test.go) | +| A Git-side push with an API edit on top lands both, and Flux applies the replayed commit | Bi-directional (Flux) | +| A new `GitRepository` revision alone makes the `Kustomization` apply, which is why the source must not fetch on its own | Bi-directional (Flux) | +| A refused edit with `onRefusal: Reapply` is reverted by its `Kustomization`, with no commit, and settles | [Refusal reapply](../test/e2e/refusal_reapply_e2e_test.go) | +| Every attempt checks the target again (admission, branch, consent) and the source binding before it stamps; a failed request goes back to the branch worker, which asks again | [`dispatcher_test.go`](../internal/applier/dispatcher_test.go), [`apply_handback_test.go`](../internal/git/apply_handback_test.go) | + +`task test-e2e` runs the Manager specs, and `task test-e2e-bi-directional` runs the bi-directional +corner. Not covered end to end yet: a remote source cluster, `Forbidden` against cluster RBAC, and the +receiver installed by the Helm chart rather than the e2e manifests. diff --git a/internal/applier/flux_guide_test.go b/internal/applier/flux_guide_test.go new file mode 100644 index 00000000..7a8fbf24 --- /dev/null +++ b/internal/applier/flux_guide_test.go @@ -0,0 +1,151 @@ +// SPDX-License-Identifier: Apache-2.0 + +package applier + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "io" + "os" + "regexp" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + utilyaml "k8s.io/apimachinery/pkg/util/yaml" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" +) + +// fluxGuide is the Flux user guide. Its "A complete example" section is the configuration this +// file holds to the code: a reader copies it, so it must be accepted as written. +const fluxGuide = "../../docs/flux.md" + +// TestFluxGuide_TheCompleteExampleIsAccepted reads the guide's complete example and checks it +// against the code that validates and drives a Flux applier: the GitOps Reverser objects decode +// into the API types with no unknown field, ApplierConfigured comes out True for the Flux objects +// as written, and a request stamps the GitRepository. It also holds the example to the guide's own +// advice, so the page cannot recommend one thing and show another. +func TestFluxGuide_TheCompleteExampleIsAccepted(t *testing.T) { + objects := guideExampleObjects(t) + + var provider v1alpha3.GitProvider + var target v1alpha3.GitTarget + var secret corev1.Secret + decodeStrict(t, objects, "GitProvider", &provider) + decodeStrict(t, objects, "GitTarget", &target) + decodeStrict(t, objects, "Secret", &secret) + repository := one(t, objects, "GitRepository") + kustomization := one(t, objects, "Kustomization") + assert.NotContains(t, objects, "Receiver", "the guide tells the reader to remove the Flux Receiver") + + // The GitOps Reverser objects are wired to each other. + assert.Equal(t, provider.Name, target.Spec.GitProviderRef.Name) + assert.Equal(t, provider.Namespace, target.Namespace, "a GitTarget names a GitProvider in its own namespace") + require.NotNil(t, provider.Spec.Receiver, "the example serves a push receiver") + assert.Equal(t, secret.Name, provider.Spec.Receiver.SecretRef.Name) + assert.Equal(t, provider.Namespace, secret.Namespace) + assert.NotEmpty(t, secret.StringData[v1alpha3.ReceiverTokenKey], "the receiver reads the token key") + require.NotNil(t, target.Spec.Applier) + assert.Equal(t, v1alpha3.RefusalActionReapply, target.Spec.OnRefusal) + assert.LessOrEqual(t, target.Spec.Applier.EffectiveQuietPeriod(), time.Hour) + + // The Flux objects follow the guide's advice. + for _, obj := range []*unstructured.Unstructured{repository, kustomization} { + interval, _, _ := unstructured.NestedString(obj.Object, "spec", "interval") + d, err := time.ParseDuration(interval) + require.NoError(t, err, "%s spec.interval", obj.GetKind()) + assert.GreaterOrEqual( + t, + d, + time.Hour, + "%s: a short interval fetches, and so applies, on its own", + obj.GetKind(), + ) + } + suspended, _, _ := unstructured.NestedBool(kustomization.Object, "spec", "suspend") + assert.False(t, suspended, "a suspended Kustomization ignores the requests") + + // The applier, as written, is accepted and driven. + ref := types.NamespacedName{ + Namespace: target.Spec.Applier.KustomizationRef.Namespace, Name: target.Spec.Applier.KustomizationRef.Name, + } + if ref.Namespace == "" { + ref.Namespace = target.Namespace + } + want := Binding{RepositoryURL: provider.Spec.URL, Branch: target.Spec.Branch} + + client := newSourceClient(kustomization, repository) + flux := pinnedFlux(client) + flux.CanI = allowAll + verdict := flux.Validate(context.Background(), ref, want) + assert.True(t, verdict.OK, "ApplierConfigured for the guide's example: %s: %s", verdict.Reason, verdict.Message) + + outcome, err := flux.Request(context.Background(), ref, want, testCommit, v1alpha3.ApplyReasonForeignCommit) + require.NoError(t, err) + assert.Equal(t, OutcomeStampedSource, outcome, "a source with no artifact yet is asked to fetch") + assert.NotEmpty(t, requestedAt(t, client, gitRepositoryGVR(), repository.GetNamespace(), repository.GetName())) +} + +// guideExampleObjects returns every object in the yaml blocks of the guide's "A complete example" +// section, by kind. +func guideExampleObjects(t *testing.T) map[string][]*unstructured.Unstructured { + t.Helper() + content, err := os.ReadFile(fluxGuide) + require.NoError(t, err) + text := string(content) + + const heading = "\n## A complete example\n" + start := strings.Index(text, heading) + require.GreaterOrEqual(t, start, 0, "%s must keep its %q section", fluxGuide, strings.TrimSpace(heading)) + section := text[start+len(heading):] + if end := strings.Index(section, "\n## "); end >= 0 { + section = section[:end] + } + + objects := map[string][]*unstructured.Unstructured{} + blocks := regexp.MustCompile("(?s)```yaml\n(.*?)```").FindAllStringSubmatch(section, -1) + require.NotEmpty(t, blocks, "the complete example holds yaml blocks") + for _, block := range blocks { + decoder := utilyaml.NewYAMLOrJSONDecoder(bytes.NewBufferString(block[1]), 4096) + for { + obj := &unstructured.Unstructured{} + if err := decoder.Decode(&obj.Object); err != nil { + if errors.Is(err, io.EOF) { + break + } + require.NoError(t, err) + } + if len(obj.Object) == 0 { + continue + } + objects[obj.GetKind()] = append(objects[obj.GetKind()], obj) + } + } + return objects +} + +// one returns the only object of kind in the example. +func one(t *testing.T, objects map[string][]*unstructured.Unstructured, kind string) *unstructured.Unstructured { + t.Helper() + require.Len(t, objects[kind], 1, "the complete example holds exactly one %s", kind) + return objects[kind][0] +} + +// decodeStrict decodes the only object of kind into into, failing on a field the type does not have. +func decodeStrict(t *testing.T, objects map[string][]*unstructured.Unstructured, kind string, into runtime.Object) { + t.Helper() + encoded, err := json.Marshal(one(t, objects, kind).Object) + require.NoError(t, err) + decoder := json.NewDecoder(bytes.NewReader(encoded)) + decoder.DisallowUnknownFields() + require.NoError(t, decoder.Decode(into), "the guide's %s must decode with no unknown field", kind) +} From 0847f820be43079fe31ccbdd55ca66a1edda4a34 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 21:18:12 +0000 Subject: [PATCH 17/36] refactor(applier): the branch worker owns every apply obligation The second review found every remaining problem in the hand-off between the worker's decisions and the dispatcher's retries: a retry skipped the quiet check, outlived a refusal that had recovered, ran for a recreated GitTarget, or waited on another applier's branch-wide backoff. The dispatcher now carries out one attempt and reports back. It keeps no queue, never retries, and never asks for something weaker than it was told: when no GitTarget still holds the request's reason it asks nothing. The worker keeps what each applier is owed, one attempt out at a time, with the applier's own backoff, and decides every retry on a quiet branch at the tip as it is then, giving back a refusal only while it still stands. Requests carry each GitTarget's uid, and an obligation owed to an incarnation that is gone is dropped. The three relations get their own units: the branch (repository and branch, shared across workers of different GitProviders) owns quietness, the applier owns obligations and retries, the GitTarget owns consent. A shared Kustomization is asked once and waits for the longest quiet period of the GitTargets naming it. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/main.go | 2 - docs/design/apply-trigger-follow-ups.md | 278 ------- docs/design/apply-trigger.md | 206 ++++++ docs/design/bi-directional-right.md | 426 ----------- ...push-notification-and-reconcile-trigger.md | 0 internal/applier/dispatcher.go | 677 +++++------------- internal/applier/dispatcher_test.go | 603 ++++++---------- internal/git/apply_attempt_test.go | 333 +++++++++ internal/git/apply_handback_test.go | 260 ------- internal/git/apply_request.go | 176 +++-- internal/git/apply_siblings.go | 114 +++ internal/git/apply_trigger.go | 593 ++++++++------- internal/git/apply_trigger_test.go | 48 +- internal/git/branch_worker.go | 46 +- internal/git/refusal_reapply.go | 61 +- internal/git/refusal_reapply_test.go | 47 +- internal/git/types.go | 3 - internal/git/worker_manager.go | 8 + internal/telemetry/apply_requests.go | 30 +- 19 files changed, 1593 insertions(+), 2318 deletions(-) delete mode 100644 docs/design/apply-trigger-follow-ups.md create mode 100644 docs/design/apply-trigger.md delete mode 100644 docs/design/bi-directional-right.md rename docs/{design => finished}/push-notification-and-reconcile-trigger.md (100%) create mode 100644 internal/git/apply_attempt_test.go delete mode 100644 internal/git/apply_handback_test.go create mode 100644 internal/git/apply_siblings.go diff --git a/cmd/main.go b/cmd/main.go index a138d00d..d4863e15 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -186,8 +186,6 @@ func main() { applyDispatcher := applier.NewDispatcher( mgr.GetClient(), watchMgr.SourceDynamicClient, mgr.GetEventRecorderFor("gittarget"), ctrl.Log.WithName("applier")) - // A request the dispatcher cannot deliver goes back to its branch worker, which owes it again. - applyDispatcher.HandBack = workerManager fatalIfErr(mgr.Add(applyDispatcher), "unable to add apply request dispatcher to manager") workerManager.SetApplyRequester(applyDispatcher) diff --git a/docs/design/apply-trigger-follow-ups.md b/docs/design/apply-trigger-follow-ups.md deleted file mode 100644 index c5c1c636..00000000 --- a/docs/design/apply-trigger-follow-ups.md +++ /dev/null @@ -1,278 +0,0 @@ -# Apply trigger: follow-ups - -> **design**: open. Written 2026-10-16, after the Flux applier landed on `docs-bi-directional-right` -> (#419). Index: [`../INDEX.md`](../INDEX.md) -> The design this continues: [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md). -> The user-facing model: [`../bi-directional.md`](../bi-directional.md). - -The branch makes GitOps Reverser decide when a Flux `Kustomization` applies a folder. This page -records how the 2026-10-16 review's findings were fixed, then what comes next: Argo CD, the -breaking change, the smaller loose ends, and the parts of the feature the tests do not reach yet. - -## 1. Where the branch stands - -Built and tested: - -- **The decision**, on the branch worker ([`apply_trigger.go`](../../internal/git/apply_trigger.go)). - `applyDebt` turns every remote observation into what the appliers are owed: a `ForeignCommit` - for a tip the worker did not push, a `CatchUp` for its own pushes and for its first observation. - `serviceAppliers` sends requests only on a quiet branch, closes an open window when the branch - moved under it, and waits `quietPeriod` from the last decided write before a catch-up. -- **The refusal action**, `onRefusal: Reapply`, which owes the refusing target's applier a - `Refusal` request ([`refusal_reapply.go`](../../internal/git/refusal_reapply.go)). -- **Delivery**, in [`internal/applier`](../../internal/applier/): the Flux client (stamp the - `GitRepository`, or the `Kustomization` when the artifact already holds the commit), the - `Validate` behind `ApplierConfigured`, and the dispatcher (per-applier superseding, a check of - every target before every attempt, a short retry and then a hand-back to the worker, Events, the - ledger behind `status.applier`, kept per GitTarget uid and forgotten on deletion). -- **The receiver**, `GitProvider.spec.receiver` on `--receiver-bind-address` - ([`receiver.go`](../../internal/webhook/receiver.go)), which turns a verified delivery into a - refresh that always reaches the remote. - -Not built: the Argo CD applier, the webhook applier. The API accepts `kustomizationRef` only. - -## 2. The review's findings, fixed - -The review of `21df16be` (`docs/reviews/20261016-review.md`, kept out of the tree) found six -problems with one cause: the worker decided a request once, and nothing after that decision checked -it again. The dispatcher attempted it for up to four minutes later, against whatever the cluster, -the `GitTarget` and the branch had become. GitOps Reverser now controls when deployments happen, so -each decision has to hold both when it is made and when a patch is sent. All six are fixed on the -branch, each with the reviewer's reproduction kept as a regression test. - -### 2.1 Every attempt checks again (findings 1, 2, 3) - -Before each attempt, retries included, the dispatcher reads each `GitTarget` in the job again and -keeps only those that still name this applier on this branch and that `authz.GitTargetAdmitted` -admits (finding 1: a shared source client is no permission to use it). A `Refusal` needs -`git.ConsentsToReapply` at the attempt; a target that withdrew consent is asked a `CatchUp` instead. -For that, a job keeps each target's own reason, merged with `git.StrongerApplyReason`. A job with no -target left is dropped and counted `outcome="dropped"`. - -Then `Flux.Request` checks the source binding (`Binding{RepositoryURL, Branch}`, read from the -`GitProvider` at the attempt) against the objects it is about to stamp, and returns a permanent -error on a mismatch (finding 2). `Validate` takes the same `Binding`, which is the shape §3.2's -`Applier` interface needs. - -**Decided: no namespace gate on `kustomizationRef`.** The earlier draft held a `kustomizationRef` -in another namespace to the WatchRule rule (`allowAnySourceNamespace`). It is not gated: once the -source binding is checked at the patch, a request reaches only a `Kustomization` that applies this -`GitProvider`'s repository and branch. Whoever can create a `GitTarget` on that `GitProvider` can -already write what that `Kustomization` applies, so an early reconcile adds no power, and Flux's -usual layout (`Kustomization`s in `flux-system`) keeps working without widening WatchRule -delegation. - -### 2.2 A waiting refusal is checked when it stops waiting (finding 3) - -`reapplyOwed` is keyed by `refusalKey`, and `refusalRecovered` cancels the matching entries as it -does `refusalPending`. `serviceRefusals` reads consent again before it sends. - -### 2.3 A request that cannot be delivered goes back to the worker (finding 4) - -The dispatcher retries for about the shortest `quietPeriod` of the job's targets (immediately for a -permanent error), then hands every target back with its reason (`git.ApplyHandBack`, implemented by -the `WorkerManager`). A retry is the worker's decision replayed without the quiet check, so it is -kept short; the worker is what can tell whether the branch is quiet. - -The worker keeps hand-backs in its `applyDebt` (written from the dispatcher's goroutine, then a wake -on the queue), due after a backoff of 15 seconds doubling to 5 minutes. When due and the branch is -quiet: a `ForeignCommit` is sent to those targets; a `Refusal` is owed again only while the refusal -stands (an entry in `coveredRefusal`, which is now recorded even for an empty observation), and so -only with consent; anything else becomes a `CatchUp` and waits for the quiet period. - -`ApplyRequest.HandBacks` carries the count through the worker and back, so the backoff grows and -`ApplyRequestFailed` is recorded only for a request at zero: once per failing request. - -### 2.4 The refusal promise is narrowed (finding 5) - -The five places now say that only this target's applier is asked, that a `GitRepository` that has -to fetch wakes every `Kustomization` reading it, as for any push, and that a folder needing strict -isolation gets its own `GitRepository`. - -### 2.5 Validation is bounded (finding 6) - -`applierVerdict` runs the client lookup and `Validate` under a 10-second deadline. A timeout is -`ReasonCheckFailed`, so the published verdict is kept. The per-reconcile cost (§6) is still open. - -## 3. Next: the Argo CD applier - -### 3.1 What it has to do - -The design is -[§3.6 of the trigger design](push-notification-and-reconcile-trigger.md#36-what-a-request-does). -In short: GitOps Reverser sets `operation.sync` on the `Application` exactly as Argo CD's own -automated sync builds it -([`appcontroller.go`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2628-L2641)), -pinned to the commit the worker checked, and the `Application` runs with -`syncPolicy.automated.enabled: false` (Argo CD 3.1+) so nothing else syncs it. - -Three properties make it different from Flux, and each one has to survive into the code: - -- **No `CatchUp`.** With automated sync off, Argo CD applies nothing on its own, so an - `Application` behind GitOps Reverser's own pushes has nothing stale to apply. And every sync runs - the `Application`'s resource hooks, so syncing for no change would run them for no reason. - `ForeignCommit` and `Refusal` only. -- **The commit is pinned.** The operation names `revision`, so the sync applies what the worker - checked, not whatever the tip has become by the time Argo CD gets to it. Flux cannot do this, - which is why it gets `CatchUp`; Argo CD can, which is why it does not need it. -- **A running operation is waited for, never replaced.** `operation` is a single slot, and - Argo CD's own `SetAppOperation` refuses to overwrite one. A request that finds it occupied retries - later; the dispatcher's backoff already covers this. - -### 3.2 How I would build it - -1. **API.** Add `applicationRef` (`meta.NamespacedObjectReference`) beside `kustomizationRef`, make - both optional, and add a CEL rule on `GitTargetApplier` requiring exactly one. Document that - `quietPeriod` is ignored for `applicationRef`. Regenerate, and extend - [`hack/crdfields/fields.yaml`](../../hack/crdfields/fields.yaml). -2. **An applier interface in `internal/applier`.** Today the dispatcher calls `Flux` directly - (`attempt` builds `NewFlux`; the controller calls `Flux.Validate`). Introduce: - - ```go - // Binding is the repository and branch the applier must apply from. - type Binding struct{ RepositoryURL, Branch string } - - type Applier interface { - Request(ctx context.Context, want Binding, commit string, reason v1alpha3.ApplyReason) (Outcome, error) - Validate(ctx context.Context, want Binding) Verdict - } - ``` - - `Request` takes the binding because it checks it before it writes anything (§2.2). An - `Application` whose source no longer matches is refused at delivery the same way a - `Kustomization` is. Then add a constructor that picks the implementation from the `GitTargetApplier`. Add the kind to - `applierKey`, so a `Kustomization` and an `Application` with one name never supersede each - other. Keep `applier_kind` on the metric (`kustomization`, `application`). -3. **`argocd.go`.** Read the `Application` (`argoproj.io/v1alpha1`, `applications`) through the - dynamic client. - - `Request`: refuse `CatchUp` (outcome `Current`, nothing sent), and otherwise build the - operation from the `Application`'s own spec: `prune` from `syncPolicy.automated.prune`, - `syncOptions` from `syncPolicy.syncOptions`, `retry` from `syncPolicy.retry` or `{limit: 5}`, - `source` from `spec.source` (or `sources` and `revisions` for a multi-source `Application`), - `initiatedBy.username: gitops-reverser`. Write it with an **update**, not a patch, guarded by - the `resourceVersion` the read returned. A conflict or an existing `operation` is a retry. - - `Validate`: `ApplierNotFound`; `SourceMismatch` when no source has a `repoURL` that is the - same repository (`repourl.Same`) with `targetRevision` equal to the branch (`HEAD` and an empty - `targetRevision` mean the remote's default branch, which is a mismatch unless that is the - branch); `AutomatedSyncEnabled` when `syncPolicy.automated` is present without - `enabled: false`; `Forbidden` from an access review for `update` on `applications`. - - `AutomatedSyncEnabled` still sends requests: they are correct, only the guarantee is missing. - It should be a **stall** in `applierReadiness`, because it needs a human, beside - `SourceMismatch` and `Forbidden`. -4. **The worker.** `serviceCatchUp` asks every applier on the branch. It must skip `applicationRef` - targets, which means `appliersOnBranch` carries the kind. That is the only worker change. -5. **RBAC.** `config/rbac/argocd-applier-role.yaml` and a binding (`get` and `update` on - `argoproj.io` `applications`, `create` on `selfsubjectaccessreviews`), included in - `config/rbac/kustomization.yaml` the way the Flux pair is, and Helm `applierAccess.argoCD` - beside `applierAccess.flux`. -6. **e2e, the Argo CD corner.** The corner's specs currently prove Argo CD's own behavior - (`selfHeal: true` flaps, `selfHeal: false` keeps the edit). Keep the bookkeeping spec and the - split-ownership spec, and replace the `selfHeal` phases with: - - the race spec, as in the Flux corner, with `automated.enabled: false` and `applicationRef`; - - a spec that sets `timeout.reconciliation` short (the corner installs Argo CD, so this is - `argocd-cm`) and asserts the `Application`'s operation history holds only GitOps Reverser's - syncs while edits are in flight; - - the Git-side push with an API edit on top, asserting the synced revision is the replayed tip; - - the refusal spec against an `Application`. -7. **Docs.** Bring back the Argo CD section of the guide as the trigger design describes it; the - [docs-first version of the guide](https://github.com/ConfigButler/gitops-reverser/blob/6e700ec8/docs/bi-directional.md#configure-argo-cd) - on this branch's history is a good starting text. Update `configuration.md`'s applier section, - the architecture ground rule, and the trigger design's build order. - -### 3.3 Questions to settle before writing it - -- **`HEAD` as `targetRevision`.** Many `Application`s track `HEAD`. Is that a match for a - `GitTarget` on the remote's default branch? Resolving it needs the remote's HEAD, which the - worker's observation has (`ParentBranch` while the branch is absent, not otherwise). Simplest: - require an explicit branch, and say so in the `SourceMismatch` message. -- **Hooks on `ForeignCommit`.** Every applier on the branch is asked, whether or not the foreign - commit touched its folder (open question 3 in the design). For Flux that is a no-op apply; for - Argo CD it runs hooks. If that is a real complaint, the narrowing is the `Application`'s own - `source.path` compared with the diff, which is the Application's own declaration, not an - inference. -- **Hub Argo CD.** An `Application` usually lives in the Argo CD cluster, not the cluster it - deploys to. The current rule (the applier is looked up in the source cluster) makes the common - Argo CD topology unsupported. It needs a way to say where the `Application` lives: the control - cluster, or a kubeconfig. Decide this first, because it shapes the API. - -## 4. The breaking change - -`onRefusal: PushEmptyCommit` is gone, and `Reapply` needs a Flux applier, so until §3 lands a -folder that Argo CD applies has no automatic revert. GitOps Reverser has no users yet, so there is -nothing to migrate and no reason to hold the merge for Argo CD, which can come in its own PR. Do not -bring `PushEmptyCommit` back as a fallback. - -[`UPGRADING.md`](../UPGRADING.md) gives the order the old CRD allows: `Ignore`, then upgrade, then -set `spec.applier`, then `Reapply`. - -## 5. The webhook applier - -After Argo CD. `webhook: {url, secretRef}`, a POST of `{"repository", "branch", "commit", -"reason"}` signed with `X-Signature: sha256=` (Flux's `generic-hmac` header), behind the same -`Applier` interface. It gets every reason, `CatchUp` included, because nothing says the tool on the -other end does not apply on its own. No in-process plugin, for the reason the trigger design gives. - -## 6. Smaller loose ends - -- **The receiver listens on the elected replica only.** A standby refuses the connection, so a - `Service` across replicas loses deliveries to it. With one replica (the default) this is - invisible. Either bind on every replica and answer `503` until elected, which a Git host retries - or at least reports, or route to the leader. Open question 2 in the trigger design. -- **`ApplierConfigured` reads the source cluster on every reconcile**: two GETs and two access - reviews. A target that is not converged reconciles every ten seconds, so a broken target costs - that much. Cache the verdict per `(cluster, applier, generation)` for a minute, or check only on - the steady tick and on spec changes. The deadline is in place (§2.5). -- **`SourceDynamicClient` serves only declared source clusters.** The dispatcher retries until the - GitTarget's watches have declared the cluster, which covers startup. A target whose declaration - keeps failing therefore never gets its requests delivered, and only the retry log says why. Count - it in `apply_requests_total{outcome="failed"}` with a reason, or surface it in `ApplierConfigured`. - This check answers whether the cluster has a client yet. Whether this target may use that client - is §2.1. -- **The dispatcher's inbox is capped at 1024 and drops the oldest with a log line only.** That - can only fill if the loop is wedged, but a drop there is a lost `ForeignCommit`. Count it. -- **A catch-up when nothing was pushed since the last one.** The first observation after a restart - owes a catch-up, which is right; a refresh that confirms an unchanged tip owes nothing, which is - also right. Check that a worker replaced because its `GitProvider` was repointed starts with a - baseline of the new repository and not a `ForeignCommit` against the old one's tip. I believe it - does (a new worker has a fresh `applyDebt`), and nothing tests it. -- **The trigger design's open question 3**, path scoping for `ForeignCommit`, stays open. - -## 7. What the tests do not reach yet - -Ordered by how much I would worry about each. Three items left this list on 2026-10-16, all proved -by `Manager Flux Applier` (`test/e2e/flux_applier_e2e_test.go`): the receiver end to end (Gitea -delivers a signed push event to the operator's receiver Service, and Flux applies the push with -nothing else on the branch); a foreign push found by the refresher alone, with no receiver and no -API edit, on the steady tick of about five minutes; and a request for a `SourceMismatch` applier, -which never stamps the other branch's source and is delivered once the source is fixed. - -1. **The race spec's margin.** It edits twice a few seconds apart and asserts no extra commit for - 30 seconds, which depends on the corner's commit window and push cooldown. The unit equivalent - now exists in two halves: - `TestServiceAppliers_AWriteDecidedAndPushedInOneWakeRestartsTheQuietPeriod` and - `TestServiceAppliers_ASecondWriteInsideTheQuietPeriodRestartsIt`. Both move `quietSince` by hand - rather than injecting a clock. -2. **A remote source cluster.** Every test of the applier uses the operator's own cluster. A - `ClusterProvider` with a kubeconfig, a `Kustomization` in that cluster, and the - `Forbidden` path when the kubeconfig's identity lacks the grant, are all untested. -3. **`ApplierConfigured` against real RBAC.** `ApplierNotFound` and `SourceMismatch` are covered - end to end by `Manager Flux Applier`. `Forbidden` (drop the flux-applier binding) is unit-tested - with a fake access check only. -4. **Several GitTargets naming one `Kustomization`.** The dispatcher's dedupe is unit-tested; no - e2e has two folders on one branch applied by one `Kustomization`. -5. **A foreign commit while the remote is unreachable.** The rule is that the request waits with - the pending writes (§3.4 of the design). Nothing tests that it does not go out during the - outage, or that it goes out once the push lands. The same test with a `Refusal` owed during the - outage, and the target set to `Ignore` before the outage ends, covers §2.2 end to end. -6. **A foreign commit closing a window a `CommitRequest` is attached to.** - `closeWindowForForeignCommit` finalizes it early; the request should resolve `Committed` with - the replayed commit. Untested. -7. **The CEL rule rejecting `Reapply` without an applier** is not exercised by an envtest. A stored - `PushEmptyCommit` needs no test, because no user has one (§4). -8. **The Helm install.** Local e2e is `INSTALL_MODE=config-dir`, so the chart's receiver flag, - `Service` and `applierAccess.flux` role are covered by `helm template` tests only. CI's - quickstart leg installs the chart, but with neither value set. -9. **The race detector in CI.** `applyDebt` is fed from `publishObservation`, which can run off - the loop goroutine (the first clone), and the dispatcher delivers on goroutines of its own. Both - are lock-guarded, and their tests pass under `-race -count=3` locally (2026-10-16), but CI does - not run `-race`, so a later change can regress this unnoticed. diff --git a/docs/design/apply-trigger.md b/docs/design/apply-trigger.md new file mode 100644 index 00000000..109843e9 --- /dev/null +++ b/docs/design/apply-trigger.md @@ -0,0 +1,206 @@ +# The apply trigger + +> **design**: decided 2026-10-16; the Flux applier, the push receiver and `onRefusal: Reapply` are +> built, Argo CD and the webhook applier are the aim. Index: [`../INDEX.md`](../INDEX.md). +> How to use it: [`../flux.md`](../flux.md) and [`../bi-directional.md`](../bi-directional.md). +> Fields: [`../configuration.md`](../configuration.md#who-applies-this-folder-specapplier). + +**GitOps Reverser decides when Git is applied to the cluster.** Flux and Argo CD keep fetching, +rendering, applying, pruning and reporting health. They stop applying at their own will: their +short timers, webhooks and automated sync are switched off, and GitOps Reverser asks them to apply +at moments it has checked. + +This page is why, what we learned building it, and what is still open. + +## The problem + +Two API edits to one field, a few seconds apart, are enough. GitOps Reverser pushes the first +commit. Flux fetches it, the second edit lands, and Flux applies the first commit over it. GitOps +Reverser faithfully mirrors that revert as Flux's commit, Flux applies the second commit, and the +branch ping-pongs until a fetch happens to land on the tip. There was no conflict to decide: both +edits came from the API side. gitops-api hit exactly this in production (two saves, then four +reconciler-authored commits). + +The cause is not a bug in either tool. Each apply was correct; it ran at the wrong moment. + +## The principle + +**An apply is safe only when nothing live is missing from the commit being applied.** Between an +edit and its push, the edit is live and absent from Git: in an open commit window, or in the +writes waiting to be pushed. Only the branch worker can see those. So only it can say when an apply +is safe, and every trigger the applier runs by itself fires without that knowledge. + +A request is sent only on a **quiet branch**: no open commit window and no pending writes. Three +things owe one: + +| Reason | When | Asked | +| --- | --- | --- | +| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Open windows are closed early and pending writes replayed onto it and pushed first, so the applied commit carries the API side's answer | Every applier on the branch | +| `Refusal` | An edit had no legal destination in Git and the target opted in with `onRefusal: Reapply` | The refusing target's applier | +| `CatchUp` | The branch has been quiet for `quietPeriod` after GitOps Reverser's own pushes, or since it started | Every Flux applier on the branch | + +## What we learned + +Each of these changed the design, and each is the reason for a rule the code now follows. + +**Filtering webhooks is not enough; every timer is a trigger.** The first ask was to forward only +foreign pushes to the applier. But Argo CD's timed refresh syncs every new tip two to three minutes +after each push, with `selfHeal: false` too, and Flux applies on every interval. Neither can be +filtered from outside, so the applier's own triggers go, and GitOps Reverser makes the requests. + +**Flux has no fetch-only mode.** A new `GitRepository` artifact makes every `Kustomization` reading +it apply at once, and `suspend: true` also blocks the requests. So a source that fetches on its own, +on a short interval or a webhook into a Flux `Receiver`, is a `Kustomization` that applies on its +own. The source stays on a long backstop interval, and `CatchUp` keeps its artifact current, because +Flux still applies on a restart from whatever artifact it holds, and a stale one would revert edits +already in Git. + +**Not every commit GitOps Reverser authors is a no-op to apply.** A replayed `F + W` carries +somebody else's `F`, and a refusal needs an apply to be reverted. So the decision is state on the +worker ("this applier is owed an apply"), set by observations and cleared by a delivered request, +not a verdict on each push notification. A notification is a hint: the worker reads the remote and +judges from that, which gets late, duplicate and rolled-back deliveries right. + +**A decision has to hold when the patch is sent, not only when it is made.** The first review found +that a request decided once was attempted for minutes against whatever the cluster had become: a +target its `ClusterProvider` no longer admitted still drove the applier, and a source that followed +another branch was still stamped. Every attempt now reads each target again, and the Flux client +checks that the source fetches this repository and branch before it stamps anything. + +**A retry is a decision too, so only the worker may make one.** The second review found every +remaining problem in the hand-off between the worker's decisions and the deliverer's retries: a +retry skipped the quiet check, outlived a refusal that had recovered, ran for a recreated target, or +waited on another applier's backoff. The fix was structural rather than four patches. The worker +owns every obligation until an attempt delivers it. The deliverer carries out one attempt and +reports back; it never retries, and never asks for something weaker than it was told. A retry is +decided like a first attempt: on a quiet branch, at the tip as it is then, from what still stands. + +**Three relations, three units.** One branch worker serves many `GitTarget`s, many `GitTarget`s can +name one `Kustomization`, and one `GitRepository` can feed many `Kustomization`s. Keeping each +concern on the right unit is what makes the 1:n cases fall out: + +| Unit | Owns | Why | +| --- | --- | --- | +| The **branch** (repository and branch) | Quietness | A `Kustomization` applies the tip whichever folder an edit is in. Two `GitProvider`s naming one repository give one branch two workers, so a worker is quiet only when its siblings are too | +| The **applier** | Obligations and retries | One attempt out at a time, its own backoff, one attempt for every `GitTarget` naming it. A shared applier waits for the longest `quietPeriod` among them, because its apply reaches all their folders | +| The **`GitTarget`**, by uid | Configuration and consent | Consent to a `Refusal` is per target, and a target deleted and recreated under the same name inherits nothing owed to the old one | + +**A shared `GitRepository` cannot be isolated, so the promise was narrowed.** When a request has to +make the source fetch, every `Kustomization` on that source applies the new revision. That is how +Flux works. The branch is quiet, so they apply a tip with nothing live missing; the guide says a +folder needing strict isolation gets its own `GitRepository`. + +## The design, as built + +```mermaid +flowchart LR + subgraph WORKER["Branch worker (one per GitProvider and branch)"] + OBS["Observations
push · fetch · notification"] + OWE["Owed, per applier
ForeignCommit · Refusal · CatchUp"] + QUIET{"Quiet branch?
this worker and its siblings"} + end + EXEC["Dispatcher
one attempt, no retries"] + FLUX["Flux
GitRepository · Kustomization"] + + OBS --> OWE --> QUIET + QUIET -- "yes, and due" --> EXEC + EXEC -- "recheck targets, check source,
stamp requestedAt" --> FLUX + EXEC -. "report: Delivered · Current ·
Failed · Rejected · Dropped" .-> OWE +``` + +- **Observing.** Every observation of the remote passes one point. The first after a start is a + baseline (owes `CatchUp`, never `ForeignCommit`, because an apply on startup would revert edits + made while the operator was down). A tip the worker pushed owes `CatchUp`; any other owes + `ForeignCommit`. When in doubt it is foreign, which costs one apply on a quiet branch. +- **Deciding.** At the end of every wake, if the branch is quiet, each applier that is owed something, + has no attempt out, and is past its backoff (and, for a `CatchUp`, its quiet period) gets one + attempt. The attempt carries away what it covers; each target in it carries its uid and its own + reason. +- **Attempting.** The dispatcher re-reads each target. It asks only for those that are still the + same incarnation, still name this applier on this branch, are still admitted by their + `ClusterProvider`, and for a `Refusal` still consent. If no target still holds the request's + reason, it asks nothing (`Dropped`). Then Flux: stamp `reconcile.fluxcd.io/requestedAt` on the + `GitRepository` when its artifact lacks the commit, else on the `Kustomization`; a `CatchUp` whose + artifact holds the commit stamps nothing. A source on another repository or branch is `Rejected`. +- **Settling.** A delivered attempt is done. Any other outcome gives back what the attempt carried, + a refusal only if it still stands, and starts that applier's backoff (2s doubling to 5m, at least + 15s after a rejection). `ApplyRequestFailed` is recorded for the first failure of a run. +- **Learning about pushes.** A push webhook into GitOps Reverser's receiver (seconds), a rejected + compare-and-swap push (at the next publication), or the periodic refresh (minutes). All three + feed the same observation point. + +## The apply window: what is still racy + +**A request is asynchronous, and that leaves a window.** The patch returns at once; Flux fetches and +applies a few seconds later. An API edit that lands in those seconds, to a field the +`Kustomization` applies, is reverted by the apply, and GitOps Reverser mirrors the revert as Flux's +commit. The edit survives in Git history only if its own commit was pushed first. + +The quiet-branch rule confines that window to three moments: right after somebody else's push, right +after a refusal the target asked to have reverted, and right after `quietPeriod` of silence on a +branch whose artifact is behind. It does not close it, and making the request synchronous would not +either: waiting for the apply to finish does not stop a user from editing during it. Only refusing +edits would, which halts the API for everyone to protect a few seconds, so we do not. + +What would make the window smaller and visible, in order of cost: + +1. **Know when the window closes.** Flux copies the requested stamp into the `Kustomization`'s + `status.lastHandledReconcileAt` once it has handled it. The dispatcher can report `Delivered` + only then, so an attempt stays "out" for exactly the apply's duration and nothing is stacked on + an apply still running. +2. **Name the collision.** An edit observed while an attempt is out, on an object the applier owns, + is the case that can be reverted. Recording it (an Event on the `GitTarget`, a counter) turns a + silent revert into a visible one. +3. **Let the API win the window.** A write by the applier's own field manager during an attempt we + made, that undoes a field edited after that attempt was sent, is our apply arriving late, not new + intent. Not mirroring it keeps the edit in Git, and the next request applies it back. This is the + rule "for an object both sides changed, the API value wins", applied to the apply window. It needs + the field manager from `managedFields` (no audit), and it is the only option that changes what + reaches Git, so it needs its own decision. + +## The aim beyond Flux + +**Argo CD (`applicationRef`).** The `Application` runs with `syncPolicy.automated.enabled: false` +(Argo CD 3.1 and later), so its refresh only compares and nothing syncs it but us. A request sets +`operation.sync` exactly as Argo CD's automated sync builds it, with `prune`, `syncOptions` and +`retry` from the `Application`, and **`revision` pinned to the commit the worker checked**. Three +consequences: + +- **No `CatchUp`.** With automated sync off Argo CD applies nothing on its own, so there is no stale + artifact to catch up, and every sync runs the `Application`'s hooks. +- **The commit is pinned**, so a sync applies what the worker checked, not whatever the tip became. +- **A running operation is waited for, never replaced**, as Argo CD's own `SetAppOperation` does. + +The open question that shapes its API: a hub Argo CD keeps `Application`s in a management cluster, +not the cluster they deploy to, and today an applier is looked up in the source cluster. + +**A webhook applier** for any other tool: POST `{"repository", "branch", "commit", "reason"}`, +signed `X-Signature: sha256=` the way Flux's `generic-hmac` `Receiver` checks. It gets +every reason, `CatchUp` included, since nothing says the tool does not apply on its own. No +in-process plugin: it would run with every credential the operator holds. + +## What it costs + +- **GitOps Reverser becomes the fast path to a deployment.** While it is down, a Git-side push + reaches the cluster through `flux reconcile`, a manual sync, or the long backstop interval. + Bi-directional editing already stops in that state. +- **The receiver runs on the elected replica only.** A standby refuses deliveries; with more than + one replica they need routing to the leader. +- **Every applier on the branch is asked on a `ForeignCommit`**, whether or not the push touched its + folder. Harmless for Flux; for Argo CD it runs hooks. Narrowing it needs the applier's own source + paths, so it waits for a measured complaint. + +## What proves it + +| Claim | Test | +| --- | --- | +| The decision: quiet branch, reasons, quiet period, baseline | [`apply_trigger_test.go`](../../internal/git/apply_trigger_test.go) | +| Retries are decided by the worker; recovered refusals, withdrawn consent and recreated targets are not retried; each applier keeps its own clock; one attempt for a shared applier; a busy sibling worker holds the branch | [`apply_attempt_test.go`](../../internal/git/apply_attempt_test.go) | +| Every attempt rechecks targets, never asks for something weaker, and never stamps a source on another branch | [`dispatcher_test.go`](../../internal/applier/dispatcher_test.go), [`flux_test.go`](../../internal/applier/flux_test.go) | +| The original race: two API edits end at the later value with no extra commit | [bi-directional Flux corner](../../test/e2e/flux_bi_directional_e2e_test.go) | +| `SourceMismatch` is never stamped and is delivered once fixed; a Gitea webhook and the refresh alone each get a push applied | [Manager Flux Applier](../../test/e2e/flux_applier_e2e_test.go) | +| A refused edit is reverted by its `Kustomization` | [Refusal reapply](../../test/e2e/refusal_reapply_e2e_test.go) | +| The Flux guide's example is accepted by the code | [`flux_guide_test.go`](../../internal/applier/flux_guide_test.go) | + +The history of how this was decided, with the options weighed and the Git-side reference the code +cites, is in [`../finished/push-notification-and-reconcile-trigger.md`](../finished/push-notification-and-reconcile-trigger.md). diff --git a/docs/design/bi-directional-right.md b/docs/design/bi-directional-right.md deleted file mode 100644 index 83634d45..00000000 --- a/docs/design/bi-directional-right.md +++ /dev/null @@ -1,426 +0,0 @@ -# Feedback: the reconciler should only be woken by commits the reverser did not make - -_From the [gitops-api](https://github.com/ConfigButler/gitops-api) team to gitops-reverser, -2026-10-15. Wishlist ask **#29** ([upstream-wishlist.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/upstream-wishlist.md)). Written to be -readable from the gitops-reverser repo; no gitops-api knowledge assumed. **We run `v0.52.0`.** Every -gitops-reverser link is pinned to -[`400edf68`](https://github.com/ConfigButler/gitops-reverser/tree/400edf68242086dde5f650ac9835072f2bdfb5e1) -(`main` today)._ - -> **Response (2026-10-16): GitOps Reverser takes over the reconciliation triggers.** Flux and -> Argo CD keep doing what they are good at, but they apply when GitOps Reverser asks, not on their -> own schedule. The ask is right, and filtering webhooks turns out not to be enough: see -> [§9](#9-response-gitops-reverser-takes-over-the-triggers). - -**The ask, in one sentence.** Make gitops-reverser the component that tells the GitOps reconciler to -apply, and have it do so **only when the branch gained a commit the reverser did not author**, -never for its own pushes. - -**Why.** In the bi-directional setup your own guide recommends, every commit the reverser pushes -wakes Flux, and Flux then applies that commit to a cluster that may already have moved past it. -Two ordinary API edits a few seconds apart are enough. The older commit is applied over the newer -edit, the reverser faithfully mirrors that revert into Git as the reconciler's commit, and the -branch ping-pongs until a fetch happens to land on the tip. Which value survives is luck. There is -no conflict being decided here: **both edits came from the API side.** - -## 1. What we saw - -Our product is an editor over a kcp workspace. A user's save is a Server-Side Apply into the -workspace, gitops-reverser (one instance, `v0.52.0`) mirrors it to the tenant's repository, and a -Flux `GitRepository` + `Kustomization` (Flux 2.9.6) applies the repository back. Our Git-host webhook -handler wakes the **`GitRepository` only**, never the `Kustomization`, as -[`bi-directional.md` → Configure Flux](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#configure-flux) -recommends (we learned that one the hard way too). - -Our browser check fires two saves at once. We serialise them ourselves, so the second save writes -only after the first save's `CommitRequest` is `Ready=True/Committed`. One run, 2026-10-15, tenant -repository history: - -| Time (UTC) | Commit | Author | Change to `data` | -|---|---|---|---| -| 15:53:00 | `3b08d12a` | Simon (save A) | `+ a` | -| 15:53:05 | `08b9bf84` | Simon (save B) | `- a`, `+ b` | -| 15:53:11 | `ad719e84` | `flux-tenant-configbutler` | `+ a` | -| 15:53:19 | `f22d63e1` | `flux-tenant-configbutler` | `- a` | -| 15:53:26 | `f4fbc60b` | `flux-tenant-configbutler` | `+ a` | -| 15:53:32 | `d88fc968` | `flux-tenant-configbutler` | `- a` | - -Save B's `- a` is our harness, not the race: both saves applied under one field manager, so B's -apply released A's key. Everything after it is the race. - -The same pattern appeared in the 2026-10-13 run. To check the mechanism, we patched our -serialisation back to "release on `Committed`" and reran with both saves writing **the same -field**. The result was two reconciler-authored commits after the saves, `a03446a1` and `49b4baea`. -The `Kustomization` never settled on the later save's commit. - -```mermaid -sequenceDiagram - autonumber - actor U as User (two saves) - participant API as Kubernetes API - participant Rev as GitOps Reverser - participant Git as Git branch - participant Flux as Flux (GitRepository + Kustomization) - - U->>API: save A (x = 1) - API-->>Rev: watch event - Rev->>Git: push A - Git-->>Flux: push webhook → fetch A - U->>API: save B (x = 2) - Note over API: live is now AHEAD of A - API-->>Rev: watch event - Flux->>API: apply A (x = 1) - Note over API: B is reverted in the cluster - Rev->>Git: push B (x = 2) - API-->>Rev: watch event: x = 1, by Flux - Rev->>Git: push "x = 1" authored by Flux - Git-->>Flux: webhook → fetch B → apply (x = 2) - API-->>Rev: watch event: x = 2, by Flux - Rev->>Git: push "x = 2" authored by Flux - Note over Git,Flux: lags one commit behind, ping-pongs
until a fetch lands on the tip -``` - -Steps 5 and 7 are the whole bug: the cluster moves, then Flux applies the commit before it. -Everything after them is the reverser doing exactly what it should, which is why nothing errors: -mirroring stays faithful and every commit is attributed. - -## 2. Why it happens, in your own terms - -[`bi-directional.md` → Why the loop stops](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#why-the-loop-stops) -gives the reason the loop settles: _"When the reconciler applies a commit Reverser wrote, the live -object already holds those values, because they were read from that object in the first place."_ -That carries a precondition the page states once (_"With no intervening edits"_) but nothing -enforces: **the live object has not moved since.** The page is also already explicit that nothing -reserves the time: - -- [The guide's short version](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md): - _"There is no merge, no lock, and no timing guarantee."_ -- [How a refused edit can undo an allowed one](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#how-a-refused-edit-can-undo-an-allowed-one): - _"The same race exists whenever a new Git revision arrives … Shortening the commit window reduces - exposure but does not reserve time against an apply."_ -- [`facts/gitops-apply-and-field-ignore.md`](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/facts/gitops-apply-and-field-ignore.md): - _"a reverse merge alone cannot stop the forward applier from enforcing an older Git value before - publication completes."_ - -We agree with all of that. Our point is narrower. **For a commit the reverser authored, the apply -can never do anything useful.** That commit _is_ the cluster's state, read from it. Applying it back -has two possible outcomes: a no-op, if the cluster has not moved, or a revert, if it has. No case -exists where waking the reconciler for the reverser's own push brings in new intent. The whole -exposure in §1 comes from triggers that cannot pay off. - -[§1.2 of the push-notification design](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#12-which-side-wins-before-anything-else) -already draws this line: _"A push we did not author is new truth … getting it applied quickly is -the point."_ The converse is the ask: **a push we did author is not news to the cluster, and -nothing should be told to apply it.** - -## 3. The ask: the reverser decides when the reconciler applies - -```mermaid -flowchart LR - api(["API edit"]) --> cluster[("Cluster")] - cluster -- "watch event" --> rev["GitOps Reverser"] - rev -- "commit + CAS push" --> git[("Git branch")] - human(["Someone pushes to Git"]) --> git - git -- "push webhook (one, to the reverser)" --> rev - rev -- "after = a tip WE pushed?" --> drop["ignore:
the cluster already has it"] - rev == "branch gained a commit we did NOT author" ==> rec["Flux Receiver / Argo CD webhook"] - rec -- "apply" --> cluster - - style rev fill:#e8f4fd,stroke:#2196f3 - style rec fill:#fff3e0,stroke:#ff9800 - style drop fill:#eeeeee,stroke:#9e9e9e -``` - -This is your [Option 4](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#option-4-take-over-the-webhook-and-forward-it) -(the Git host talks only to the reverser) combined with -[Option 3](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#option-3-receive-a-push-notification)'s -dedupe table. One change: **the dedupe decides whether to forward, not only whether to fetch.** - -| `after` on the delivery | Meaning | Option 3 today | **Asked for** | -|---|---|---|---| -| a tip the reverser pushed | our own push coming back | ignore | **ignore, and do not forward** | -| anything else | somebody else moved the branch | invalidate `baseTrusted` | invalidate, **then forward** (§4 says when, §5 how) | - -As written, Option 4 forwards every delivery, so it would keep the race exactly as it is today. -[Option 2](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#option-2-tell-the-reconciler-after-we-push) -("tell the reconciler after we push") is the race made into a feature, and we would ask you **not** -to build it in that form. Its own selling point ("the loop closes without the user configuring a Git -host webhook") is a loop that has nothing left to close. - -The bookkeeping this needs is the bookkeeping Option 3 already names: `lastCommitSHA` must advance -on a successful push, or every own push looks foreign. Option 3's argument against a `before` field -still holds. A set of our own SHAs cannot tell our own delayed delivery from a rollback through our -commits. Misclassifying our own push as foreign costs one useless apply, which is today's -behaviour. Misclassifying the other way would lose a human's push. So **when in doubt, forward**. - -The reverser also has two other ways to learn the branch moved, both shipped. Each should feed the -same rule: - -- a **rejected compare-and-swap push**, whose advertisement names a hash none of its publications - produced; -- **`--git-refresh-interval`**, which reports `status.remote.verifiedBy: Fetch` for exactly that - case - ([Telling Reverser that somebody else pushed](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#telling-reverser-that-somebody-else-pushed)). - -So even an installation with **no** inbound webhook could trigger the reconciler correctly: late, -bounded by the refresh interval, but never for its own commits. - -## 4. Ordering: forward after the replay, not before - -A foreign commit `F` can land while the reverser still holds API edits it has not pushed yet: an -open commit window, or writes retained for replay. Call those edits `W`. They are live in the -cluster, absent from Git, and so absent from `F`. - -If the reconciler is woken at `F`, it applies `F` and reverts `W` in the cluster before `W` ever -reaches Git. Replay still puts `W` on top of `F` in Git, but the reverser then mirrors the revert -over it, and the API side loses. Your §1.2 table says the API wins this case (the top-left cell); -the early apply makes it lose. - -```mermaid -sequenceDiagram - participant Human as Git-side author - participant Git - participant Rev as GitOps Reverser - participant Rec as Flux / Argo CD - participant API as Cluster - - Note over Rev,API: open window W: an API edit Git does not have yet - Human->>Git: push F - Git-->>Rev: webhook (after = F, foreign) - Rev->>Rev: invalidate, fetch F, replay W onto F - Rev->>Git: CAS push F + W' - Rev-->>Rec: reconcile, at the tip it just pushed - Rec->>API: apply F + W' - Note over API: W' equals live for W's objects: a no-op there.
Only F's changes land. -``` - -**Asked for:** when a foreign commit arrives with retained or open writes, request the reconcile -after the replayed push lands, so the applied revision already carries the API side's answer. With -no pending writes, forward at once. This narrows the remaining window, and does not claim to close -it. An edit that lands between that push and the apply is still exposed, as your guide says. But -that exposure now exists **only when somebody else pushed to Git**, which is a genuine two-sided -event. Today it follows every commit the reverser makes. - -Option 5's caveats ("it cannot carry correctness") apply in full, and we are not asking for a hold -on the watch or on the push. This is only _when to send one HTTP request_. - -## 5. How: the request that wakes the reconciler - -The reverser learns of a foreign commit in one of two ways: a host delivery reaches it, or it -detects the move itself. Each case needs a different call. - -| Reconciler | With a host delivery (5.1) | Without one (5.2) | -|---|---|---| -| Flux | forward it to a `Receiver` of the host's type (`github`, `gitlab`, …) | POST to a `generic-hmac` `Receiver` | -| Argo CD | forward it to `/api/webhook` | a synthesized GitHub-shaped push to `/api/webhook` | - -In both cases **the reverser names no `GitRepository` and no `Application`.** The downstream already -holds that mapping: the `Receiver` in `spec.resources`, and Argo CD in every `Application` whose -source matches the repository and branch. This is your Option 4's load-bearing reason, and it holds -here too. - -### 5.1 With a host delivery: forward the bytes, later if need be - -When the Git host's webhook points at the reverser, forward the delivery exactly as Option 4 -describes: the same body and headers, with nothing re-signed or re-serialised. #29 changes only -_whether_ the delivery is forwarded (an own push is dropped) and _when_ (a foreign push waits for -the replay in §4). Holding a delivery back is safe for two reasons: - -- **The signature has no expiry.** GitHub's `X-Hub-Signature-256` is an HMAC over the body alone, - and GitLab's `X-Gitlab-Token` is a static token. A delivery forwarded a few seconds late still - verifies. -- **Neither downstream applies the delivery's `after`.** Each treats the delivery as "look at this - branch now" and resolves the tip itself. A delivery that names `F` but is forwarded after - `F + W'` lands therefore applies `F + W'`, which is the state §4 asks for. - -### 5.2 Without one: the reverser's own request - -Three of the triggers in §3 bring no delivery to forward: a rejected compare-and-swap push, a -`--git-refresh-interval` fetch, and any installation whose host webhook cannot point at the -reverser. **Ours is in the last group.** A GitHub App has one webhook URL for all of its -installations, and ours belongs to gitops-api. We would relay `{repository, branch, after}` to your -[Option 3 endpoint](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/design/push-notification-and-reconcile-trigger.md#41-the-wire-contract-for-option-3), -so the reverser would never hold a GitHub-signed delivery. - -In these cases the reverser makes its own request. That is your Option 2's outbound leg, with the -dedupe from §3 in front of it: - -- **Flux: a `generic-hmac` `Receiver`.** Send a POST to the path the Receiver publishes in - `status.webhookPath` (served by the `webhook-receiver` Service), with - `X-Signature: sha256=`. This is the call - `flux trigger receiver` makes. The body is free-form, so §4.1's three fields will do. The - Receiver's `spec.resources` should list the `GitRepository` only. The source is enough, and a - poked `Kustomization` applies the artifact it already holds, which is the commit _before_ the one - that triggered it ([field-ownership.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/field-ownership.md)). -- **Argo CD: a synthesized push event.** Argo CD has no generic receiver type, so the reverser POSTs - a minimal GitHub-shaped push to `/api/webhook`. It carries `X-GitHub-Event: push`, `ref`, `after` - and `repository.html_url`, and is signed with `webhook.github.secret` from `argocd-secret` when one - is set. Leave the `commits` file lists empty. As your Option 4 notes, a payload that lists no - changed files makes `AppFilesHaveChanged` return true, so Argo CD refreshes every matching - `Application` instead of guessing paths. This is the inverse of Option 4's parser, and it can come - after Flux. - -**Annotating the object directly** (`reconcile.fluxcd.io/requestedAt` on the `GitRepository`, -`argocd.argoproj.io/refresh` on the `Application`) also works. It is what our `triggerFluxForPush` -does today. We would still not recommend it for the reverser, because the reverser would then have -to know which object syncs the folder and hold patch rights on it. That is the ownership interpreter -your Option 4 avoids. A `Receiver` keeps the mapping where the user wrote it. - -Option 2's four rules apply unchanged. The POST requests a reconcile and never reports one as -applied. It runs off the publication path, retries on its own, and never fails a publication that -succeeded. - -## 6. What we would stop doing - -```mermaid -flowchart TB - subgraph TODAY["Today (gitops-api)"] - direction LR - gh1[("GitHub")] -- "every push" --> api1["gitops-api webhook"] - api1 -- "poke GitRepository" --> f1["Flux"] - rev1["Reverser"] -- push --> gh1 - end - subgraph ASKED["With #29"] - direction LR - gh2[("GitHub")] -- "every push" --> rev2["Reverser"] - rev2 -- "foreign pushes only" --> f2["Flux Receiver"] - rev2 -- push --> gh2 - end -``` - -- **Our webhook stops waking Flux.** We wake the `GitRepository` on every push today - (`triggerFluxForPush`), the reverser's own included. That is the trigger in §1. With #29 the - webhook relays the push to the reverser instead (§5.2), and the reverser becomes the only - component that decides a revision should be applied. -- **Flux keeps its own timers as the backstop**, and they stay long (`24h` on both objects). Your - Option 4's warning, _"'long intervals' must never be read as 'no intervals'"_, holds. One - addition for your Flux guidance: a **short** `GitRepository` interval (the `1m` that - [Configure Flux](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#configure-flux) - recommends) picks up the reverser's own commits too, because a new artifact revision wakes the - `Kustomization`. That brings §1 back on a timer. Under #29 the short poll would undo what the - filter buys, so the recommendation should say so. -- **Our stopgap goes.** Until then we serialise our own saves behind Flux's echo. A save holds a - per-repository gate until the `Kustomization` reports `lastAppliedRevision` = that save's commit, - so Flux's apply of it is a no-op before the next save may write - ([forward-leg-loop.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/forward-leg-loop.md#the-third-lap-a-save-that-overtakes-the-echo-2026-10-15)). - This is your deferred - [Option 4, "applier watermark"](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/future/git-api-three-way-comparison.md#option-4-use-the-appliers-applied-revision-watermark), - done crudely, outside the engine, and only for writes that pass through our server. A `kubectl - apply` straight into the workspace is not covered. It costs a queued save 10–15 s (30 s when - another push lands in between), and it is the kind of guard in one component that we would rather the engine made unnecessary. - -## 7. A test you could add - -[What the tests prove](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/bi-directional.md#what-the-tests-prove) -says _"no spec tests simultaneous writes to one field from both sides, because the outcome depends -on arrival order."_ This case is different: **both writes come from the API side**, so the expected -outcome does not depend on a policy. It is simply the later write. Under the recommended Flux -configuration (Receiver on the `GitRepository`, webhook on every push): - -1. Patch one field of one object to `1`. Wait for its commit to push. -2. Within ~2 s of that push, patch the same field to `2`. -3. Wait for both commits, plus two reconciler intervals. -4. Assert that the live value is `2`, that Git's file holds `2`, and that **no commit after step 2 is - authored by the reconciler's identity**. - -On our cluster, step 4's last clause fails on today's configuration and passes with our stopgap. It -should pass with #29 and no stopgap. We would rather it lived in your bi-directional corner than in -our browser harness, because it is a property of the engine plus the reconciler, not of our editor. - -## 8. What we are not asking for - -- **Not a merge, and not a conflict policy.** The - [three-way comparison record](https://github.com/ConfigButler/gitops-reverser/blob/400edf68242086dde5f650ac9835072f2bdfb5e1/docs/future/git-api-three-way-comparison.md) - stays deferred as far as we are concerned. #29 is only about not causing a conflict where none - exists. -- **Not a filter on the audit stream or on events.** The reverser must keep mirroring every write, - the reconciler's included. A reconciler-authored commit is exactly how we saw this, and it should - stay visible - ([forward-leg-loop.md](https://github.com/ConfigButler/gitops-api/blob/main/docs/design/forward-leg-loop.md) - is why we feel strongly about that). -- **Not a guarantee of being first.** As your Option 4 says, a timer, a retry, or `flux reconcile` - can always reach Git without the reverser. #29 removes the trigger that fires on every own - commit. It does not claim to remove every trigger. - -## 9. Response: GitOps Reverser takes over the triggers - -_From gitops-reverser, 2026-10-16. The design is -[push-notification-and-reconcile-trigger.md](push-notification-and-reconcile-trigger.md); the -guide as it will read is [bi-directional.md](../bi-directional.md)._ - -**The short version: GitOps Reverser takes over the reconciliation triggers. It has to.** Flux and -Argo CD keep doing what they are good at: fetching, rendering, applying, pruning, and reporting -health. They stop doing it at their own will. An apply happens because GitOps Reverser asked for -it on a branch with nothing pending, because a human asked, or because of a long backstop timer. - -We accept the rule and the ordering in §4, and we go further than §8's last bullet: the webhook is -not the only trigger to take away, because every timer the applier runs is a trigger too. - -### 9.1 Why it has to - -An apply is safe only when nothing live is missing from the commit being applied. GitOps Reverser -is the only component that can know that, because it holds the open commit windows and the writes -waiting to be pushed. An apply it did not ask for is one nobody checked. - -### 9.2 Timers are triggers too - -§6 notes this for Flux's short `GitRepository` poll. On Argo CD it is worse. The timed refresh -(`timeout.reconciliation`, 120s plus up to 60s of jitter) resolves the branch tip, and with -`selfHeal: false` automated sync skips only a commit it already synced -([`alreadyAttemptedSync`](https://github.com/argoproj/argo-cd/blob/3cbae653d41ced2fcdec5d07c25fff2693c4242e/controller/appcontroller.go#L2648-L2670)). -Every push GitOps Reverser makes is a new commit, so two to three minutes after each push the whole -`Application` is synced over whatever is pending. Filtering webhooks does nothing about it. - -### 9.3 What we take from the ask, and what we change - -We take the rule (apply when the branch gained commits we did not author), the ordering in §4, and -the test in §7. What changes: - -1. **The applier is named, on the `GitTarget`.** `spec.applier` holds a `kustomizationRef`, an - `applicationRef` or a `webhook`. The objection to naming orchestrator objects was to inferring - which one applies a folder; a declared reference infers nothing. That also replaces §5's - `Receiver` and synthesized-event plumbing: GitOps Reverser patches the applier through the - Kubernetes API, as `flux reconcile --with-source` does, or sets the `Application`'s sync - operation pinned to the checked commit. -2. **Argo CD runs with automated sync off.** `automated.enabled: false` (Argo CD 3.1 and later) - keeps `prune` configured; GitOps Reverser's sync copies it. A refresh then only compares, so - Argo CD's timer can stay. -3. **Flux needs one request for our own commits after all.** Flux has no refresh-only mode: its - timers and a controller restart apply whatever artifact it holds, and with long intervals that - artifact falls behind our pushes. Applying it would revert published edits. So once the branch - has been quiet for `quietPeriod` (`30s`), GitOps Reverser asks Flux to fetch, reason `CatchUp`. - The wait keeps it out of an editing session like the one in §1. -4. **Not every commit we author is a no-op**, and the design does not rely on that. Our replayed - `F + W'` carries `F`; a refusal needs an apply too. So the decision is state on the branch - worker ("the applier is behind"), set by any foreign commit or eligible refusal and cleared by a - request on a quiet branch, not a verdict on each delivery. A delivery is a hint: the worker reads - the remote's advertisement and judges from that, which handles late deliveries, rollbacks onto - our own older commits, and deliveries for our own replayed push. -5. **The hold does not wait for commit windows to close.** A busy user would hold a foreign push - back indefinitely. On a foreign commit the worker closes the windows, replays, pushes, then - requests. -6. **`onRefusal: PushEmptyCommit` becomes `onRefusal: Reapply`.** With the applier named, the - refusal asks that one applier to apply the tip, once the branch is quiet, instead of moving the - branch for everything watching it. -7. **Suppression is per applier, by construction.** A reconciler for another cluster is not - declared on this `GitTarget`, so we never ask it anything and its own triggers are its business. - -### 9.4 What changes for gitops-api - -- Your webhook stops waking Flux. Relay `{repository, branch, after}` to the tenant `GitProvider`'s - receiver (`type: generic-hmac`, path in `status.webhookPath`). -- Declare the tenant `Kustomization` as each `GitTarget`'s `kustomizationRef`, and drop any Flux - `Receiver` for the repository. -- The `lastAppliedRevision` gate goes. Flux still applies your saves eventually, through `CatchUp`, - but only after the branch has been quiet, which is what the gate was approximating. -- `GitRepository` and `Kustomization` intervals go long (`24h`); the `1m` poll goes. - -### 9.5 What we accept, and what is open - -- **GitOps Reverser becomes the fast path to a deployment.** While it is down, a Git-side push - reaches the cluster only through a manual sync, `flux reconcile`, or Flux's backstop interval. -- **An apply request cannot reserve time.** An edit made while a requested apply runs is still - reverted. That now happens only after a foreign push, a refusal, or a quiet period, not after - every commit we make. -- **Open:** an applier in a cluster other than the one it applies to (hub Argo CD), and routing - deliveries with more than one replica. Both are in the design's open questions. diff --git a/docs/design/push-notification-and-reconcile-trigger.md b/docs/finished/push-notification-and-reconcile-trigger.md similarity index 100% rename from docs/design/push-notification-and-reconcile-trigger.md rename to docs/finished/push-notification-and-reconcile-trigger.md diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index 3775cd75..374e1787 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -4,9 +4,8 @@ package applier import ( "context" + "errors" "fmt" - "slices" - "sort" "sync" "time" @@ -33,9 +32,8 @@ import ( const ( // EventReasonApplyRequested is a request that reached the applier. EventReasonApplyRequested = "ApplyRequested" - // EventReasonApplyRequestFailed is a request that could not be delivered: handed back to the - // branch worker after its retries, which asks again once the branch is quiet. Recorded once per - // failing request, not on every retry or on every time the worker asks again. + // EventReasonApplyRequestFailed is the first failed attempt of a run of them. The branch worker + // keeps the obligation and asks again on a quiet branch; the Event is not repeated for its retries. EventReasonApplyRequestFailed = "ApplyRequestFailed" ) @@ -43,110 +41,58 @@ const ( const ( applierKindKustomization = "kustomization" - outcomeSent = "sent" - outcomeFailed = "failed" - outcomeSuperseded = "superseded" - outcomeDropped = "dropped" + outcomeSent = "sent" + outcomeFailed = "failed" + outcomeDropped = "dropped" ) -// maxInbox bounds the requests waiting to be resolved. The loop drains the inbox the moment it is -// woken and never waits on an applier while doing so (delivery runs on its own goroutines), so the -// inbox only fills when the loop is wedged, which is a bug; the bound keeps that bug from also -// being a leak. -const maxInbox = 1024 - // shortCommitLength is how much of a commit an Event names. const shortCommitLength = 8 +// errNotRunning fails an attempt made before the dispatcher started or after it stopped. +var errNotRunning = errors.New("the apply request dispatcher is not running") + // SourceClients resolves a source cluster's dynamic client by ClusterProvider name // (GitTarget.SourceCluster()). In production it is (*watch.Manager).SourceDynamicClient. type SourceClients func(ctx context.Context, clusterID string) (dynamic.Interface, error) -// Backoff is the retry schedule for one apply request: Initial after the first failure, doubling, -// capped at Max. How long a request is retried is not the schedule's to say (see Dispatcher). -type Backoff struct { - Initial time.Duration - Max time.Duration -} - -// DefaultBackoff retries quickly: the retries are for a blip of a few seconds. -func DefaultBackoff() Backoff { - return Backoff{Initial: defaultBackoffInitial, Max: defaultBackoffMax} -} - -const ( - defaultBackoffInitial = 2 * time.Second - defaultBackoffMax = 15 * time.Second -) - -// delay is the wait after the given number of consecutive failures (at least one). -func (b Backoff) delay(failures int) time.Duration { - d := b.Initial - for i := 1; i < failures && d < b.Max; i++ { - d *= 2 - } - return min(d, b.Max) -} - -// DefaultAttemptTimeout bounds one attempt: a source cluster that does not answer fails the -// attempt, and the backoff decides when to try again. +// DefaultAttemptTimeout bounds one attempt: a source cluster that does not answer fails it, and the +// branch worker decides when to try again. const DefaultAttemptTimeout = 30 * time.Second -// Dispatcher delivers the branch workers' apply requests to the appliers they name. It implements +// Dispatcher carries out the apply requests the branch workers decide. It implements // git.ApplyRequester, and runs as a leader-elected manager.Runnable, the same replica the branch // workers run on. // -// A request names GitTargets; the dispatcher resolves each to its applier and asks every applier -// once, however many targets share it. Each applier is delivered to independently, on its own -// goroutine and with its own retries, so one that does not answer never holds another. -// -// A newer request for an applier supersedes one not yet sent, because the newer one names a commit -// at least as new and was decided on a quiet branch too: it is always safe to apply in place of the -// older. A CatchUp never weakens what it supersedes, though. A ForeignCommit or Refusal stamps the -// Kustomization when the artifact already holds the commit, and a CatchUp does not, so a CatchUp -// arriving behind a Refusal keeps the Refusal's reason with the newer commit; otherwise the revert -// the Refusal asked for would be lost whenever the artifact had already caught up. Reasons are kept -// per GitTarget, because consent to a Refusal is per GitTarget. +// It keeps no queue and makes no decision of its own. Each request is one attempt for one applier, +// carried out on its own goroutine and reported back to the worker that decided it; the worker owns +// the obligation until an attempt delivers it, and decides every retry on a quiet branch. So nothing +// goes out that a worker did not check on a quiet branch, and nothing here can turn a delayed +// decision into an apply nobody checked. // -// The worker decided each request once, and an attempt can come later, against whatever the -// cluster, the GitTarget and the branch have become. So every attempt, retries included, first -// reads each GitTarget again and keeps only those that still name this applier on this branch, -// that their ClusterProvider still admits, and, for a Refusal, that still consent; a GitTarget that -// no longer consents is asked a CatchUp instead. The request then checks that the Kustomization -// applies the GitProvider's repository and branch before it stamps anything. -// -// A failed request is retried only briefly: for about the shortest quiet period of its GitTargets, -// or not at all when the failure is permanent. A retry is the worker's decision replayed without -// its quiet check, and by the time a long retry lands, live edits may be waiting in an open window -// that the apply would revert. After that it is handed back to the branch worker -// (git.ApplyHandBack), which owes it again and sends it once the branch is quiet. +// What it does check is what can change between the worker's decision and the patch. It reads each +// GitTarget again and asks only for those that are still the incarnation the worker decided for, +// still name this applier on this branch, are still admitted by their ClusterProvider, and, for a +// Refusal, still consent. When no target still holds the request's reason it asks nothing at all, +// rather than ask for something weaker: a weaker request is the worker's to decide. Then the Flux +// client checks the source binding before it stamps. type Dispatcher struct { reader client.Reader sourceClients SourceClients recorder record.EventRecorder log logr.Logger - // Backoff is the retry schedule. Set before Start. - Backoff Backoff - // AttemptTimeout bounds one delivery attempt. Set before Start. + // AttemptTimeout bounds one attempt. Set before Start. AttemptTimeout time.Duration - // HandBack takes back the requests that could not be delivered. Set before Start; nil gives - // them up (the CLI and tests). - HandBack git.ApplyHandBack - // Now is the clock requestedAt stamps and the ledger read. Retries are scheduled on real time. + // Now is the clock requestedAt stamps and the ledger read. Now func() time.Time - inboxMu sync.Mutex - inbox []git.ApplyRequest - wake chan struct{} + runMu sync.RWMutex + run context.Context + attempts sync.WaitGroup ledgerMu sync.RWMutex ledger map[string]ledgerEntry - - // Owned by the Start loop. - pending map[applierKey]*job - inflight map[applierKey]*job - results chan result } var ( @@ -168,75 +114,46 @@ func NewDispatcher( sourceClients: sourceClients, recorder: recorder, log: log, - Backoff: DefaultBackoff(), AttemptTimeout: DefaultAttemptTimeout, Now: time.Now, - wake: make(chan struct{}, 1), ledger: map[string]ledgerEntry{}, - pending: map[applierKey]*job{}, - inflight: map[applierKey]*job{}, - results: make(chan result), } } // NeedLeaderElection is true: the branch workers that decide requests run only on the leader. func (d *Dispatcher) NeedLeaderElection() bool { return true } -// RequestApply queues a request and returns at once; it is called on a branch worker's loop. -// -// A queued request for exactly the same branch, reason and targets is replaced in place rather -// than queued twice: the newer one names the newer commit, and both would resolve to the same -// appliers. Past maxInbox the oldest request is dropped, and said so in the log. -func (d *Dispatcher) RequestApply(req git.ApplyRequest) { - if len(req.Targets) == 0 { - return - } - req.Targets = slices.Clone(req.Targets) - - d.inboxMu.Lock() - replaced := false - for i := range d.inbox { - if sameScope(d.inbox[i], req) { - d.inbox[i] = req - replaced = true - break - } - } - var dropped *git.ApplyRequest - if !replaced { - if len(d.inbox) >= maxInbox { - oldest := d.inbox[0] - dropped = &oldest - d.inbox = d.inbox[1:] - } - d.inbox = append(d.inbox, req) - } - d.inboxMu.Unlock() - - if dropped != nil { - d.log.Error(nil, "apply request inbox full; dropped the oldest request", - "branch", dropped.Branch.String(), "reason", dropped.Reason, "commit", dropped.Commit) - } - select { - case d.wake <- struct{}{}: - default: - } +// Start runs until ctx ends, and then waits for the attempts still running, which end with it. +func (d *Dispatcher) Start(ctx context.Context) error { + d.runMu.Lock() + d.run = ctx + d.runMu.Unlock() + d.log.Info("apply request dispatcher started") + <-ctx.Done() + d.runMu.Lock() + d.run = nil + d.runMu.Unlock() + d.attempts.Wait() + return nil } -// sameScope reports whether two requests are for the same branch, reason and set of targets. -func sameScope(a, b git.ApplyRequest) bool { - if a.Branch != b.Branch || a.Reason != b.Reason || len(a.Targets) != len(b.Targets) { - return false - } - keys := func(refs []itypes.ResourceReference) []string { - out := make([]string, len(refs)) - for i, ref := range refs { - out[i] = ref.Key() - } - sort.Strings(out) - return out +// RequestApply starts one attempt and returns at once; it is called on a branch worker's loop. The +// attempt reports on req.Report when it ends. +func (d *Dispatcher) RequestApply(req git.ApplyRequest) { + d.runMu.RLock() + ctx := d.run + if ctx != nil { + d.attempts.Add(1) + } + d.runMu.RUnlock() + if ctx == nil { + req.Report(git.ApplyResult{ID: req.ID, Outcome: git.ApplyFailed, Err: errNotRunning}) + return } - return slices.Equal(keys(a.Targets), keys(b.Targets)) + go func() { + defer d.attempts.Done() + req.Report(d.attempt(ctx, req)) + }() } // ledgerEntry is the last request delivered for one GitTarget, and which incarnation of it the @@ -269,436 +186,180 @@ func (d *Dispatcher) Forget(ref itypes.ResourceReference) { delete(d.ledger, ref.Key()) } -// applierKey identifies one applier for one branch. The branch is part of it so that two branches -// naming one Kustomization (a misconfiguration ApplierConfigured reports as SourceMismatch) never -// have one branch's commit sent in place of the other's. -type applierKey struct { - branch git.BranchKey - cluster string - namespace string - name string -} - -func (k applierKey) ref() types.NamespacedName { - return types.NamespacedName{Namespace: k.namespace, Name: k.name} -} - -// job is one applier's share of one or more requests. -type job struct { - key applierKey - commit string - targets map[string]jobTarget - // handBacks is how many times these requests were handed back before (git.ApplyRequest). - handBacks int - // failures counts failed attempts, and firstFailure is when the first one settled. - failures int - firstFailure time.Time - due time.Time -} - -// jobTarget is one GitTarget a job asks for, with its own reason. -type jobTarget struct { - ref itypes.ResourceReference - reason v1alpha3.ApplyReason - // quietPeriod is the GitTarget's spec.applier.quietPeriod when the request was resolved, which - // bounds how long the job is retried. - quietPeriod time.Duration -} - -// reason is what the applier is asked: the strongest of the job's GitTargets' reasons. -func (j *job) reason() v1alpha3.ApplyReason { - return strongestReason(j.targets) -} - -func strongestReason(targets map[string]jobTarget) v1alpha3.ApplyReason { - var reason v1alpha3.ApplyReason - for _, target := range targets { - reason = git.StrongerApplyReason(reason, target.reason) - } - return reason -} - -// retryBudget is how long a failing job is retried before it is handed back: the shortest quiet -// period among its GitTargets. A request a quiet period old is as stale as the worker allows a -// catch-up to be, and the worker is the one that can tell whether the branch is still quiet. -func (j *job) retryBudget() time.Duration { - budget := v1alpha3.DefaultApplierQuietPeriod - for i, target := range j.sortedTargets() { - if i == 0 || target.quietPeriod < budget { - budget = target.quietPeriod - } - } - return budget -} - -// absorb takes over an older job this one supersedes: its GitTargets, each keeping the stronger of -// its two reasons (git.StrongerApplyReason), and its hand-back count. -func (j *job) absorb(older *job) { - for key, target := range older.targets { - if mine, ok := j.targets[key]; ok { - target.reason = git.StrongerApplyReason(target.reason, mine.reason) - target.quietPeriod = mine.quietPeriod - } - j.targets[key] = target - } - j.handBacks = max(j.handBacks, older.handBacks) -} - -// sortedTargets returns the job's GitTargets in a stable order. -func (j *job) sortedTargets() []jobTarget { - keys := make([]string, 0, len(j.targets)) - for key := range j.targets { - keys = append(keys, key) - } - sort.Strings(keys) - out := make([]jobTarget, 0, len(keys)) - for _, key := range keys { - out = append(out, j.targets[key]) - } - return out -} - // eligibleTarget is a GitTarget an attempt found may still ask the applier, as read for it. type eligibleTarget struct { target *v1alpha3.GitTarget reason v1alpha3.ApplyReason } -// result is what one attempt did. eligible is empty when no GitTarget may still ask the applier, -// in which case nothing was sent and err is nil. -type result struct { - job *job - eligible []eligibleTarget - outcome Outcome - err error -} - -// Start runs the delivery loop until ctx ends. Attempts still running then are cancelled with it. -func (d *Dispatcher) Start(ctx context.Context) error { - d.log.Info("apply request dispatcher started") - var attempts sync.WaitGroup - defer attempts.Wait() - - for { - d.takeInbox(ctx) - d.startDue(ctx, &attempts) - - var timer *time.Timer - var fire <-chan time.Time - if next, ok := d.nextDue(); ok { - timer = time.NewTimer(time.Until(next)) - fire = timer.C - } - select { - case <-ctx.Done(): - if timer != nil { - timer.Stop() - } - return nil - case <-d.wake: - case res := <-d.results: - d.settle(ctx, res) - case <-fire: - } - if timer != nil { - timer.Stop() - } +// attempt carries out one request and returns its report. +func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.ApplyResult { + ctx, cancel := context.WithTimeout(ctx, d.AttemptTimeout) + defer cancel() + log := d.log.WithValues("applier", req.Applier.String(), "branch", req.Branch.String(), + "reason", req.Reason, "commit", req.Commit) + report := func(outcome git.ApplyOutcome, err error) git.ApplyResult { + return git.ApplyResult{ID: req.ID, Outcome: outcome, Err: err} } -} - -// takeInbox resolves every queued request into per-applier jobs and queues them. -func (d *Dispatcher) takeInbox(ctx context.Context) { - d.inboxMu.Lock() - requests := d.inbox - d.inbox = nil - d.inboxMu.Unlock() - for _, req := range requests { - for _, j := range d.resolve(ctx, req) { - d.enqueue(ctx, j) - } + eligible, want, err := d.recheck(ctx, req) + if err != nil { + return d.failure(ctx, req, nil, report(git.ApplyFailed, err), log) + } + if !holdsReason(eligible, req.Reason) { + d.count(ctx, req.Reason, outcomeDropped) + log.Info("no GitTarget may still ask this applier for this; asking nothing") + return report(git.ApplyDropped, nil) } -} -// resolve turns a request into one job per applier its targets name. A target that is gone, or -// declares no applier, is skipped: the worker decided on the targets it knew, and either can have -// changed since. Everything else about a target is checked at every attempt (recheck). The read is -// the manager's cache, so it costs the loop no round trip. -func (d *Dispatcher) resolve(ctx context.Context, req git.ApplyRequest) []*job { - jobs := map[applierKey]*job{} - var order []applierKey - for _, ref := range req.Targets { - target := &v1alpha3.GitTarget{} - if err := d.reader.Get(ctx, client.ObjectKey{Namespace: ref.Namespace, Name: ref.Name}, target); err != nil { - if !apierrors.IsNotFound(err) { - d.log.Error(err, "cannot read GitTarget for an apply request; skipping it", - "gitTarget", ref.String(), "reason", req.Reason, "commit", req.Commit) - } - continue - } - if target.Spec.Applier == nil { - continue - } - key := applierKeyFor(req.Branch, target) - j := jobs[key] - if j == nil { - j = &job{key: key, commit: req.Commit, handBacks: req.HandBacks, targets: map[string]jobTarget{}} - jobs[key] = j - order = append(order, key) - } - j.targets[ref.Key()] = jobTarget{ - ref: ref, reason: req.Reason, quietPeriod: target.Spec.Applier.EffectiveQuietPeriod(), - } + sourceClient, err := d.sourceClients(ctx, req.Applier.Cluster) + if err != nil { + return d.failure(ctx, req, eligible, + report(git.ApplyFailed, fmt.Errorf("source cluster %q: %w", req.Applier.Cluster, err)), log) } - out := make([]*job, 0, len(order)) - for _, key := range order { - out = append(out, jobs[key]) + flux := NewFlux(sourceClient) + flux.Now = d.Now + outcome, err := flux.Request(ctx, applierName(req.Applier), want, req.Commit, req.Reason) + switch { + case isPermanent(err): + return d.failure(ctx, req, eligible, report(git.ApplyRejected, err), log) + case err != nil: + return d.failure(ctx, req, eligible, report(git.ApplyFailed, err), log) } - return out -} -// applierKeyFor is the applier a GitTarget names, for branch. Its kustomizationRef namespace -// defaults to the GitTarget's own. -func applierKeyFor(branch git.BranchKey, target *v1alpha3.GitTarget) applierKey { - kustomization := target.Spec.Applier.KustomizationRef - namespace := kustomization.Namespace - if namespace == "" { - namespace = target.Namespace + d.count(ctx, req.Reason, outcomeSent) + if outcome == OutcomeCurrent { + log.V(1).Info("applier already holds the commit; nothing to request") + return report(git.ApplyCurrent, nil) } - return applierKey{branch: branch, cluster: target.SourceCluster(), namespace: namespace, name: kustomization.Name} + log.Info("apply requested", "outcome", outcome) + d.delivered(req, eligible) + return report(git.ApplyDelivered, nil) } -// enqueue queues a job, superseding the applier's job that is waiting to be sent, if any. A job -// already being attempted is not superseded: it was sent. The new job waits behind it. -func (d *Dispatcher) enqueue(ctx context.Context, j *job) { - if older, ok := d.pending[j.key]; ok { - d.count(ctx, older.reason(), outcomeSuperseded) - j.absorb(older) - } - d.pending[j.key] = j +// applierName is the Kustomization an applier reference names. +func applierName(ref git.ApplierRef) types.NamespacedName { + return types.NamespacedName{Namespace: ref.Namespace, Name: ref.Name} } -// startDue starts an attempt for every job that is due and whose applier has none running. -func (d *Dispatcher) startDue(ctx context.Context, attempts *sync.WaitGroup) { - now := time.Now() - for key, j := range d.pending { - if _, busy := d.inflight[key]; busy || j.due.After(now) { - continue +// holdsReason reports whether an eligible target still carries the request's reason. +func holdsReason(eligible []eligibleTarget, reason v1alpha3.ApplyReason) bool { + for _, e := range eligible { + if e.reason == reason { + return true } - delete(d.pending, key) - d.inflight[key] = j - attempts.Add(1) - go func() { - defer attempts.Done() - res := d.attempt(ctx, j) - select { - case d.results <- res: - case <-ctx.Done(): - } - }() } -} - -// nextDue is the earliest time a waiting job becomes due, among those that can start then. -func (d *Dispatcher) nextDue() (time.Time, bool) { - var next time.Time - found := false - for key, j := range d.pending { - if _, busy := d.inflight[key]; busy { - continue - } - if !found || j.due.Before(next) { - next, found = j.due, true + return false +} + +// failure counts a failed attempt and, for the first failure of a run, records ApplyRequestFailed +// on the targets it was for. +func (d *Dispatcher) failure( + ctx context.Context, req git.ApplyRequest, eligible []eligibleTarget, res git.ApplyResult, log logr.Logger, +) git.ApplyResult { + d.count(ctx, req.Reason, outcomeFailed) + log.Error(res.Err, "apply attempt failed; the branch worker decides when to try again", + "outcome", res.Outcome, "priorFailures", req.PriorFailures) + if req.PriorFailures == 0 && d.recorder != nil { + for _, e := range eligible { + d.recorder.Eventf(e.target, corev1.EventTypeWarning, EventReasonApplyRequestFailed, + "could not ask %s to apply %s (%s), and will ask again once the branch is quiet: %v", + applierName(req.Applier), describeCommit(req.Commit), e.reason, res.Err) } } - return next, found + return res } -// attempt makes one delivery attempt. It runs on its own goroutine and reads j without writing -// it: the loop leaves a job alone while it is in flight. -func (d *Dispatcher) attempt(ctx context.Context, j *job) result { - ctx, cancel := context.WithTimeout(ctx, d.AttemptTimeout) - defer cancel() - eligible, want, err := d.recheck(ctx, j) - if err != nil || len(eligible) == 0 { - return result{job: j, err: err} - } - reason := strongestEligible(eligible) - sourceClient, err := d.sourceClients(ctx, j.key.cluster) - if err != nil { - return result{job: j, eligible: eligible, err: fmt.Errorf("source cluster %q: %w", j.key.cluster, err)} - } - flux := NewFlux(sourceClient) - flux.Now = d.Now - outcome, err := flux.Request(ctx, j.key.ref(), want, j.commit, reason) - return result{job: j, eligible: eligible, outcome: outcome, err: err} -} - -// recheck reads each of the job's GitTargets again and returns those that may still ask its -// applier, each with the reason it may ask for, and the binding the applier must apply. A read -// that fails is an error, so the attempt is retried rather than a GitTarget dropped on a blip. -func (d *Dispatcher) recheck(ctx context.Context, j *job) ([]eligibleTarget, Binding, error) { +// recheck reads each of the request's GitTargets again and returns those that may still ask its +// applier, with the reason each asks for, and the binding the applier must apply. A read that fails +// is an error, so the attempt fails rather than a GitTarget being dropped on a blip. +func (d *Dispatcher) recheck(ctx context.Context, req git.ApplyRequest) ([]eligibleTarget, Binding, error) { provider := &v1alpha3.GitProvider{} - providerKey := client.ObjectKey{Namespace: j.key.branch.RepoNamespace, Name: j.key.branch.RepoName} + providerKey := client.ObjectKey{Namespace: req.Branch.RepoNamespace, Name: req.Branch.RepoName} if err := d.reader.Get(ctx, providerKey, provider); err != nil { if apierrors.IsNotFound(err) { return nil, Binding{}, nil } return nil, Binding{}, fmt.Errorf("read GitProvider %s: %w", providerKey, err) } - want := Binding{RepositoryURL: provider.Spec.URL, Branch: j.key.branch.Branch} + want := Binding{RepositoryURL: provider.Spec.URL, Branch: req.Branch.Branch} var eligible []eligibleTarget - for _, candidate := range j.sortedTargets() { - target, reason, err := d.recheckTarget(ctx, j.key, candidate) + for _, candidate := range req.Targets { + target, err := d.recheckTarget(ctx, req, candidate) if err != nil { return nil, Binding{}, err } if target != nil { - eligible = append(eligible, eligibleTarget{target: target, reason: reason}) + eligible = append(eligible, eligibleTarget{target: target, reason: candidate.Reason}) } } return eligible, want, nil } -// recheckTarget reads one GitTarget again and returns it, with the reason it may ask for, when it -// may still ask the applier, or nil when it may not. +// recheckTarget reads one GitTarget again and returns it when it may still ask the applier for its +// reason, or nil when it may not. // // A ClusterProvider's source client is shared by every GitTarget that declared the cluster, so a -// client being available is no permission to use it: a GitTarget its ClusterProvider does not -// admit would otherwise drive the applier on the strength of another GitTarget's admission. +// client being available is no permission to use it: a GitTarget its ClusterProvider does not admit +// would otherwise drive the applier on the strength of another GitTarget's admission. func (d *Dispatcher) recheckTarget( - ctx context.Context, key applierKey, candidate jobTarget, -) (*v1alpha3.GitTarget, v1alpha3.ApplyReason, error) { - log := d.log.WithValues("gitTarget", candidate.ref.String(), "applier", key.ref().String()) + ctx context.Context, req git.ApplyRequest, candidate git.ApplyTarget, +) (*v1alpha3.GitTarget, error) { + log := d.log.WithValues("gitTarget", candidate.Ref.String(), "applier", req.Applier.String()) target := &v1alpha3.GitTarget{} - objKey := client.ObjectKey{Namespace: candidate.ref.Namespace, Name: candidate.ref.Name} + objKey := client.ObjectKey{Namespace: candidate.Ref.Namespace, Name: candidate.Ref.Name} if err := d.reader.Get(ctx, objKey, target); err != nil { if apierrors.IsNotFound(err) { - return nil, "", nil + return nil, nil //nolint:nilnil // a GitTarget that is gone asks nothing, and is no error } - return nil, "", fmt.Errorf("read GitTarget %s: %w", candidate.ref, err) + return nil, fmt.Errorf("read GitTarget %s: %w", candidate.Ref, err) } - if !target.DeletionTimestamp.IsZero() || target.Spec.Applier == nil || - target.Namespace != key.branch.RepoNamespace || target.Spec.GitProviderRef.Name != key.branch.RepoName || - target.Spec.Branch != key.branch.Branch || applierKeyFor(key.branch, target) != key { - log.V(1).Info("GitTarget no longer names this applier on this branch; not asking for it") - return nil, "", nil + if target.UID != candidate.UID || !target.DeletionTimestamp.IsZero() || !namesApplier(target, req) { + log.V(1).Info("GitTarget is no longer the one that named this applier on this branch; not asking for it") + return nil, nil //nolint:nilnil // as above } decision, err := authz.GitTargetAdmitted(ctx, d.reader, target) if err != nil { - return nil, "", fmt.Errorf("check GitTarget %s against its ClusterProvider: %w", candidate.ref, err) + return nil, fmt.Errorf("check GitTarget %s against its ClusterProvider: %w", candidate.Ref, err) } if !decision.Allowed { log.Info("GitTarget is not admitted by its ClusterProvider; not asking its applier", "reason", decision.Reason) - return nil, "", nil - } - reason := candidate.reason - if reason == v1alpha3.ApplyReasonRefusal && !git.ConsentsToReapply(target) { - // Consent was withdrawn after the worker decided. The target's applier may still need to - // catch up, which needs no consent. - reason = v1alpha3.ApplyReasonCatchUp - } - return target, reason, nil -} - -// settle records the result of an attempt: delivered, dropped, superseded by a request that -// arrived while it ran, retried later, or handed back. -func (d *Dispatcher) settle(ctx context.Context, res result) { - j := res.job - delete(d.inflight, j.key) - log := d.log.WithValues("applier", j.key.ref().String(), "cluster", j.key.cluster, - "branch", j.key.branch.String(), "reason", j.reason(), "commit", j.commit) - - switch { - case res.err == nil && len(res.eligible) == 0: - d.count(ctx, j.reason(), outcomeDropped) - log.Info("no GitTarget may still ask this applier; dropping the request") - return - case res.err == nil: - reason := strongestEligible(res.eligible) - d.count(ctx, reason, outcomeSent) - if res.outcome == OutcomeCurrent { - log.V(1).Info("applier already holds the commit; nothing to request") - return - } - log.Info("apply requested", "outcome", res.outcome) - d.delivered(j, res.eligible) - return - } - - j.failures++ - now := time.Now() - if j.firstFailure.IsZero() { - j.firstFailure = now - } - if newer, ok := d.pending[j.key]; ok { - // A newer request is waiting, so this one is not retried: the newer one is what to send. - d.count(ctx, j.reason(), outcomeSuperseded) - newer.absorb(j) - log.Info("apply request failed and is superseded by a newer one", "error", res.err.Error()) - return - } - if isPermanent(res.err) || now.Sub(j.firstFailure) >= j.retryBudget() { - d.giveBack(ctx, j, res.err, log) - return + return nil, nil //nolint:nilnil // as above } - delay := d.Backoff.delay(j.failures) - j.due = now.Add(delay) - d.pending[j.key] = j - log.Info("apply request failed; retrying", "attempt", j.failures, "retryIn", delay, "error", res.err.Error()) -} - -func strongestEligible(eligible []eligibleTarget) v1alpha3.ApplyReason { - var reason v1alpha3.ApplyReason - for _, target := range eligible { - reason = git.StrongerApplyReason(reason, target.reason) + if candidate.Reason == v1alpha3.ApplyReasonRefusal && !git.ConsentsToReapply(target) { + log.V(1).Info("GitTarget no longer consents to a Refusal request; not asking for it") + return nil, nil //nolint:nilnil // as above } - return reason + return target, nil } -// giveBack stops retrying a job: it reports the failure, if this is the first time these requests -// failed, and hands every GitTarget back to the branch worker with its own reason. Every one is -// handed back, not only those the last attempt found eligible: the worker owes them again, and -// the attempt that sends them checks them again. -func (d *Dispatcher) giveBack(ctx context.Context, j *job, err error, log logr.Logger) { - d.count(ctx, j.reason(), outcomeFailed) - if j.handBacks == 0 { - d.failed(ctx, j, err) - } - if d.HandBack == nil { - log.Error(err, "apply request failed; giving up", "attempts", j.failures) - return - } - log.Error(err, "apply request failed; handing it back to the branch worker", - "attempts", j.failures, "handBacks", j.handBacks+1) - byReason := map[v1alpha3.ApplyReason][]itypes.ResourceReference{} - var reasons []v1alpha3.ApplyReason - for _, target := range j.sortedTargets() { - if _, seen := byReason[target.reason]; !seen { - reasons = append(reasons, target.reason) - } - byReason[target.reason] = append(byReason[target.reason], target.ref) +// namesApplier reports whether a GitTarget still names the request's applier on its branch. +func namesApplier(target *v1alpha3.GitTarget, req git.ApplyRequest) bool { + if target.Spec.Applier == nil || target.Namespace != req.Branch.RepoNamespace || + target.Spec.GitProviderRef.Name != req.Branch.RepoName || target.Spec.Branch != req.Branch.Branch { + return false } - for _, reason := range reasons { - d.HandBack.HandBackApply(git.ApplyRequest{ - Branch: j.key.branch, Reason: reason, Commit: j.commit, - Targets: byReason[reason], HandBacks: j.handBacks + 1, - }) + ref := target.Spec.Applier.KustomizationRef + namespace := ref.Namespace + if namespace == "" { + namespace = target.Namespace } + return target.SourceCluster() == req.Applier.Cluster && namespace == req.Applier.Namespace && + ref.Name == req.Applier.Name } // delivered records a delivered request in the ledger and as an ApplyRequested Event on every // GitTarget it was for, each with its own reason. -func (d *Dispatcher) delivered(j *job, eligible []eligibleTarget) { +func (d *Dispatcher) delivered(req git.ApplyRequest, eligible []eligibleTarget) { at := metav1.NewTime(d.Now()) d.ledgerMu.Lock() for _, e := range eligible { d.ledger[itypes.NewResourceReference(e.target.Name, e.target.Namespace).Key()] = ledgerEntry{ uid: e.target.UID, status: v1alpha3.GitTargetApplierStatus{ - RequestedCommit: j.commit, + RequestedCommit: req.Commit, RequestedAt: at.DeepCopy(), Reason: e.reason, }, @@ -711,25 +372,7 @@ func (d *Dispatcher) delivered(j *job, eligible []eligibleTarget) { } for _, e := range eligible { d.recorder.Eventf(e.target, corev1.EventTypeNormal, EventReasonApplyRequested, - "asked %s to apply %s (%s)", j.key.ref(), describeCommit(j.commit), e.reason) - } -} - -// failed records a request that could not be delivered as an ApplyRequestFailed Event on every -// GitTarget it was for. The Event needs the object, so a GitTarget that cannot be read is skipped. -func (d *Dispatcher) failed(ctx context.Context, j *job, err error) { - if d.recorder == nil { - return - } - for _, candidate := range j.sortedTargets() { - target := &v1alpha3.GitTarget{} - objKey := client.ObjectKey{Namespace: candidate.ref.Namespace, Name: candidate.ref.Name} - if d.reader.Get(ctx, objKey, target) != nil { - continue - } - d.recorder.Eventf(target, corev1.EventTypeWarning, EventReasonApplyRequestFailed, - "could not ask %s to apply %s (%s) after %d attempts, and will ask again once the branch is quiet: %v", - j.key.ref(), describeCommit(j.commit), candidate.reason, j.failures, err) + "asked %s to apply %s (%s)", applierName(req.Applier), describeCommit(req.Commit), e.reason) } } @@ -744,12 +387,12 @@ func describeCommit(commit string) string { return commit } -// count records one apply_requests_total decision. +// count records one apply_requests_total attempt. func (d *Dispatcher) count(ctx context.Context, reason v1alpha3.ApplyReason, outcome string) { if telemetry.ApplyRequestsTotal == nil { return } - telemetry.ApplyRequestsTotal.Add(ctx, 1, metric.WithAttributes( + telemetry.ApplyRequestsTotal.Add(context.WithoutCancel(ctx), 1, metric.WithAttributes( attribute.String("applier_kind", applierKindKustomization), attribute.String("reason", string(reason)), attribute.String("outcome", outcome), diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index b27a1ac3..e3c3d6ec 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -63,12 +63,6 @@ func reapplying(target *v1alpha3.GitTarget) *v1alpha3.GitTarget { return target } -// quietFor sets a GitTarget's quiet period, which is also how long its requests are retried. -func quietFor(target *v1alpha3.GitTarget, quietPeriod time.Duration) *v1alpha3.GitTarget { - target.Spec.Applier.QuietPeriod = &metav1.Duration{Duration: quietPeriod} - return target -} - // clusterProvider admits the GitTargets in the named namespaces. func clusterProvider(name string, admits ...string) *v1alpha3.ClusterProvider { return &v1alpha3.ClusterProvider{ @@ -90,8 +84,9 @@ func controlPlane() []client.Object { } } -func ksRef(namespace, name string) *fluxmeta.NamespacedObjectReference { - return &fluxmeta.NamespacedObjectReference{Namespace: namespace, Name: name} +// checkoutRef names the Kustomization flux-system/checkout. +func checkoutRef() *fluxmeta.NamespacedObjectReference { + return &fluxmeta.NamespacedObjectReference{Namespace: fluxSystemNS, Name: "checkout"} } func ref(name string) itypes.ResourceReference { return itypes.NewResourceReference(name, targetNS) } @@ -99,58 +94,44 @@ func ref(name string) itypes.ResourceReference { return itypes.NewResourceRefere // uid is the uid gitTarget gives the GitTarget of that name. func uid(name string) types.UID { return types.UID("uid-" + name) } -// sourceStub hands out one fake source cluster's client, failing the first failFirst calls (or -// every call, with failFirst < 0), and remembers which cluster ids it was asked for. +// checkout is the applier most tests ask: flux-system/checkout in the default cluster. +var checkout = git.ApplierRef{Cluster: v1alpha3.DefaultClusterProviderName, Namespace: fluxSystemNS, Name: "checkout"} + +// request is one attempt for an applier to apply testCommit, for the named targets, all asking for +// reason. +func request(applier git.ApplierRef, reason v1alpha3.ApplyReason, names ...string) git.ApplyRequest { + req := git.ApplyRequest{ID: 1, Branch: testBranchKey, Applier: applier, Reason: reason, Commit: testCommit} + for _, name := range names { + req.Targets = append(req.Targets, git.ApplyTarget{Ref: ref(name), UID: uid(name), Reason: reason}) + } + return req +} + +// sourceStub hands out one fake source cluster's client, failing every call while failing is set, +// and remembers which cluster ids it was asked for. type sourceStub struct { - client *dynamicfake.FakeDynamicClient - mu sync.Mutex - failFirst int - calls int - clusters []string + client *dynamicfake.FakeDynamicClient + mu sync.Mutex + failing bool + clusters []string } func (s *sourceStub) get(_ context.Context, clusterID string) (dynamic.Interface, error) { s.mu.Lock() defer s.mu.Unlock() - s.calls++ s.clusters = append(s.clusters, clusterID) - if s.failFirst < 0 || s.calls <= s.failFirst { - return nil, fmt.Errorf("source cluster unreachable (call %d)", s.calls) + if s.failing { + return nil, fmt.Errorf("source cluster %s unreachable", clusterID) } return s.client, nil } -func (s *sourceStub) callCount() int { - s.mu.Lock() - defer s.mu.Unlock() - return s.calls -} - type dispatcherFixture struct { dispatcher *Dispatcher control client.Client source *sourceStub recorder *record.FakeRecorder reader *sdkmetric.ManualReader - handBacks *handBackRecorder -} - -// handBackRecorder records the requests the dispatcher hands back. -type handBackRecorder struct { - mu sync.Mutex - requests []git.ApplyRequest -} - -func (h *handBackRecorder) HandBackApply(req git.ApplyRequest) { - h.mu.Lock() - defer h.mu.Unlock() - h.requests = append(h.requests, req) -} - -func (h *handBackRecorder) got() []git.ApplyRequest { - h.mu.Lock() - defer h.mu.Unlock() - return append([]git.ApplyRequest(nil), h.requests...) } func newDispatcherFixture(t *testing.T, targets []client.Object, sourceObjs ...runtime.Object) *dispatcherFixture { @@ -166,29 +147,14 @@ func newDispatcherFixture(t *testing.T, targets []client.Object, sourceObjs ...r source := &sourceStub{client: newSourceClient(sourceObjs...)} recorder := record.NewFakeRecorder(64) - handBacks := &handBackRecorder{} d := NewDispatcher(controlClient, source.get, recorder, logr.Discard()) - d.Backoff = Backoff{Initial: time.Millisecond, Max: 4 * time.Millisecond} d.Now = func() time.Time { return stampTime } - d.HandBack = handBacks - return &dispatcherFixture{ - dispatcher: d, control: controlClient, source: source, recorder: recorder, reader: reader, handBacks: handBacks, - } + return &dispatcherFixture{dispatcher: d, control: controlClient, source: source, recorder: recorder, reader: reader} } -// start runs the dispatcher until the test ends. -func (f *dispatcherFixture) start(t *testing.T) { - t.Helper() - ctx, cancel := context.WithCancel(context.Background()) - done := make(chan struct{}) - go func() { - defer close(done) - _ = f.dispatcher.Start(ctx) - }() - t.Cleanup(func() { - cancel() - <-done - }) +// attempt carries out one request synchronously, as the dispatcher's goroutine does. +func (f *dispatcherFixture) attempt(req git.ApplyRequest) git.ApplyResult { + return f.dispatcher.attempt(context.Background(), req) } func (f *dispatcherFixture) count(reason v1alpha3.ApplyReason, outcome string) int64 { @@ -198,14 +164,6 @@ func (f *dispatcherFixture) count(reason v1alpha3.ApplyReason, outcome string) i return value } -// waitForCount waits until the counter reaches want. -func (f *dispatcherFixture) waitForCount(t *testing.T, reason v1alpha3.ApplyReason, outcome string, want int64) { - t.Helper() - require.Eventually(t, func() bool { return f.count(reason, outcome) == want }, - 5*time.Second, time.Millisecond, "apply_requests_total{reason=%s,outcome=%s} never reached %d (at %d)", - reason, outcome, want, f.count(reason, outcome)) -} - // events drains the recorded Events. func (f *dispatcherFixture) events() []string { var out []string @@ -229,405 +187,248 @@ func (f *dispatcherFixture) patches() []string { return out } -// TestDispatcher_AsksEachApplierOnce: targets sharing an applier ask it once; a target with no -// applier, or one that is gone, asks nothing; an omitted kustomizationRef namespace is the target's. -func TestDispatcher_AsksEachApplierOnce(t *testing.T) { +// behindSource is a Kustomization flux-system/checkout whose GitRepository on testBranch holds an +// older commit than testCommit, so any request that is sent stamps the GitRepository. +func behindSource() []runtime.Object { + return []runtime.Object{ + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit), + } +} + +// TestDispatcher_AsksOnceForEveryTarget: one attempt asks its applier once for all its targets, +// records each target's own reason, and looks the applier up in its source cluster. +func TestDispatcher_AsksOnceForEveryTarget(t *testing.T) { f := newDispatcherFixture(t, []client.Object{ - gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""), - gitTarget("b", ksRef(fluxSystemNS, "checkout"), ""), - gitTarget("c", ksRef("", "own"), "prod-eu-1"), - gitTarget("no-applier", nil, ""), + reapplying(gitTarget("a", checkoutRef(), "")), + gitTarget("b", checkoutRef(), ""), }, - kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), - gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit), - kustomizationObj(targetNS, "own", gitRepoSourceRef("own")), - gitRepositoryObj(targetNS, "own", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit), + behindSource()..., + ) + req := request(checkout, v1alpha3.ApplyReasonRefusal, "a") + req.Targets = append( + req.Targets, + git.ApplyTarget{Ref: ref("b"), UID: uid("b"), Reason: v1alpha3.ApplyReasonCatchUp}, ) - f.start(t) - - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a"), ref("b"), ref("c"), ref("no-applier"), ref("deleted")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeSent, 2) - assert.ElementsMatch(t, []string{"gitrepositories " + fluxSystemNS, "gitrepositories " + targetNS}, f.patches(), - "one source stamp per applier, however many targets share it") - assert.ElementsMatch(t, []string{v1alpha3.DefaultClusterProviderName, "prod-eu-1"}, f.source.clusters, - "each applier is looked up in its target's source cluster") + res := f.attempt(req) - want := v1alpha3.GitTargetApplierStatus{ - RequestedCommit: testCommit, - RequestedAt: &metav1.Time{Time: stampTime}, - Reason: v1alpha3.ApplyReasonForeignCommit, - } - for _, name := range []string{"a", "b", "c"} { + assert.Equal(t, git.ApplyResult{ID: 1, Outcome: git.ApplyDelivered}, res) + assert.Equal(t, []string{"gitrepositories " + fluxSystemNS}, f.patches(), "one stamp for the applier") + assert.Equal(t, []string{v1alpha3.DefaultClusterProviderName}, f.source.clusters) + for name, want := range map[string]v1alpha3.ApplyReason{ + "a": v1alpha3.ApplyReasonRefusal, "b": v1alpha3.ApplyReasonCatchUp, + } { got, ok := f.dispatcher.LastRequest(ref(name), uid(name)) require.True(t, ok, name) - assert.True(t, got.RequestedAt.Equal(want.RequestedAt), name) - assert.Equal(t, want.RequestedCommit, got.RequestedCommit, name) - assert.Equal(t, want.Reason, got.Reason, name) + assert.Equal(t, testCommit, got.RequestedCommit, name) + assert.Equal(t, want, got.Reason, name) + assert.True(t, got.RequestedAt.Equal(&metav1.Time{Time: stampTime}), name) } - _, ok := f.dispatcher.LastRequest(ref("no-applier"), uid("no-applier")) - assert.False(t, ok, "a target without an applier records no request") - assert.ElementsMatch(t, []string{ - "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)", - "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)", - "Normal ApplyRequested asked team-a/own to apply 4f2c1ab9 (ForeignCommit)", + "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (Refusal)", + "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (CatchUp)", }, f.events()) + assert.Equal(t, int64(1), f.count(v1alpha3.ApplyReasonRefusal, outcomeSent)) } -// TestDispatcher_RetriesThenDelivers: a failed attempt is retried on the backoff, and only the -// delivery counts. -func TestDispatcher_RetriesThenDelivers(t *testing.T) { - f := newDispatcherFixture(t, - []client.Object{reapplying(gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""))}, - kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), - gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), - ) - f.source.failFirst = 2 - f.start(t) - - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonRefusal, outcomeSent, 1) - - assert.Equal(t, 3, f.source.callCount(), "two failures, then the attempt that landed") - assert.Equal(t, int64(0), f.count(v1alpha3.ApplyReasonRefusal, outcomeFailed)) - assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches(), - "a refusal at the artifact's commit stamps the kustomization") - got, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) - require.True(t, ok) - assert.Equal(t, v1alpha3.ApplyReasonRefusal, got.Reason) -} - -// TestDispatcher_TheLedgerIsPerIncarnation: a GitTarget deleted and recreated under the same name -// does not inherit its predecessor's last request, and a forgotten one holds nothing. -func TestDispatcher_TheLedgerIsPerIncarnation(t *testing.T) { +// TestDispatcher_LooksTheApplierUpInItsCluster: the applier is the request's, in the source cluster +// it names, under its own namespace and name; here a GitTarget whose kustomizationRef omits the +// namespace and so names one in its own. +func TestDispatcher_LooksTheApplierUpInItsCluster(t *testing.T) { + own := git.ApplierRef{Cluster: "prod-eu-1", Namespace: targetNS, Name: "own"} f := newDispatcherFixture(t, - []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, - kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), - gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit), + []client.Object{gitTarget("c", &fluxmeta.NamespacedObjectReference{Name: "own"}, "prod-eu-1")}, + kustomizationObj(targetNS, "own", gitRepoSourceRef("own-source")), + gitRepositoryObj(targetNS, "own-source", testRepoURL, branchRef(testBranch), ""), ) - f.start(t) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeSent, 1) - _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) - assert.True(t, ok) - _, ok = f.dispatcher.LastRequest(ref("a"), "the-recreated-one") - assert.False(t, ok, "a recreated GitTarget does not inherit the last request") - f.dispatcher.Forget(ref("a")) - _, ok = f.dispatcher.LastRequest(ref("a"), uid("a")) - assert.False(t, ok) + assert.Equal(t, git.ApplyDelivered, f.attempt(request(own, v1alpha3.ApplyReasonForeignCommit, "c")).Outcome) + assert.Equal(t, []string{"prod-eu-1"}, f.source.clusters) + assert.Equal(t, []string{"gitrepositories " + targetNS}, f.patches()) } -// TestDispatcher_HandsBackAfterItsRetryBudget is the review's finding 4: a request that kept -// failing was given up on, while the worker had already cleared its debt, so nothing asked again -// until an unrelated commit. It is now retried for about one quiet period and handed back to the -// worker, and the failure is reported once, not again when the worker's next request fails too. -func TestDispatcher_HandsBackAfterItsRetryBudget(t *testing.T) { - f := newDispatcherFixture(t, []client.Object{ - quietFor(gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""), 20*time.Millisecond), - gitTarget("b", ksRef(fluxSystemNS, "checkout"), ""), - }) - f.source.failFirst = -1 - f.start(t) +// TestDispatcher_AFailureIsReportedAndNeverRetried: retrying is the branch worker's decision, on a +// quiet branch. The dispatcher reports the failure, and records ApplyRequestFailed only for the +// first failure of a run. +func TestDispatcher_AFailureIsReportedAndNeverRetried(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{gitTarget("a", checkoutRef(), "")}) + f.source.failing = true - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a"), ref("b")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) - - assert.Greater(t, f.source.callCount(), 1, "retried within the budget before handing back") - require.Eventually(t, func() bool { return len(f.handBacks.got()) == 1 }, 5*time.Second, time.Millisecond) - assert.Equal(t, git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a"), ref("b")}, HandBacks: 1, - }, f.handBacks.got()[0], "every GitTarget goes back, with its reason and one more hand-back") - _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) - assert.False(t, ok, "nothing was delivered") + res := f.attempt(request(checkout, v1alpha3.ApplyReasonForeignCommit, "a")) + assert.Equal(t, git.ApplyFailed, res.Outcome) + require.Error(t, res.Err) + assert.Len(t, f.source.clusters, 1, "one attempt, no retry") events := f.events() - require.Len(t, events, 2, "one Warning per GitTarget") - const want = "Warning ApplyRequestFailed could not ask flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)" - assert.True(t, strings.HasPrefix(events[0], want), events[0]) - - // The worker sends it again, and it fails again. - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a"), ref("b")}, HandBacks: 1, - }) - require.Eventually(t, func() bool { return len(f.handBacks.got()) == 2 }, 5*time.Second, time.Millisecond) - assert.Equal(t, 2, f.handBacks.got()[1].HandBacks) - assert.Empty(t, f.events(), "a failure already reported is not reported again") -} - -// TestDispatcher_PermanentFailureIsHandedBackAtOnce: a Kustomization a request cannot work with is -// not retried; the worker asks again later, in case it was fixed. -func TestDispatcher_PermanentFailureIsHandedBackAtOnce(t *testing.T) { - f := newDispatcherFixture(t, - []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, - kustomizationObj(fluxSystemNS, "checkout", map[string]any{"kind": "Bucket", "name": "platform"}), - ) - f.start(t) + require.Len(t, events, 1) + assert.True(t, strings.HasPrefix(events[0], + "Warning ApplyRequestFailed could not ask flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)"), events[0]) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) - assert.Equal(t, 1, f.source.callCount()) - require.Eventually(t, func() bool { return len(f.handBacks.got()) == 1 }, 5*time.Second, time.Millisecond) + retry := request(checkout, v1alpha3.ApplyReasonForeignCommit, "a") + retry.PriorFailures = 1 + assert.Equal(t, git.ApplyFailed, f.attempt(retry).Outcome) + assert.Empty(t, f.events(), "a failure already reported is not reported again") + assert.Equal(t, int64(2), f.count(v1alpha3.ApplyReasonForeignCommit, outcomeFailed)) + _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) + assert.False(t, ok) } -// TestDispatcher_ASourceMismatchIsNeverStamped is the review's finding 2: ApplierConfigured said -// SourceMismatch, and a foreign commit stamped the GitRepository of the other branch anyway. -func TestDispatcher_ASourceMismatchIsNeverStamped(t *testing.T) { +// TestDispatcher_ASourceMismatchIsRejectedAndNeverStamped is the first review's finding 2: +// ApplierConfigured said SourceMismatch, and a foreign commit stamped the other branch's source. +func TestDispatcher_ASourceMismatchIsRejectedAndNeverStamped(t *testing.T) { f := newDispatcherFixture(t, - []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + []client.Object{gitTarget("a", checkoutRef(), "")}, kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef("release"), "release@sha1:"+olderCommit), ) - f.start(t) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeFailed, 1) - assert.Empty(t, f.patches(), "a source that follows another branch is never stamped") - assert.Equal(t, 1, f.source.callCount(), "and a mismatch is not retried") + res := f.attempt(request(checkout, v1alpha3.ApplyReasonForeignCommit, "a")) + + assert.Equal(t, git.ApplyRejected, res.Outcome, "only a change to the applier can make it succeed") + assert.Empty(t, f.patches()) } -// TestDispatcher_ATargetItsClusterProviderDeniesAsksNothing is the review's finding 1: the source -// client is shared by every GitTarget that declared the cluster, so a GitTarget its ClusterProvider -// does not admit drove the applier on another GitTarget's admission. +// TestDispatcher_ATargetItsClusterProviderDeniesAsksNothing is the first review's finding 1: the +// source client is shared by every GitTarget that declared the cluster, so a GitTarget its +// ClusterProvider does not admit drove the applier on another GitTarget's admission. func TestDispatcher_ATargetItsClusterProviderDeniesAsksNothing(t *testing.T) { + restricted := git.ApplierRef{Cluster: "restricted", Namespace: fluxSystemNS, Name: "checkout"} f := newDispatcherFixture(t, []client.Object{ - gitTarget("admitted", ksRef(fluxSystemNS, "checkout"), ""), - gitTarget("denied", ksRef(fluxSystemNS, "restricted"), "restricted"), + gitTarget("denied", checkoutRef(), "restricted"), clusterProvider("restricted", "some-other-namespace"), }, - kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), - kustomizationObj(fluxSystemNS, "restricted", gitRepoSourceRef("platform")), - gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + behindSource()..., ) - f.start(t) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("admitted"), ref("denied")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeSent, 1) - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeDropped, 1) + res := f.attempt(request(restricted, v1alpha3.ApplyReasonForeignCommit, "denied")) - assert.Equal(t, []string{v1alpha3.DefaultClusterProviderName}, f.source.clusters, - "the denied GitTarget's cluster client is never even asked for") - _, ok := f.dispatcher.LastRequest(ref("denied"), uid("denied")) - assert.False(t, ok) + assert.Equal(t, git.ApplyDropped, res.Outcome) + assert.Empty(t, f.source.clusters, "the denied GitTarget's cluster client is never even asked for") + assert.Equal(t, int64(1), f.count(v1alpha3.ApplyReasonForeignCommit, outcomeDropped)) } -// TestDispatcher_EveryRetryChecksAgain: a GitTarget admitted when its request was decided, and -// denied before a retry, is not asked for on the retry. The same holds for a GitTarget that moved -// to another branch. -func TestDispatcher_EveryRetryChecksAgain(t *testing.T) { - for name, change := range map[string]func(ctx context.Context, c client.Client) error{ - "its ClusterProvider stops admitting it": func(ctx context.Context, c client.Client) error { - provider := &v1alpha3.ClusterProvider{} - if err := c.Get(ctx, client.ObjectKey{Name: v1alpha3.DefaultClusterProviderName}, provider); err != nil { - return err - } - provider.Spec.AccessFrom.Names = []string{"some-other-namespace"} - return c.Update(ctx, provider) - }, - "it moves to another branch": func(ctx context.Context, c client.Client) error { - target := &v1alpha3.GitTarget{} - if err := c.Get(ctx, client.ObjectKey{Namespace: targetNS, Name: "a"}, target); err != nil { - return err - } - target.Spec.Branch = "release" - return c.Update(ctx, target) - }, - } { - t.Run(name, func(t *testing.T) { - f := newDispatcherFixture( - t, - []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, - kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), - gitRepositoryObj( - fluxSystemNS, - "platform", - testRepoURL, - branchRef(testBranch), - "main@sha1:"+olderCommit, - ), - ) - f.dispatcher.Backoff = Backoff{Initial: 50 * time.Millisecond, Max: 50 * time.Millisecond} - f.source.failFirst = 1 - f.start(t) - - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - require.Eventually(t, func() bool { return f.source.callCount() == 1 }, 5*time.Second, time.Millisecond) - require.NoError(t, change(context.Background(), f.control)) - - f.waitForCount(t, v1alpha3.ApplyReasonForeignCommit, outcomeDropped, 1) - assert.Empty(t, f.patches()) - }) - } -} - -// TestDispatcher_ARefusalNeedsConsentAtEveryAttempt is the dispatcher's half of the review's -// finding 3: consent is per GitTarget, so a GitTarget suspended after its Refusal was decided is -// asked only a CatchUp, while a sibling on the same applier that still consents keeps its Refusal. -func TestDispatcher_ARefusalNeedsConsentAtEveryAttempt(t *testing.T) { - suspended := reapplying(gitTarget("suspended", ksRef(fluxSystemNS, "checkout"), "")) +// TestDispatcher_WithdrawnConsentAsksNothing is the executor's half of the second review's first +// finding. A Refusal whose target withdrew consent used to become a CatchUp sent at once, which +// fetched a behind artifact without the quiet period. Now it asks nothing, even with the artifact +// behind, and the worker decides what is still owed. +func TestDispatcher_WithdrawnConsentAsksNothing(t *testing.T) { + suspended := reapplying(gitTarget("a", checkoutRef(), "")) suspended.Spec.Suspend = true f := newDispatcherFixture(t, - []client.Object{ - suspended, - reapplying(gitTarget("consenting", ksRef(fluxSystemNS, "checkout"), "")), - reapplying(gitTarget("alone", ksRef(fluxSystemNS, "alone"), "")), - }, - kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), - kustomizationObj(fluxSystemNS, "alone", gitRepoSourceRef("platform")), - gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + []client.Object{suspended, gitTarget("b", checkoutRef(), "")}, + behindSource()..., + ) + req := request(checkout, v1alpha3.ApplyReasonRefusal, "a") + req.Targets = append( + req.Targets, + git.ApplyTarget{Ref: ref("b"), UID: uid("b"), Reason: v1alpha3.ApplyReasonCatchUp}, ) - f.start(t) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("suspended"), ref("consenting")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonRefusal, outcomeSent, 1) - consenting, ok := f.dispatcher.LastRequest(ref("consenting"), uid("consenting")) - require.True(t, ok) - assert.Equal(t, v1alpha3.ApplyReasonRefusal, consenting.Reason) - suspendedStatus, ok := f.dispatcher.LastRequest(ref("suspended"), uid("suspended")) - require.True(t, ok, "the shared stamp reached its applier too") - assert.Equal(t, v1alpha3.ApplyReasonCatchUp, suspendedStatus.Reason, "recorded as what it may ask for") - - // A Refusal whose only GitTarget withdrew consent is a CatchUp, which an applier already at the - // commit satisfies: the only stamp is the one the first request made. - alone := &v1alpha3.GitTarget{} - require.NoError(t, f.control.Get(context.Background(), client.ObjectKey{Namespace: targetNS, Name: "alone"}, alone)) - alone.Spec.OnRefusal = v1alpha3.RefusalActionIgnore - require.NoError(t, f.control.Update(context.Background(), alone)) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("alone")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonCatchUp, outcomeSent, 1) - assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches()) + res := f.attempt(req) + + assert.Equal(t, git.ApplyDropped, res.Outcome, "never a weaker request in its place") + assert.Empty(t, f.patches()) } -// TestDispatcher_NewerRequestSupersedes: a request waiting on its retry is replaced by a newer one -// for the same applier, which is sent at once with the newer commit. The newer one is a CatchUp -// and the older a Refusal, so it keeps the Refusal's reason: the artifact already holds the newer -// commit, and a CatchUp would then stamp nothing and drop the revert. -func TestDispatcher_NewerRequestSupersedes(t *testing.T) { - f := newDispatcherFixture(t, - []client.Object{reapplying(gitTarget("a", ksRef(fluxSystemNS, "checkout"), ""))}, - kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), - gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), - ) - f.dispatcher.Backoff = Backoff{Initial: time.Hour, Max: time.Hour} - f.source.failFirst = 1 - f.start(t) +// TestDispatcher_ATargetThatChangedAsksNothing: the attempt asks only for targets that are still the +// incarnation the worker decided for and still name the applier on its branch. The second review's +// third finding is the first case: a recreated GitTarget executed its predecessor's request. +func TestDispatcher_ATargetThatChangedAsksNothing(t *testing.T) { + for name, change := range map[string]func(*v1alpha3.GitTarget){ + "it was deleted and recreated": func(g *v1alpha3.GitTarget) { g.UID = "uid-recreated" }, + "it moved to another branch": func(g *v1alpha3.GitTarget) { g.Spec.Branch = "release" }, + "it names another Kustomization": func(g *v1alpha3.GitTarget) { g.Spec.Applier.KustomizationRef.Name = "other" }, + "it names no applier any more": func(g *v1alpha3.GitTarget) { g.Spec.Applier = nil }, + "it applies through another cluster": func(g *v1alpha3.GitTarget) { + g.Spec.ClusterProviderRef = &fluxmeta.LocalObjectReference{Name: "prod-eu-1"} + }, + } { + t.Run(name, func(t *testing.T) { + target := gitTarget("a", checkoutRef(), "") + change(target) + f := newDispatcherFixture(t, []client.Object{target}, behindSource()...) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, Commit: olderCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - require.Eventually(t, func() bool { return f.source.callCount() == 1 }, 5*time.Second, time.Millisecond) + res := f.attempt(request(checkout, v1alpha3.ApplyReasonForeignCommit, "a")) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonCatchUp, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonRefusal, outcomeSent, 1) - - assert.Equal(t, int64(1), f.count(v1alpha3.ApplyReasonRefusal, outcomeSuperseded), - "the failed Refusal at the older commit was superseded, not retried") - assert.Equal(t, int64(0), f.count(v1alpha3.ApplyReasonCatchUp, outcomeSent)) - assert.Equal(t, []string{"kustomizations " + fluxSystemNS}, f.patches()) - got, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) - require.True(t, ok) - assert.Equal(t, testCommit, got.RequestedCommit, "the newer commit is the one sent") - assert.Equal(t, v1alpha3.ApplyReasonRefusal, got.Reason) + assert.Equal(t, git.ApplyDropped, res.Outcome) + assert.Empty(t, f.patches()) + _, ok := f.dispatcher.LastRequest(ref("a"), target.UID) + assert.False(t, ok, "and records nothing for the incarnation it finds") + }) + } } // TestDispatcher_CatchUpAlreadyCurrent: a CatchUp the applier already satisfies counts as sent and // records nothing, since nothing was asked of it. func TestDispatcher_CatchUpAlreadyCurrent(t *testing.T) { f := newDispatcherFixture(t, - []client.Object{gitTarget("a", ksRef(fluxSystemNS, "checkout"), "")}, + []client.Object{gitTarget("a", checkoutRef(), "")}, kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), ) - f.start(t) - f.dispatcher.RequestApply(git.ApplyRequest{ - Branch: testBranchKey, Reason: v1alpha3.ApplyReasonCatchUp, Commit: testCommit, - Targets: []itypes.ResourceReference{ref("a")}, - }) - f.waitForCount(t, v1alpha3.ApplyReasonCatchUp, outcomeSent, 1) + res := f.attempt(request(checkout, v1alpha3.ApplyReasonCatchUp, "a")) + assert.Equal(t, git.ApplyCurrent, res.Outcome) assert.Empty(t, f.patches()) assert.Empty(t, f.events()) _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) assert.False(t, ok) } -// TestDispatcher_RequestApplyNeverBlocks: with no loop running, requests queue without blocking; an -// identical scope is replaced in place, and past the bound the oldest is dropped. -func TestDispatcher_RequestApplyNeverBlocks(t *testing.T) { - f := newDispatcherFixture(t, nil) - d := f.dispatcher +// TestDispatcher_TheLedgerIsPerIncarnation: a GitTarget deleted and recreated under the same name +// does not inherit its predecessor's last request, and a forgotten one holds nothing. +func TestDispatcher_TheLedgerIsPerIncarnation(t *testing.T) { + f := newDispatcherFixture( + t, + []client.Object{gitTarget("a", checkoutRef(), "")}, + behindSource()...) + require.Equal(t, git.ApplyDelivered, + f.attempt(request(checkout, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) - same := func(commit string) git.ApplyRequest { - return git.ApplyRequest{Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit, Commit: commit, - Targets: []itypes.ResourceReference{ref("a"), ref("b")}} - } - d.RequestApply(same(olderCommit)) - reordered := same(testCommit) - reordered.Targets = []itypes.ResourceReference{ref("b"), ref("a")} - d.RequestApply(reordered) - d.RequestApply(git.ApplyRequest{Branch: testBranchKey, Reason: v1alpha3.ApplyReasonForeignCommit}) - - d.inboxMu.Lock() - require.Len(t, d.inbox, 1, "an identical scope is replaced, and an empty request is ignored") - assert.Equal(t, testCommit, d.inbox[0].Commit) - d.inboxMu.Unlock() - - for i := range maxInbox + 10 { - d.RequestApply(git.ApplyRequest{Branch: testBranchKey, Reason: v1alpha3.ApplyReasonRefusal, - Commit: testCommit, Targets: []itypes.ResourceReference{ref(fmt.Sprintf("t%d", i))}}) - } - d.inboxMu.Lock() - defer d.inboxMu.Unlock() - assert.Len(t, d.inbox, maxInbox) - assert.Equal(t, "t10", d.inbox[0].Targets[0].Name, "the oldest requests were dropped") + _, ok := f.dispatcher.LastRequest(ref("a"), uid("a")) + assert.True(t, ok) + _, ok = f.dispatcher.LastRequest(ref("a"), "the-recreated-one") + assert.False(t, ok, "a recreated GitTarget does not inherit the last request") + f.dispatcher.Forget(ref("a")) + _, ok = f.dispatcher.LastRequest(ref("a"), uid("a")) + assert.False(t, ok) } -func TestBackoff_Delay(t *testing.T) { - b := Backoff{Initial: 2 * time.Second, Max: time.Minute} - for failures, want := range map[int]time.Duration{ - 1: 2 * time.Second, 2: 4 * time.Second, 3: 8 * time.Second, 5: 32 * time.Second, 6: time.Minute, 20: time.Minute, - } { - assert.Equal(t, want, b.delay(failures), "after %d failures", failures) - } +// TestDispatcher_RequestApplyReportsFromItsOwnGoroutine: RequestApply returns at once and the +// attempt reports when it ends; before Start, or after it, an attempt fails at once. +func TestDispatcher_RequestApplyReportsFromItsOwnGoroutine(t *testing.T) { + f := newDispatcherFixture( + t, + []client.Object{gitTarget("a", checkoutRef(), "")}, + behindSource()...) + reports := make(chan git.ApplyResult, 1) + req := request(checkout, v1alpha3.ApplyReasonForeignCommit, "a") + req.Report = func(res git.ApplyResult) { reports <- res } + + f.dispatcher.RequestApply(req) + res := <-reports + assert.Equal(t, git.ApplyFailed, res.Outcome, "not running yet") + require.ErrorIs(t, res.Err, errNotRunning) + + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan struct{}) + go func() { + defer close(done) + _ = f.dispatcher.Start(ctx) + }() + require.Eventually(t, func() bool { + f.dispatcher.runMu.RLock() + defer f.dispatcher.runMu.RUnlock() + return f.dispatcher.run != nil + }, 5*time.Second, time.Millisecond) + + f.dispatcher.RequestApply(req) + assert.Equal(t, git.ApplyDelivered, (<-reports).Outcome) + cancel() + <-done } func TestStrongerApplyReason(t *testing.T) { diff --git a/internal/git/apply_attempt_test.go b/internal/git/apply_attempt_test.go new file mode 100644 index 00000000..3397440f --- /dev/null +++ b/internal/git/apply_attempt_test.go @@ -0,0 +1,333 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +// What happens to an apply attempt after the worker decided it, and how one worker serves the +// many GitTargets and appliers on its branch. The worker owns every obligation until an attempt +// delivers it; an attempt that does not deliver gives back only what still stands, and its retry is +// decided again like a first attempt. See docs/design/apply-trigger.md. + +import ( + "context" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + k8stypes "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client" + + configv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// manualLoop is applyTriggerLoop with a requester that reports nothing until the test does. +func manualLoop(t *testing.T, targets ...client.Object) (*branchWorkerEventLoop, *recordingRequester) { + t.Helper() + loop, requester := applyTriggerLoop(t, targets...) + requester.manual = true + return loop, requester +} + +// report delivers the requester's report of an attempt, as the requester's goroutine would. +func report(req ApplyRequest, outcome ApplyOutcome) { + req.Report(ApplyResult{ID: req.ID, Outcome: outcome}) +} + +// dueNow makes every applier's backoff run out. +func dueNow(loop *branchWorkerEventLoop) { + for _, state := range loop.appliers { + state.retryAt = time.Time{} + } +} + +// updateTarget changes a GitTarget in the loop's fake client. +func updateTarget(t *testing.T, loop *branchWorkerEventLoop, name string, mutate func(*configv1alpha3.GitTarget)) { + t.Helper() + target := &configv1alpha3.GitTarget{} + require.NoError( + t, + loop.w.Client.Get(context.Background(), client.ObjectKey{Name: name, Namespace: "team-a"}, target), + ) + mutate(target) + require.NoError(t, loop.w.Client.Update(context.Background(), target)) +} + +// reapplying makes a GitTarget consent to a Refusal request. +func reapplying(g *configv1alpha3.GitTarget) { g.Spec.OnRefusal = configv1alpha3.RefusalActionReapply } + +// oweAlphaRefusal owes alpha's applier a Refusal request for a standing refusal of its deployments. +func oweAlphaRefusal(loop *branchWorkerEventLoop) refusalKey { + key := refusalKeyFor(itypes.NewResourceReference("alpha", "team-a"), "deployments") + loop.oweReapply(key, owedRefusal{detail: "a base-owned field", uid: "uid-alpha"}, "obs-alpha") + return key +} + +// TestApplier_ARetryIsDecidedAgainOnAQuietBranch: an attempt that failed is not replayed blind. It +// waits out its backoff, then waits for a quiet branch like a first attempt, and names the tip as +// it is then. +func TestApplier_ARetryIsDecidedAgainOnAQuietBranch(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true)) + owesForeign(loop.w) + loop.serviceAppliers() + first := requester.taken() + require.Len(t, first, 1) + + report(first[0], ApplyFailed) + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "not before its backoff") + assert.NotNil(t, loop.applierTimer, "the backoff is armed") + + dueNow(loop) + loop.openWindow = &openWindow{} + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "not while the branch is busy") + + loop.openWindow = nil + loop.w.debt.observe(RemoteObservation{Commit: "ccc", By: ObservedByPush}) + loop.serviceAppliers() + retry := requester.taken() + require.Len(t, retry, 1) + assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, retry[0].Reason, "the obligation came back") + assert.Equal(t, "ccc", retry[0].Commit, "at the tip as it is now") + assert.Equal(t, 1, retry[0].PriorFailures, "so the requester does not report the failure twice") +} + +// TestApplier_WithdrawnConsentNeverBecomesAnImmediateApply is the second review's first finding. +// The requester used to turn a Refusal whose target withdrew consent into a CatchUp and send it at +// once, skipping the quiet period, so a behind artifact was fetched over an unpublished edit. Now +// the requester only reports that the reason no longer holds, and the worker decides: no Refusal, +// and a CatchUp only after the quiet period. +func TestApplier_WithdrawnConsentNeverBecomesAnImmediateApply(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true, reapplying, func(g *configv1alpha3.GitTarget) { + g.Spec.Applier.QuietPeriod = &metav1.Duration{Duration: time.Hour} + })) + loop.w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByFetch}) // owes a catch-up + oweAlphaRefusal(loop) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1) + require.Equal(t, configv1alpha3.ApplyReasonRefusal, out[0].Reason) + + updateTarget( + t, + loop, + "alpha", + func(g *configv1alpha3.GitTarget) { g.Spec.OnRefusal = configv1alpha3.RefusalActionIgnore }, + ) + report(out[0], ApplyDropped) + loop.serviceAppliers() + dueNow(loop) + loop.serviceAppliers() + + assert.Empty(t, requester.taken(), "the catch-up waits out the hour-long quiet period") + assert.Empty(t, loop.reapplyOwed, "and the Refusal is gone with the consent") + assert.True(t, loop.appliers[applierRef("alpha")].catchUp) +} + +// TestApplier_ARecoveredRefusalIsNotRetried is the second review's second finding: a refusal that +// recovered while its attempt was out came back with the failed attempt and was requested again. +func TestApplier_ARecoveredRefusalIsNotRetried(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true, reapplying)) + key := oweAlphaRefusal(loop) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1) + + loop.refusalRecovered(key.target, key.collection) + report(out[0], ApplyFailed) + loop.serviceAppliers() + dueNow(loop) + loop.serviceAppliers() + + assert.Empty(t, requester.taken(), "a recovered refusal has nothing left to ask for") + assert.Empty(t, loop.reapplyOwed) +} + +// TestApplier_ARecreatedTargetInheritsNothing is the second review's third finding: an obligation +// decided for one incarnation of a GitTarget executed for another created under the same name. +// Requests carry the uid, and the worker drops what was owed to an incarnation that is gone. +func TestApplier_ARecreatedTargetInheritsNothing(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true, reapplying)) + loop.openWindow = &openWindow{} + oweAlphaRefusal(loop) + + old := &configv1alpha3.GitTarget{} + require.NoError( + t, + loop.w.Client.Get(context.Background(), client.ObjectKey{Name: "alpha", Namespace: "team-a"}, old), + ) + require.NoError(t, loop.w.Client.Delete(context.Background(), old)) + recreated := applierTestTarget("alpha", true, reapplying) + recreated.UID = "uid-alpha-recreated" + require.NoError(t, loop.w.Client.Create(context.Background(), recreated)) + + loop.openWindow = nil + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "the refusal was the old incarnation's") + assert.Empty(t, loop.reapplyOwed) + + owesForeign(loop.w) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1) + assert.Equal(t, k8stypes.UID("uid-alpha-recreated"), out[0].Targets[0].UID, + "an attempt names the incarnation it was decided for") +} + +// TestApplier_EachApplierRetriesOnItsOwnClock is the second review's fourth finding: hand-backs +// shared one branch-wide deadline and count, so an applier that kept failing held back one that had +// recovered. +func TestApplier_EachApplierRetriesOnItsOwnClock(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true), applierTestTarget("bravo", true)) + owesForeign(loop.w) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 2) + + loop.appliers[applierRef("bravo")].failures = 6 // a long run of failures behind it + for _, req := range out { + report(req, ApplyFailed) + } + loop.serviceAppliers() + assert.Equal(t, 1, loop.appliers[applierRef("alpha")].failures) + assert.Equal(t, 7, loop.appliers[applierRef("bravo")].failures) + + loop.appliers[applierRef("alpha")].retryAt = time.Now().Add(-time.Second) + loop.serviceAppliers() + retry := requester.taken() + require.Len(t, retry, 1, "alpha is due; bravo's longer backoff is bravo's alone") + assert.Equal(t, applierRef("alpha"), retry[0].Applier) +} + +// TestApplier_OneAttemptForEveryTargetNamingIt: GitTargets that share a Kustomization are asked for +// together, each with its own reason, and the shared applier waits for the longest of their quiet +// periods, because its apply reaches every one of their folders. +func TestApplier_OneAttemptForEveryTargetNamingIt(t *testing.T) { + shared := func(quiet time.Duration) func(*configv1alpha3.GitTarget) { + return func(g *configv1alpha3.GitTarget) { + g.Spec.Applier = testApplier("shared") + g.Spec.Applier.QuietPeriod = &metav1.Duration{Duration: quiet} + } + } + loop, requester := applyTriggerLoop(t, + applierTestTarget("alpha", true, shared(time.Minute), reapplying), + applierTestTarget("bravo", true, shared(time.Hour)), + ) + loop.w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByPush}) + loop.serviceAppliers() + assert.Equal(t, time.Hour, loop.appliers[applierRef("shared")].quietPeriod()) + + loop.quietSince = time.Now().Add(-2 * time.Minute) + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "past alpha's quiet period, not bravo's") + + oweAlphaRefusal(loop) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1, "one attempt for the shared applier") + assert.Equal(t, configv1alpha3.ApplyReasonRefusal, out[0].Reason) + assert.Equal(t, []ApplyTarget{ + { + Ref: itypes.NewResourceReference("alpha", "team-a"), + UID: "uid-alpha", + Reason: configv1alpha3.ApplyReasonRefusal, + }, + { + Ref: itypes.NewResourceReference("bravo", "team-a"), + UID: "uid-bravo", + Reason: configv1alpha3.ApplyReasonCatchUp, + }, + }, out[0].Targets, "a Refusal applies the tip, so it answers the catch-up owed to bravo as well") + assert.False(t, loop.appliers[applierRef("shared")].catchUp) +} + +// TestApplier_AnAttemptOutHoldsTheNext: one attempt per applier at a time. What is owed while it is +// out waits for its report, and goes in the attempt after. +func TestApplier_AnAttemptOutHoldsTheNext(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true)) + owesForeign(loop.w) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1) + + loop.w.debt.observe(RemoteObservation{Commit: "ccc", By: ObservedByFetch}) + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "the first attempt is still out") + + report(out[0], ApplyDelivered) + loop.serviceAppliers() + next := requester.taken() + require.Len(t, next, 1) + assert.Equal(t, "ccc", next[0].Commit) + assert.Zero(t, next[0].PriorFailures) +} + +// TestApplier_ATargetThatNoLongerNamesAnApplierIsOwedNothing: a report for an applier nobody names +// any more is dropped with it. +func TestApplier_ATargetThatNoLongerNamesAnApplierIsOwedNothing(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true)) + owesForeign(loop.w) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1) + + updateTarget(t, loop, "alpha", func(g *configv1alpha3.GitTarget) { g.Spec.Applier = nil }) + loop.w.debt.observe(RemoteObservation{Commit: "ccc", By: ObservedByPush}) + loop.serviceAppliers() + report(out[0], ApplyFailed) + loop.serviceAppliers() + + assert.Empty(t, loop.appliers) + assert.Empty(t, requester.taken()) + assert.Nil(t, loop.applierTimer) +} + +// TestSiblings_ABusyWorkerOnTheSameBranchHoldsTheOther: two GitProviders naming one repository give +// one branch two workers, and a Kustomization applies the tip whichever of them wrote it. So one +// worker sends nothing while the other holds an open window, and is woken when it closes. +func TestSiblings_ABusyWorkerOnTheSameBranchHoldsTheOther(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) + loop.w.repo = RepoIdentity{URL: "https://github.com/acme/platform"} + loop.w.applyWake = make(chan struct{}, 1) + sibling := newMetricsTestWorker() + sibling.repo = RepoIdentity{URL: "git@github.com:acme/platform.git"} + sibling.applyWake = make(chan struct{}, 1) + other := newMetricsTestWorker() + other.repo = RepoIdentity{URL: "https://github.com/acme/elsewhere"} + siblings := newBranchSiblings() + for _, w := range []*BranchWorker{loop.w, sibling, other} { + siblings.add(w) + } + + other.publishWriteBusy(true) + sibling.publishWriteBusy(true) + owesForeign(loop.w) + loop.serviceAppliers() + assert.Empty(t, requester.taken(), "the sibling's open window is on this branch too") + assert.True(t, loop.quietSince.IsZero()) + + sibling.publishWriteBusy(false) + select { + case <-loop.w.applyWake: + default: + t.Fatal("a sibling going quiet must wake the worker it held back") + } + loop.serviceAppliers() + assert.Len(t, requester.taken(), 1, "a worker on another repository never held it") + + siblings.remove(sibling) + assert.False(t, loop.w.siblingWriteBusy()) +} + +func TestApplierRetryDelay_DoublesToACap(t *testing.T) { + for failures, want := range map[int]time.Duration{ + 1: 2 * time.Second, 2: 4 * time.Second, 4: 16 * time.Second, 8: 256 * time.Second, 9: 5 * time.Minute, + 50: 5 * time.Minute, + } { + assert.Equal(t, want, applierRetryDelay(failures, false), "after %d failures", failures) + } + assert.Equal(t, 15*time.Second, applierRetryDelay(1, true), "a rejected attempt waits for a configuration change") + assert.Equal(t, 5*time.Minute, applierRetryDelay(50, true)) +} diff --git a/internal/git/apply_handback_test.go b/internal/git/apply_handback_test.go deleted file mode 100644 index c3337209..00000000 --- a/internal/git/apply_handback_test.go +++ /dev/null @@ -1,260 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package git - -// What happens to an apply request after the worker decided it: a refusal owed while the branch -// was busy, and a request the requester could not deliver and handed back. Both wait, and both are -// checked again when they stop waiting. These are the worker's halves of the review findings 3 and -// 4 recorded in docs/design/apply-trigger-follow-ups.md §2.3 and §2.5. - -import ( - "testing" - "time" - - "github.com/stretchr/testify/assert" - "github.com/stretchr/testify/require" - "sigs.k8s.io/controller-runtime/pkg/client" - - configv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" - itypes "github.com/ConfigButler/gitops-reverser/internal/types" -) - -// refusalLoop is an event loop over refusalTouchWorker's GitTarget team-a/editing, asking for -// Reapply, whose requests are recorded. -func refusalLoop(t *testing.T) (*branchWorkerEventLoop, *recordingRequester) { - t.Helper() - w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{OnRefusal: configv1alpha3.RefusalActionReapply}) - requester := &recordingRequester{} - w.applyRequester = requester - loop := newBranchWorkerEventLoop(w, time.Minute) - t.Cleanup(loop.stopTimers) - return loop, requester -} - -// updateEditing changes team-a/editing in the worker's fake client. -func updateEditing(t *testing.T, w *BranchWorker, mutate func(*configv1alpha3.GitTarget)) { - t.Helper() - target := &configv1alpha3.GitTarget{} - require.NoError(t, w.Client.Get(w.ctx, client.ObjectKey{Name: "editing", Namespace: "team-a"}, target)) - mutate(target) - require.NoError(t, w.Client.Update(w.ctx, target)) -} - -// TestServiceRefusals_AnOwedRefusalAsksForConsentAgain: a refusal owed while the branch was busy -// waited, behind an open window or a publication outage, for as long as those lasted, and was -// sent to a target suspended or set back to Ignore in the meantime. -func TestServiceRefusals_AnOwedRefusalAsksForConsentAgain(t *testing.T) { - for name, withdraw := range map[string]func(*configv1alpha3.GitTarget){ - "suspended": func(g *configv1alpha3.GitTarget) { g.Spec.Suspend = true }, - "set to Ignore": func(g *configv1alpha3.GitTarget) { g.Spec.OnRefusal = configv1alpha3.RefusalActionIgnore }, - } { - t.Run(name, func(t *testing.T) { - loop, requester := refusalLoop(t) - loop.openWindow = &openWindow{} - loop.oweReapply(editingKey(), "a base-owned field", "observation-1") - loop.serviceAppliers() - - updateEditing(t, loop.w, withdraw) - loop.openWindow = nil - loop.serviceAppliers() - - assert.Zero(t, requester.refusals()) - assert.Empty(t, loop.reapplyOwed, "the obligation is spent, not kept for later") - }) - } -} - -// TestRefusalRecovered_CancelsWhatIsOwedForThatCollectionOnly: a refusal whose collection recovered -// while its request waited for a quiet branch has nothing left to revert. A sibling collection still -// refused keeps its own; the whole-target evaluation speaks for every collection. -func TestRefusalRecovered_CancelsWhatIsOwedForThatCollectionOnly(t *testing.T) { - configMaps := refusalKeyFor(editingRef(), "configmaps") - - loop, requester := refusalLoop(t) - loop.openWindow = &openWindow{} - loop.oweReapply(editingKey(), "a base-owned field", "observation-1") - loop.oweReapply(configMaps, "another base-owned field", "observation-2") - - loop.refusalRecovered(editingRef(), editingKey().collection) - assert.Equal(t, map[refusalKey]string{configMaps: "another base-owned field"}, loop.reapplyOwed) - - loop.openWindow = nil - loop.serviceAppliers() - assert.Equal(t, 1, requester.refusals(), "the configmaps refusal still stands") - - loop2, requester2 := refusalLoop(t) - loop2.openWindow = &openWindow{} - loop2.oweReapply(editingKey(), "a base-owned field", "observation-1") - loop2.oweReapply(configMaps, "another base-owned field", "observation-2") - loop2.refusalRecovered(editingRef(), itypes.CollectionKey{}) - loop2.openWindow = nil - loop2.serviceAppliers() - assert.Zero(t, requester2.refusals()) -} - -// TestHandBack_IsSentAgainOnlyWhenDueAndOnAQuietBranch: a handed-back request waits out its -// backoff, then waits for a quiet branch like any other, and carries its hand-back count so the -// requester does not report the same failure twice. -func TestHandBack_IsSentAgainOnlyWhenDueAndOnAQuietBranch(t *testing.T) { - loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true), applierTestTarget("bravo", true)) - owesForeign(loop.w) - loop.serviceAppliers() - require.Len(t, requester.taken(), 1, "the ForeignCommit, which then failed") - - alpha := itypes.NewResourceReference("alpha", "team-a") - loop.w.debt.handBack(ApplyRequest{ - Reason: configv1alpha3.ApplyReasonForeignCommit, Commit: "bbb", - Targets: []itypes.ResourceReference{alpha}, HandBacks: 1, - }, time.Now()) - - loop.serviceAppliers() - assert.Empty(t, requester.taken(), "not before its backoff") - require.NotNil(t, loop.applierTimer, "the backoff is armed") - - loop.w.debt.returnedDue = time.Now().Add(-time.Second) - loop.openWindow = &openWindow{} - loop.serviceAppliers() - assert.Empty(t, requester.taken(), "not while the branch is busy") - - loop.openWindow = nil - loop.serviceAppliers() - requests := requester.taken() - require.Len(t, requests, 1) - assert.Equal(t, ApplyRequest{ - Branch: BranchKey{RepoNamespace: "team-a", RepoName: "test-provider", Branch: "main"}, - Reason: configv1alpha3.ApplyReasonForeignCommit, Commit: "bbb", - Targets: []itypes.ResourceReference{alpha}, HandBacks: 1, - }, requests[0], "only the target handed back, at the tip, with its count") - assert.Nil(t, loop.applierTimer) - assert.Empty(t, loop.handBackCount, "the count went with the request") -} - -// TestHandBack_ARefusalIsOwedAgainOnlyWhileItStands: a Refusal handed back is owed again while the -// refusal has not recovered, and only with consent. One that recovered while its request was -// failing, and a handed-back CatchUp, are owed a catch-up, which waits for the quiet period. -func TestHandBack_ARefusalIsOwedAgainOnlyWhileItStands(t *testing.T) { - handBack := func(loop *branchWorkerEventLoop, reason configv1alpha3.ApplyReason) { - loop.w.debt.handBack(ApplyRequest{ - Reason: reason, Commit: "aaa", Targets: []itypes.ResourceReference{editingRef()}, HandBacks: 2, - }, time.Now()) - loop.w.debt.returnedDue = time.Time{} - } - - t.Run("standing", func(t *testing.T) { - loop, requester := refusalLoop(t) - loop.w.recordRefusalObservation(editingKey(), "observation-1") - handBack(loop, configv1alpha3.ApplyReasonRefusal) - - loop.serviceAppliers() - requests := requester.taken() - require.Len(t, requests, 1) - assert.Equal(t, configv1alpha3.ApplyReasonRefusal, requests[0].Reason) - assert.Equal(t, 2, requests[0].HandBacks) - }) - - t.Run("standing, but consent withdrawn", func(t *testing.T) { - loop, requester := refusalLoop(t) - loop.w.recordRefusalObservation(editingKey(), "observation-1") - updateEditing(t, loop.w, func(g *configv1alpha3.GitTarget) { g.Spec.Suspend = true }) - handBack(loop, configv1alpha3.ApplyReasonRefusal) - - loop.serviceAppliers() - assert.Empty(t, requester.taken()) - }) - - for _, tc := range []struct { - name string - reason configv1alpha3.ApplyReason - }{ - {"recovered", configv1alpha3.ApplyReasonRefusal}, - {"a catch-up", configv1alpha3.ApplyReasonCatchUp}, - } { - t.Run(tc.name, func(t *testing.T) { - loop, requester := refusalLoop(t) - handBack(loop, tc.reason) - - loop.serviceAppliers() - assert.Empty(t, requester.taken(), "a catch-up waits for the quiet period") - assert.Contains(t, loop.catchUpOwed, editingRef()) - - loop.quietSince = time.Now().Add(-time.Hour) - loop.serviceAppliers() - requests := requester.taken() - require.Len(t, requests, 1) - assert.Equal(t, configv1alpha3.ApplyReasonCatchUp, requests[0].Reason) - assert.Equal(t, 2, requests[0].HandBacks) - }) - } -} - -// TestHandBack_ATargetThatNoLongerNamesAnApplierIsOwedNothing. -func TestHandBack_ATargetThatNoLongerNamesAnApplierIsOwedNothing(t *testing.T) { - loop, requester := applyTriggerLoop(t, applierTestTarget("plain", false)) - loop.w.debt.handBack(ApplyRequest{ - Reason: configv1alpha3.ApplyReasonForeignCommit, Commit: "aaa", - Targets: []itypes.ResourceReference{itypes.NewResourceReference("plain", "team-a")}, HandBacks: 1, - }, time.Now()) - loop.w.debt.returnedDue = time.Time{} - - loop.serviceAppliers() - - assert.Empty(t, requester.taken()) - assert.Empty(t, loop.handBackCount) - assert.Nil(t, loop.applierTimer) -} - -// TestHandBackApply_RecordsTheDebtAndWakesTheLoop: the record is the debt; the wake only makes an -// idle loop look at it. -func TestHandBackApply_RecordsTheDebtAndWakesTheLoop(t *testing.T) { - w := newMetricsTestWorker() - req := ApplyRequest{ - Reason: configv1alpha3.ApplyReasonCatchUp, Commit: "aaa", - Targets: []itypes.ResourceReference{editingRef()}, HandBacks: 1, - } - w.handBackApply(req) - - require.Len(t, w.eventQueue, 1) - assert.True(t, (<-w.eventQueue).ApplyWake) - assert.Equal(t, map[itypes.ResourceReference]configv1alpha3.ApplyReason{ - editingRef(): configv1alpha3.ApplyReasonCatchUp, - }, w.debt.returned) - - for range cap(w.eventQueue) { - w.eventQueue <- WorkItem{} - } - req.Reason = configv1alpha3.ApplyReasonRefusal - w.handBackApply(req) - assert.Equal(t, configv1alpha3.ApplyReasonRefusal, w.debt.returned[editingRef()], - "a full queue drops the wake, never the request, and the stronger reason wins") -} - -func TestHandBackDelay_DoublesToACap(t *testing.T) { - for handBacks, want := range map[int]time.Duration{ - 0: 15 * time.Second, 1: 15 * time.Second, 2: 30 * time.Second, 3: time.Minute, - 5: 4 * time.Minute, 6: 5 * time.Minute, 50: 5 * time.Minute, - } { - assert.Equal(t, want, handBackDelay(handBacks), "after %d hand-backs", handBacks) - } -} - -func TestApplyDebt_HandBackWaitsForTheLatestBackoff(t *testing.T) { - var debt applyDebt - now := time.Now() - target := itypes.NewResourceReference("alpha", "team-a") - debt.handBack(ApplyRequest{Reason: configv1alpha3.ApplyReasonCatchUp, Targets: []itypes.ResourceReference{target}, - HandBacks: 3}, now) - debt.handBack(ApplyRequest{Reason: configv1alpha3.ApplyReasonCatchUp, Targets: []itypes.ResourceReference{target}, - HandBacks: 1}, now) - - returned, count, due := debt.takeReturned(now.Add(30 * time.Second)) - assert.Nil(t, returned) - assert.Zero(t, count) - assert.Equal(t, now.Add(time.Minute), due, "the longer backoff stands") - - returned, count, _ = debt.takeReturned(now.Add(time.Minute)) - assert.Len(t, returned, 1) - assert.Equal(t, 3, count) - returned, _, due = debt.takeReturned(now.Add(time.Hour)) - assert.Nil(t, returned) - assert.True(t, due.IsZero(), "taken once") -} diff --git a/internal/git/apply_request.go b/internal/git/apply_request.go index 69638ff0..07887817 100644 --- a/internal/git/apply_request.go +++ b/internal/git/apply_request.go @@ -6,29 +6,108 @@ import ( "slices" "time" + k8stypes "k8s.io/apimachinery/pkg/types" + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" itypes "github.com/ConfigButler/gitops-reverser/internal/types" ) -// ApplyRequest is one request, decided by a branch worker, that the appliers of some GitTargets on -// its branch apply a commit. The worker decides WHEN; delivering it is the ApplyRequester's job, off -// the worker's queue, so a slow applier never holds the branch. +// ApplierRef is one applier, as the GitTargets on a branch name it: a Flux Kustomization in a +// source cluster. Several GitTargets may name one applier, and it is asked once for all of them. +type ApplierRef struct { + // Cluster is the source cluster the applier lives in (GitTarget.SourceCluster()). + Cluster string + // Namespace and Name are the Kustomization's, the namespace defaulted to the GitTarget's own. + Namespace string + Name string +} + +func (a ApplierRef) String() string { return a.Cluster + "/" + a.Namespace + "/" + a.Name } + +// applierOf is the applier a GitTarget names. Call it only for a target with spec.applier. +func applierOf(target *v1alpha3.GitTarget) ApplierRef { + ref := target.Spec.Applier.KustomizationRef + namespace := ref.Namespace + if namespace == "" { + namespace = target.Namespace + } + return ApplierRef{Cluster: target.SourceCluster(), Namespace: namespace, Name: ref.Name} +} + +// ApplyRequest is one attempt, decided by a branch worker on a quiet branch, to have one applier +// apply a commit. The worker decides every attempt, retries included; the requester only carries it +// out, once, off the worker's goroutine, and reports how it went. So nothing is ever sent that the +// worker did not check on a quiet branch, against the GitTargets as they were at that moment. type ApplyRequest struct { + // ID identifies the attempt, so its report can be matched to it. + ID uint64 // Branch is the branch the request is about. Branch BranchKey - // Reason is why the appliers are asked. + // Applier is what is asked to apply. + Applier ApplierRef + // Reason is what the applier is asked: the strongest of the targets' reasons. Reason v1alpha3.ApplyReason // Commit is the commit to apply: the branch tip when the worker decided, which the worker has // pushed or proved by a fetch. Empty when the remote does not carry the branch. Commit string - // Targets are the GitTargets whose appliers are asked. Several may name one applier; the - // requester asks it once. - Targets []itypes.ResourceReference - // HandBacks is how many times a request for these targets has already been handed back - // undelivered (ApplyHandBack). Zero for a request the worker decided fresh. The worker's - // backoff grows with it, and the requester reports a failure only for a request at zero, so a - // standing failure is reported once rather than on every retry. - HandBacks int + // Targets are the GitTargets the request is for, each the incarnation the worker decided for + // and with its own reason. The requester asks nothing for a target that has changed since. + Targets []ApplyTarget + // PriorFailures is how many attempts for this applier failed since the last that did not. The + // requester reports a failure only for an attempt at zero, so a standing failure is reported + // once rather than on every retry. + PriorFailures int + // Report receives the attempt's outcome. It is called exactly once, and does not block. + Report func(ApplyResult) +} + +// ApplyTarget is one GitTarget an ApplyRequest is for. +type ApplyTarget struct { + Ref itypes.ResourceReference + UID k8stypes.UID + Reason v1alpha3.ApplyReason +} + +// TargetRefs lists the request's GitTargets. +func (r ApplyRequest) TargetRefs() []itypes.ResourceReference { + refs := make([]itypes.ResourceReference, 0, len(r.Targets)) + for _, target := range r.Targets { + refs = append(refs, target.Ref) + } + return refs +} + +// ApplyOutcome is how one attempt went. +type ApplyOutcome string + +const ( + // ApplyDelivered asked the applier to apply. + ApplyDelivered ApplyOutcome = "Delivered" + // ApplyCurrent asked nothing: a CatchUp found the applier already at the commit. + ApplyCurrent ApplyOutcome = "Current" + // ApplyFailed could not ask; a later attempt may. + ApplyFailed ApplyOutcome = "Failed" + // ApplyRejected could not ask, and asking again will not help until the applier's + // configuration changes: it applies another repository or branch, or no GitRepository. + ApplyRejected ApplyOutcome = "Rejected" + // ApplyDropped asked nothing, because none of the targets may still ask for the reason the + // request carried: each changed, is no longer admitted by its ClusterProvider, or withdrew its + // consent to a Refusal. The worker decides again from what the targets are now. + ApplyDropped ApplyOutcome = "Dropped" +) + +// ApplyResult is the report of one attempt. +type ApplyResult struct { + ID uint64 + Outcome ApplyOutcome + // Err is the cause of a Failed or Rejected attempt. + Err error +} + +// ApplyRequester carries out apply requests. RequestApply must not block: it is called on the +// branch worker's loop goroutine. +type ApplyRequester interface { + RequestApply(req ApplyRequest) } // StrongerApplyReason is the reason a target owed two requests is owed. A CatchUp asks for the @@ -45,68 +124,23 @@ func StrongerApplyReason(a, b v1alpha3.ApplyReason) v1alpha3.ApplyReason { return a } -// ApplyRequester delivers apply requests. RequestApply must not block: it is called on the branch -// worker's loop goroutine. -type ApplyRequester interface { - RequestApply(req ApplyRequest) -} - -// ApplyHandBack takes back a request the requester could not deliver, so the worker owes it again. -// -// The requester retries a failed request only briefly. A retry is the worker's decision replayed -// without the quiet check: the worker decided it on a quiet branch, and by the time a long retry -// lands, live edits may be waiting in an open window that the apply would revert. So the requester -// hands a request back once it has failed for about one quiet period, and the worker sends it -// again only when the branch is quiet once more. HandBackApply must not block. -type ApplyHandBack interface { - HandBackApply(req ApplyRequest) -} - -// handBackDelay is how long the worker waits before sending a request that has been handed back -// handBacks times: doubling from handBackInitialDelay, capped at handBackMaxDelay, so a permanent -// failure (a misconfigured Kustomization) is asked about every few minutes rather than in a loop. -func handBackDelay(handBacks int) time.Duration { - delay := handBackInitialDelay - for i := 1; i < handBacks && delay < handBackMaxDelay; i++ { +// applierRetryDelay is how long an applier waits after failures consecutive failed attempts: +// doubling from applierRetryInitial, capped at applierRetryMax. A rejected attempt waits at least +// applierRejectedFloor, because only a change to the applier's configuration can make it succeed. +func applierRetryDelay(failures int, rejected bool) time.Duration { + delay := applierRetryInitial + for i := 1; i < failures && delay < applierRetryMax; i++ { delay *= 2 } - return min(delay, handBackMaxDelay) + delay = min(delay, applierRetryMax) + if rejected { + delay = max(delay, applierRejectedFloor) + } + return delay } const ( - handBackInitialDelay = 15 * time.Second - handBackMaxDelay = 5 * time.Minute + applierRetryInitial = 2 * time.Second + applierRetryMax = 5 * time.Minute + applierRejectedFloor = 15 * time.Second ) - -// HandBackApply hands a request back to the worker for its branch. A branch with no worker has -// nothing to owe it to: a worker created later starts by owing every applier a catch-up. -func (m *WorkerManager) HandBackApply(req ApplyRequest) { - worker, found := m.GetWorkerForTarget(req.Branch.RepoName, req.Branch.RepoNamespace, req.Branch.Branch) - if !found { - return - } - worker.handBackApply(req) -} - -// handBackApply records a handed-back request in the debt and wakes the loop, which sends it -// again once it is due and the branch is quiet (serviceAppliers). The record is the debt, not the -// wake: a wake dropped on a full queue costs nothing, because the work already queued wakes the -// loop, and every wake ends by servicing the appliers. -func (w *BranchWorker) handBackApply(req ApplyRequest) { - w.debt.handBack(req, time.Now()) - w.Log.Info("An apply request could not be delivered; owing it again", - "branch", w.Branch, "reason", string(req.Reason), "gitTargets", len(req.Targets), - "handBacks", req.HandBacks, "retryIn", handBackDelay(req.HandBacks)) - w.inflightItems.Add(1) - w.pendingResyncsMu.Lock() - defer w.pendingResyncsMu.Unlock() - if w.stoppingLocked() { - w.inflightItems.Add(-1) - return - } - select { - case w.eventQueue <- WorkItem{ApplyWake: true}: - default: - w.inflightItems.Add(-1) - } -} diff --git a/internal/git/apply_siblings.go b/internal/git/apply_siblings.go new file mode 100644 index 00000000..55d02cbd --- /dev/null +++ b/internal/git/apply_siblings.go @@ -0,0 +1,114 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +import ( + "sync" + + "github.com/ConfigButler/gitops-reverser/internal/repourl" +) + +// A branch worker is keyed by its GitProvider, so two GitProviders naming one repository give one +// branch two workers. A Kustomization applies the tip of that branch whichever of them wrote it, so +// the branch is quiet only when BOTH are: an apply one worker asks for while the other holds an open +// window would revert that window's edits before they reach Git. branchSiblings is what lets each +// see the other. + +// branchSiblings groups the live workers by the repository and branch they write. +type branchSiblings struct { + mu sync.Mutex + byKey map[string]map[*BranchWorker]struct{} +} + +func newBranchSiblings() *branchSiblings { + return &branchSiblings{byKey: map[string]map[*BranchWorker]struct{}{}} +} + +// siblingKey is the repository, compared as repourl does, and the branch. A worker that knows no +// repository (the CLI, tests) has no siblings. +func siblingKey(w *BranchWorker) (string, bool) { + if w.repo.URL == "" { + return "", false + } + return repourl.Normalize(w.repo.URL) + "\x00" + w.Branch, true +} + +// add registers a worker that is about to start. A nil registry (a WorkerManager built by hand in a +// test) registers nothing. +func (s *branchSiblings) add(w *BranchWorker) { + key, ok := siblingKey(w) + if s == nil || !ok { + return + } + s.mu.Lock() + defer s.mu.Unlock() + if s.byKey[key] == nil { + s.byKey[key] = map[*BranchWorker]struct{}{} + } + s.byKey[key][w] = struct{}{} + w.siblings = s +} + +// remove forgets a worker that is stopping, and wakes the others: whatever it held busy is gone. +func (s *branchSiblings) remove(w *BranchWorker) { + key, ok := siblingKey(w) + if s == nil || !ok { + return + } + s.mu.Lock() + delete(s.byKey[key], w) + if len(s.byKey[key]) == 0 { + delete(s.byKey, key) + } + s.mu.Unlock() + s.wakeOthers(w) +} + +// othersBusy reports whether another worker on w's repository and branch is busy writing. +func (s *branchSiblings) othersBusy(w *BranchWorker) bool { + key, ok := siblingKey(w) + if !ok { + return false + } + s.mu.Lock() + defer s.mu.Unlock() + for sibling := range s.byKey[key] { + if sibling != w && sibling.busyWriting.Load() { + return true + } + } + return false +} + +// wakeOthers wakes every other worker on w's repository and branch, so one that held back an +// attempt because w was busy looks again. +func (s *branchSiblings) wakeOthers(w *BranchWorker) { + key, ok := siblingKey(w) + if !ok { + return + } + s.mu.Lock() + others := make([]*BranchWorker, 0, len(s.byKey[key])) + for sibling := range s.byKey[key] { + if sibling != w { + others = append(others, sibling) + } + } + s.mu.Unlock() + for _, sibling := range others { + sibling.wakeApplier() + } +} + +// publishWriteBusy records whether this worker holds live writes not yet on the remote, and wakes +// its siblings when it stops: they may have held an attempt back for it. +func (w *BranchWorker) publishWriteBusy(busy bool) { + if was := w.busyWriting.Swap(busy); was && !busy && w.siblings != nil { + w.siblings.wakeOthers(w) + } +} + +// siblingWriteBusy reports whether another worker on this repository and branch is busy writing. +func (w *BranchWorker) siblingWriteBusy() bool { + return w.siblings != nil && w.siblings.othersBusy(w) +} diff --git a/internal/git/apply_trigger.go b/internal/git/apply_trigger.go index a026096a..bc04175f 100644 --- a/internal/git/apply_trigger.go +++ b/internal/git/apply_trigger.go @@ -9,6 +9,7 @@ import ( "sync" "time" + k8stypes "k8s.io/apimachinery/pkg/types" "sigs.k8s.io/controller-runtime/pkg/client" "github.com/ConfigButler/gitops-reverser/api/v1alpha3" @@ -16,13 +17,12 @@ import ( ) // This file is the branch worker's half of the apply trigger: deciding WHEN the appliers of the -// GitTargets on this branch may apply. See docs/design/push-notification-and-reconcile-trigger.md -// part 3. +// GitTargets on this branch apply. See docs/design/apply-trigger.md. // -// An apply is safe only when nothing live is missing from the commit being applied, and only this -// worker can see the edits that are live and not yet pushed: they are its open commit window and its -// pending writes. So a request goes out only on a QUIET branch, with neither, and the applier's own -// triggers are switched off by the installer. Three things owe a request: +// An apply is safe only when nothing live is missing from the commit being applied, and only the +// workers on this branch can see the edits that are live and not yet pushed: their open commit +// windows and pending writes. So an attempt goes out only on a QUIET branch, with neither on this +// worker or on any other worker writing the same repository and branch. Three things owe one: // // - ForeignCommit: an observation of the remote found the branch somewhere this worker did not // push it. Every applier on the branch is asked as soon as the branch is quiet, and an open @@ -31,12 +31,21 @@ import ( // - Refusal: a refused edit on a target with spec.onRefusal: Reapply (refusal_reapply.go). // - CatchUp: this worker's own pushes. Flux applies on its own timers and on a restart from // whatever artifact it holds, and an artifact behind these pushes would revert edits already in -// Git. Asked only once the branch has stayed quiet for the target's spec.applier.quietPeriod, so -// the apply does not land in the middle of an editing session. - -// applyDebt is what the branch's appliers are owed because of what was OBSERVED about the remote. It -// is fed from publishObservation, the one point every observation passes through, which can run -// before the loop starts (the first clone), hence the lock. +// Git. Asked only once the branch has stayed quiet for the applier's quiet period. +// +// The three relations are kept apart on purpose. The BRANCH is the unit of quietness, because a +// Kustomization applies the tip whichever GitTarget's folder an edit is in. The APPLIER is the unit +// of obligation: one attempt in flight at a time, its own retry clock, and one attempt for all the +// GitTargets that name it. The GITTARGET, by uid, is the unit of configuration and consent. +// +// The worker owns every obligation until it is delivered. The requester carries out one attempt and +// reports back; it never retries, so a retry is decided here like any first attempt, on a quiet +// branch, against what the targets are now. + +// applyDebt is what the branch's appliers are owed because of what was OBSERVED about the remote, +// and the reports of attempts that came back. It is written from other goroutines (the first clone +// publishes an observation before the loop starts; the requester reports from its own), hence the +// lock. type applyDebt struct { mu sync.Mutex // known is whether anything has been observed yet, and tip is the last commit observed. Empty @@ -46,13 +55,8 @@ type applyDebt struct { // foreign and catchUp are owed until the loop takes them on a quiet branch. foreign bool catchUp bool - // returned holds the requests the requester handed back undelivered, per GitTarget with the - // reason it was owed, until returnedDue. returnedCount is the most hand-backs any of them has - // been through. They live here rather than on the loop because the requester hands back from - // its own goroutine. - returned map[itypes.ResourceReference]v1alpha3.ApplyReason - returnedCount int - returnedDue time.Time + // results are the reports of finished attempts, until the loop reads them. + results []ApplyResult } // observe classifies one observation. The first one is a baseline, not a move: nothing says what @@ -98,93 +102,101 @@ func (d *applyDebt) restore(foreign, catchUp bool) { d.catchUp = d.catchUp || catchUp } -// handBack records a request the requester could not deliver, due after its backoff. A target -// handed back twice keeps the stronger reason (StrongerApplyReason). -func (d *applyDebt) handBack(req ApplyRequest, now time.Time) { - d.mu.Lock() - defer d.mu.Unlock() - d.mergeReturnedLocked(req.Reason, req.Targets, req.HandBacks) - if due := now.Add(handBackDelay(req.HandBacks)); due.After(d.returnedDue) { - d.returnedDue = due - } -} - -func (d *applyDebt) mergeReturnedLocked( - reason v1alpha3.ApplyReason, targets []itypes.ResourceReference, handBacks int, -) { - if d.returned == nil { - d.returned = map[itypes.ResourceReference]v1alpha3.ApplyReason{} - } - for _, target := range targets { - d.returned[target] = StrongerApplyReason(d.returned[target], reason) - } - d.returnedCount = max(d.returnedCount, handBacks) -} - -// takeReturned returns the handed-back requests and clears them, once they are due. Before then it -// returns none, and when they come due. -func (d *applyDebt) takeReturned(now time.Time) (map[itypes.ResourceReference]v1alpha3.ApplyReason, int, time.Time) { +// foreignOwed reports whether a ForeignCommit is owed, without taking it. +func (d *applyDebt) foreignOwed() bool { d.mu.Lock() defer d.mu.Unlock() - if len(d.returned) == 0 { - return nil, 0, time.Time{} - } - if now.Before(d.returnedDue) { - return nil, 0, d.returnedDue - } - returned, count := d.returned, d.returnedCount - d.returned, d.returnedCount, d.returnedDue = nil, 0, time.Time{} - return returned, count, time.Time{} + return d.foreign } -// restoreReturned puts back what takeReturned returned, due at once, for a caller that could not -// act on it. -func (d *applyDebt) restoreReturned(returned map[itypes.ResourceReference]v1alpha3.ApplyReason, count int) { +func (d *applyDebt) addResult(res ApplyResult) { d.mu.Lock() defer d.mu.Unlock() - for target, reason := range returned { - d.mergeReturnedLocked(reason, []itypes.ResourceReference{target}, count) - } + d.results = append(d.results, res) } -// foreignOwed reports whether a ForeignCommit is owed, without taking it. -func (d *applyDebt) foreignOwed() bool { +func (d *applyDebt) takeResults() []ApplyResult { d.mu.Lock() defer d.mu.Unlock() - return d.foreign + results := d.results + d.results = nil + return results } // applierRetryInterval is how soon a wake retries after it could not list the branch's appliers. const applierRetryInterval = 10 * time.Second -// branchApplier is one GitTarget on this branch that names an applier. -type branchApplier struct { - target itypes.ResourceReference +// branchTarget is one GitTarget on this branch that names an applier, as listed on this pass. +type branchTarget struct { + ref itypes.ResourceReference + uid k8stypes.UID + applier ApplierRef quietPeriod time.Duration + // consents is whether it asks for a Refusal request now (ConsentsToReapply). + consents bool } // appliersOnBranch lists the GitTargets on this worker's (provider, branch) that name an applier. A // GitTarget names its GitProvider in its own namespace, so the provider's namespace is the only one // to look in. The client is cached, so this costs no round trip. -func (w *BranchWorker) appliersOnBranch(ctx context.Context) ([]branchApplier, error) { +func (w *BranchWorker) appliersOnBranch(ctx context.Context) ([]branchTarget, error) { var targets v1alpha3.GitTargetList if err := w.Client.List(ctx, &targets, client.InNamespace(w.GitProviderNamespace)); err != nil { return nil, fmt.Errorf("list GitTargets: %w", err) } - appliers := make([]branchApplier, 0, len(targets.Items)) + listed := make([]branchTarget, 0, len(targets.Items)) for i := range targets.Items { target := &targets.Items[i] if !target.DeletionTimestamp.IsZero() || target.Spec.Applier == nil || target.Spec.GitProviderRef.Name != w.GitProviderRef || target.Spec.Branch != w.Branch { continue } - appliers = append(appliers, branchApplier{ - target: itypes.NewResourceReference(target.Name, target.Namespace), + listed = append(listed, branchTarget{ + ref: itypes.NewResourceReference(target.Name, target.Namespace), + uid: target.UID, + applier: applierOf(target), quietPeriod: target.Spec.Applier.EffectiveQuietPeriod(), + consents: ConsentsToReapply(target), }) } - sort.Slice(appliers, func(i, j int) bool { return appliers[i].target.String() < appliers[j].target.String() }) - return appliers, nil + sort.Slice(listed, func(i, j int) bool { return listed[i].ref.String() < listed[j].ref.String() }) + return listed, nil +} + +// applierState is what one applier on this branch is owed, and how its attempts are going. +type applierState struct { + ref ApplierRef + // targets are the GitTargets that name it, as listed on the last pass. + targets []branchTarget + // foreign and catchUp are owed until an attempt carries them. + foreign bool + catchUp bool + // inflight is the attempt the requester has not reported yet. While it is out, nothing else is + // sent to this applier: what is owed meanwhile waits for the report. + inflight *inflightAttempt + // failures counts attempts that did not deliver since the last that did, and retryAt is when the + // next attempt may go. + failures int + retryAt time.Time +} + +// inflightAttempt is what an attempt carries away from its applier's obligations. A failed attempt +// gives it back (restoreAttempt); a delivered one keeps it. +type inflightAttempt struct { + id uint64 + foreign bool + catchUp bool + refusals map[refusalKey]owedRefusal +} + +// quietPeriod is how long the branch must be quiet before a CatchUp: the longest among the +// GitTargets naming the applier, because its apply reaches every one of their folders. +func (s *applierState) quietPeriod() time.Duration { + var longest time.Duration + for _, target := range s.targets { + longest = max(longest, target.quietPeriod) + } + return longest } // writeBusy reports whether the loop holds anything live that is not yet on the remote, or is in a @@ -211,249 +223,295 @@ func (l *branchWorkerEventLoop) closeWindowForForeignCommit() { l.finalizeOpenWindowWithReason(windowFinalizeReasonForeignCommit) } -// serviceAppliers sends whatever apply requests are owed, if the branch is quiet. It runs at the end -// of every wake, after the wake's writes have been decided and committed, so "quiet" is what the -// wake left behind. +// serviceAppliers settles the reports that came back, and sends whatever attempts are due if the +// branch is quiet. It runs at the end of every wake, after the wake's writes have been decided and +// committed, so "quiet" is what the wake left behind. func (l *branchWorkerEventLoop) serviceAppliers() { w := l.w if w.applyRequester == nil { return } - if l.writeBusy() { - l.quietSince = time.Time{} - l.stopApplierTimer() - return - } now := time.Now() - if l.quietSince.IsZero() { - l.quietSince = now - } - if len(w.eventQueue) > 0 { - // More work is already queued and the next wake will look again. The branch is still quiet - // until that work opens a window, so quietSince is kept. + l.settleApplyResults(now) + if !l.branchQuiet(now) || len(w.eventQueue) > 0 { + // Queued work: the next wake looks again. The branch is still quiet until that work opens + // a window, so quietSince is kept. return } tip, foreign, catchUp := w.debt.take() - returned, returnedCount, returnedDue := w.debt.takeReturned(now) - if !foreign && !catchUp && len(returned) == 0 && len(l.reapplyOwed) == 0 && len(l.catchUpOwed) == 0 { - l.armApplierTimer(now, returnedDue) + if !foreign && !catchUp && len(l.reapplyOwed) == 0 && !l.appliersOwe() { + l.stopApplierTimer() return } - declared, ok := l.declaredAppliers() - if !ok { + targets, err := w.appliersOnBranch(w.ctx) + if err != nil { w.debt.restore(foreign, catchUp) - w.debt.restoreReturned(returned, returnedCount) - return - } - if catchUp { - l.oweCatchUp(declared) - } - l.sendOwed(tip, foreign, l.oweReturned(returned, returnedCount, declared), declared, now, returnedDue) -} - -// sendOwed sends what serviceAppliers found owed on a quiet branch, and arms the timer for what is -// not due yet. -func (l *branchWorkerEventLoop) sendOwed( - tip string, - foreign bool, - returnedForeign []itypes.ResourceReference, - declared map[itypes.ResourceReference]time.Duration, - now, returnedDue time.Time, -) { - if foreign && len(declared) > 0 { - // The tip is the foreign commit with every pending write replayed onto it: the branch is - // quiet, so anything that was pending has been published on top. Every applier on the - // branch is asked, which also answers whatever else they were owed. - l.requestApply(v1alpha3.ApplyReasonForeignCommit, tip, sortedTargetKeys(declared)) - clear(l.reapplyOwed) - clear(l.catchUpOwed) - l.armApplierTimer(now, returnedDue) + w.Log.Error(err, "Cannot list the appliers on this branch; retrying", "branch", w.Branch) + l.resetApplierTimer(applierRetryInterval) return } - if len(returnedForeign) > 0 { - for _, target := range returnedForeign { - delete(l.catchUpOwed, target) - } - l.requestApply(v1alpha3.ApplyReasonForeignCommit, tip, returnedForeign) + l.syncAppliers(targets) + for _, state := range l.appliers { + state.foreign = state.foreign || foreign + state.catchUp = state.catchUp || catchUp } - l.serviceRefusals(tip, declared) - l.armApplierTimer(now, l.serviceCatchUp(tip, declared, now), returnedDue) + l.armApplierTimer(now, l.sendDue(tip, now)) } -// declaredAppliers maps every GitTarget on this branch that names an applier to its quiet period. -// On a failure to list them a retry is armed, and the caller puts back what it took. -func (l *branchWorkerEventLoop) declaredAppliers() (map[itypes.ResourceReference]time.Duration, bool) { - appliers, err := l.w.appliersOnBranch(l.w.ctx) - if err != nil { - l.w.Log.Error(err, "Cannot list the appliers on this branch; retrying", "branch", l.w.Branch) - l.resetApplierTimer(applierRetryInterval) - return nil, false +// branchQuiet reports whether the branch holds nothing live that is not yet on the remote, on this +// worker or on any other writing the same repository and branch, and keeps quietSince: when it last +// became quiet, zero while it is busy. +func (l *branchWorkerEventLoop) branchQuiet(now time.Time) bool { + busy := l.writeBusy() + l.w.publishWriteBusy(busy) + if busy || l.w.siblingWriteBusy() { + l.quietSince = time.Time{} + l.stopApplierTimer() + return false } - declared := make(map[itypes.ResourceReference]time.Duration, len(appliers)) - for _, applier := range appliers { - declared[applier.target] = applier.quietPeriod + if l.quietSince.IsZero() { + l.quietSince = now } - return declared, true + return true } -// oweCatchUp records that every declared applier is owed a catch-up with this worker's pushes. -func (l *branchWorkerEventLoop) oweCatchUp(declared map[itypes.ResourceReference]time.Duration) { - if l.catchUpOwed == nil { - l.catchUpOwed = map[itypes.ResourceReference]time.Duration{} - } - for target, quietPeriod := range declared { - l.catchUpOwed[target] = quietPeriod +// appliersOwe reports whether any applier is owed something or has an attempt out. +func (l *branchWorkerEventLoop) appliersOwe() bool { + for _, state := range l.appliers { + if state.foreign || state.catchUp || state.inflight != nil { + return true + } } + return false } -// oweReturned owes again what the requester handed back, now that it is due, and returns the -// targets owed a ForeignCommit, which the caller sends at once. -// -// A Refusal is owed again only while the refusal still stands: one that recovered while its -// request was failing has nothing left to revert, and its target is owed a catch-up instead, as is -// a handed-back CatchUp. Both go through the ordinary paths, so a Refusal is sent only with consent -// and a catch-up only after the quiet period. -func (l *branchWorkerEventLoop) oweReturned( - returned map[itypes.ResourceReference]v1alpha3.ApplyReason, - count int, - declared map[itypes.ResourceReference]time.Duration, -) []itypes.ResourceReference { - var foreign []itypes.ResourceReference - for _, target := range sortedTargetKeys(returned) { - quietPeriod, ok := declared[target] - if !ok { - continue // its applier was removed since; nothing is owed to it any more +// syncAppliers makes the applier states match the GitTargets listed now. An applier no GitTarget +// names any more is forgotten, with whatever it was owed: nobody asks for it. A refusal owed for a +// GitTarget that is gone, or is now another incarnation, is dropped the same way. +func (l *branchWorkerEventLoop) syncAppliers(targets []branchTarget) { + if l.appliers == nil { + l.appliers = map[ApplierRef]*applierState{} + } + named := map[ApplierRef][]branchTarget{} + uids := map[itypes.ResourceReference]k8stypes.UID{} + for _, target := range targets { + named[target.applier] = append(named[target.applier], target) + uids[target.ref] = target.uid + } + for ref := range l.appliers { + if _, ok := named[ref]; !ok { + delete(l.appliers, ref) } - if l.handBackCount == nil { - l.handBackCount = map[itypes.ResourceReference]int{} + } + for ref, named := range named { + state := l.appliers[ref] + if state == nil { + state = &applierState{ref: ref} + l.appliers[ref] = state } - l.handBackCount[target] = max(l.handBackCount[target], count) - switch reason := returned[target]; { - case reason == v1alpha3.ApplyReasonForeignCommit: - foreign = append(foreign, target) - case reason == v1alpha3.ApplyReasonRefusal && l.oweStandingRefusals(target): - default: - if l.catchUpOwed == nil { - l.catchUpOwed = map[itypes.ResourceReference]time.Duration{} - } - l.catchUpOwed[target] = quietPeriod + state.targets = named + } + for key, owed := range l.reapplyOwed { + if uids[key.target] != owed.uid { + delete(l.reapplyOwed, key) } } - return foreign } -// oweStandingRefusals owes target's applier a Refusal request again for every refusal of the -// target that has not recovered, and reports whether there was any. -func (l *branchWorkerEventLoop) oweStandingRefusals(target itypes.ResourceReference) bool { - standing := l.w.standingRefusals(target) - if l.reapplyOwed == nil && len(standing) > 0 { - l.reapplyOwed = map[refusalKey]string{} +// sendDue sends an attempt to every applier that is owed one, is not waiting on one, and whose +// retry and quiet periods have run out. It returns the earliest time one still waiting comes due. +func (l *branchWorkerEventLoop) sendDue(tip string, now time.Time) time.Time { + var earliest time.Time + wait := func(at time.Time) { + if earliest.IsZero() || at.Before(earliest) { + earliest = at + } } - for _, key := range standing { - if _, owed := l.reapplyOwed[key]; !owed { - l.reapplyOwed[key] = "a Refusal request that could not be delivered" + for _, ref := range l.sortedAppliers() { + state := l.appliers[ref] + if state.inflight != nil { + continue } + refusals := l.refusalsOwed(state) + if !state.foreign && len(refusals) == 0 && !state.catchUp { + continue + } + due := state.retryAt + if !state.foreign && len(refusals) == 0 { + due = latest(due, l.quietSince.Add(state.quietPeriod())) + } + if now.Before(due) { + wait(due) + continue + } + l.sendAttempt(state, tip, refusals) } - return len(standing) > 0 + return earliest } -// serviceRefusals sends each owed Refusal request to its target's applier alone, once per target -// however many of its collections are refused. -// -// Consent is read again here. A refusal owed while the branch was busy can wait behind an open -// window or a publication outage for as long as those last, and a target suspended or set back to -// Ignore in the meantime has withdrawn it. -func (l *branchWorkerEventLoop) serviceRefusals( - tip string, declared map[itypes.ResourceReference]time.Duration, -) { - targets := map[itypes.ResourceReference]struct{}{} - for key := range l.reapplyOwed { - targets[key.target] = struct{}{} - } - clear(l.reapplyOwed) - for _, target := range sortedTargetKeys(targets) { - if _, ok := declared[target]; !ok { - continue // its applier was removed since the refusal; consent went with it +// refusalsOwed is the refusals owed for the GitTargets naming this applier that still consent. A +// refusal of a target that no longer consents is dropped: its consent was the obligation. +func (l *branchWorkerEventLoop) refusalsOwed(state *applierState) map[refusalKey]owedRefusal { + consents := map[itypes.ResourceReference]bool{} + for _, target := range state.targets { + consents[target.ref] = target.consents + } + owed := map[refusalKey]owedRefusal{} + for key, refusal := range l.reapplyOwed { + consenting, named := consents[key.target] + if !named { + continue } - if !l.w.refusalConsent(l.w.ctx, target) { + if !consenting { + delete(l.reapplyOwed, key) continue } - // A reapply applies the tip, which is everything a catch-up would have asked for. - delete(l.catchUpOwed, target) - l.requestApply(v1alpha3.ApplyReasonRefusal, tip, []itypes.ResourceReference{target}) + owed[key] = refusal } + return owed } -// serviceCatchUp asks every applier whose quiet period has run out to catch up, and returns when -// the earliest one still waiting comes due (zero when none waits). -func (l *branchWorkerEventLoop) serviceCatchUp( - tip string, declared map[itypes.ResourceReference]time.Duration, now time.Time, -) time.Time { - var due []itypes.ResourceReference - var earliest time.Time - for target, quietPeriod := range l.catchUpOwed { - if _, ok := declared[target]; !ok { - delete(l.catchUpOwed, target) +// sendAttempt hands one attempt for an applier to the requester and moves what it carries out of +// the applier's obligations, into the attempt. Each target is asked for its own reason: every one +// for a ForeignCommit; otherwise a Refusal for a target with a refusal owed, and a CatchUp for the +// rest when one is owed. Any delivered attempt applies the tip, so it answers an owed CatchUp too. +func (l *branchWorkerEventLoop) sendAttempt(state *applierState, tip string, refusals map[refusalKey]owedRefusal) { + refused := map[itypes.ResourceReference]bool{} + for key := range refusals { + refused[key.target] = true + } + var targets []ApplyTarget + var reason v1alpha3.ApplyReason + for _, target := range state.targets { + var own v1alpha3.ApplyReason + switch { + case state.foreign: + own = v1alpha3.ApplyReasonForeignCommit + case refused[target.ref]: + own = v1alpha3.ApplyReasonRefusal + case state.catchUp: + own = v1alpha3.ApplyReasonCatchUp + default: continue } - at := l.quietSince.Add(quietPeriod) - if !now.Before(at) { - due = append(due, target) - delete(l.catchUpOwed, target) + targets = append(targets, ApplyTarget{Ref: target.ref, UID: target.uid, Reason: own}) + reason = StrongerApplyReason(reason, own) + } + + l.nextApplyID++ + attempt := &inflightAttempt{ + id: l.nextApplyID, foreign: state.foreign, catchUp: state.catchUp, refusals: refusals, + } + state.foreign, state.catchUp, state.inflight = false, false, attempt + for key := range refusals { + delete(l.reapplyOwed, key) + } + + w := l.w + w.Log.Info("Asking the applier to apply", + "branch", w.Branch, "applier", state.ref.String(), "reason", string(reason), "commit", tip, + "gitTargets", len(targets), "priorFailures", state.failures) + w.applyRequester.RequestApply(ApplyRequest{ + ID: attempt.id, + Branch: BranchKey{RepoNamespace: w.GitProviderNamespace, RepoName: w.GitProviderRef, Branch: w.Branch}, + Applier: state.ref, + Reason: reason, + Commit: tip, + Targets: targets, + PriorFailures: state.failures, + Report: w.reportApplyResult, + }) +} + +// settleApplyResults reads the reports that came back. A delivered attempt is done. One that did +// not deliver gives back what it carried and waits out its applier's backoff, so its retry is +// decided again on a quiet branch at whatever the tip is then, never replayed blind. A report for an +// applier no GitTarget names any more is dropped with it. +func (l *branchWorkerEventLoop) settleApplyResults(now time.Time) { + for _, res := range l.w.debt.takeResults() { + state := l.applierAwaiting(res.ID) + if state == nil { continue } - if earliest.IsZero() || at.Before(earliest) { - earliest = at + attempt := state.inflight + state.inflight = nil + switch res.Outcome { + case ApplyDelivered, ApplyCurrent: + state.failures, state.retryAt = 0, time.Time{} + continue + case ApplyFailed, ApplyRejected, ApplyDropped: } + state.failures++ + state.retryAt = now.Add(applierRetryDelay(state.failures, res.Outcome == ApplyRejected)) + l.restoreAttempt(state, attempt) + l.w.Log.Info("An apply attempt did not deliver; deciding it again on a quiet branch", + "branch", l.w.Branch, "applier", state.ref.String(), "outcome", string(res.Outcome), + "failures", state.failures, "retryIn", state.retryAt.Sub(now), "error", errString(res.Err)) } - if len(due) > 0 { - sortReferences(due) - l.requestApply(v1alpha3.ApplyReasonCatchUp, tip, due) +} + +// restoreAttempt gives an applier back what an attempt that did not deliver carried. A refusal comes +// back only while it still stands: one that recovered while the attempt was out has nothing left to +// revert. +func (l *branchWorkerEventLoop) restoreAttempt(state *applierState, attempt *inflightAttempt) { + state.foreign = state.foreign || attempt.foreign + state.catchUp = state.catchUp || attempt.catchUp + for key, refusal := range attempt.refusals { + if !l.w.refusalStands(key) { + continue + } + if l.reapplyOwed == nil { + l.reapplyOwed = map[refusalKey]owedRefusal{} + } + if _, owed := l.reapplyOwed[key]; !owed { + l.reapplyOwed[key] = refusal + } } - return earliest } -// armApplierTimer points the applier timer at the earliest of the given deadlines that is set, or -// stops it when none is. -func (l *branchWorkerEventLoop) armApplierTimer(now time.Time, deadlines ...time.Time) { - var earliest time.Time - for _, at := range deadlines { - if !at.IsZero() && (earliest.IsZero() || at.Before(earliest)) { - earliest = at +// applierAwaiting is the applier whose attempt has this id out, or nil. +func (l *branchWorkerEventLoop) applierAwaiting(id uint64) *applierState { + for _, state := range l.appliers { + if state.inflight != nil && state.inflight.id == id { + return state } } - if earliest.IsZero() { - l.stopApplierTimer() - return + return nil +} + +// reportApplyResult is ApplyRequest.Report: it records the report and wakes the loop to settle it. +// It is called from the requester's goroutine. +func (w *BranchWorker) reportApplyResult(res ApplyResult) { + w.debt.addResult(res) + w.wakeApplier() +} + +// wakeApplier wakes the loop without queueing a work item, so the wake neither counts as queued +// work nor makes the branch look busy. A wake already pending is enough. +func (w *BranchWorker) wakeApplier() { + select { + case w.applyWake <- struct{}{}: + default: } - l.resetApplierTimer(earliest.Sub(now)) } -// requestApply hands one request to the requester, which delivers it off this goroutine. It -// carries, and consumes, the hand-back count of its targets. -func (l *branchWorkerEventLoop) requestApply( - reason v1alpha3.ApplyReason, commit string, targets []itypes.ResourceReference, -) { - handBacks := 0 - for _, target := range targets { - handBacks = max(handBacks, l.handBackCount[target]) - delete(l.handBackCount, target) - } - l.w.Log.Info("Asking the appliers to apply", - "branch", l.w.Branch, "reason", string(reason), "commit", commit, "gitTargets", len(targets), - "handBacks", handBacks) - l.w.applyRequester.RequestApply(ApplyRequest{ - Branch: BranchKey{ - RepoNamespace: l.w.GitProviderNamespace, - RepoName: l.w.GitProviderRef, - Branch: l.w.Branch, - }, - Reason: reason, - Commit: commit, - Targets: targets, - HandBacks: handBacks, - }) +func (l *branchWorkerEventLoop) sortedAppliers() []ApplierRef { + refs := make([]ApplierRef, 0, len(l.appliers)) + for ref := range l.appliers { + refs = append(refs, ref) + } + sort.Slice(refs, func(i, j int) bool { return refs[i].String() < refs[j].String() }) + return refs +} + +// armApplierTimer points the applier timer at a deadline, or stops it when there is none. +func (l *branchWorkerEventLoop) armApplierTimer(now, at time.Time) { + if at.IsZero() { + l.stopApplierTimer() + return + } + l.resetApplierTimer(at.Sub(now)) } func (l *branchWorkerEventLoop) applierTimerC() <-chan time.Time { @@ -475,15 +533,16 @@ func (l *branchWorkerEventLoop) stopApplierTimer() { } } -func sortedTargetKeys[V any](set map[itypes.ResourceReference]V) []itypes.ResourceReference { - targets := make([]itypes.ResourceReference, 0, len(set)) - for target := range set { - targets = append(targets, target) +func latest(a, b time.Time) time.Time { + if a.After(b) { + return a } - sortReferences(targets) - return targets + return b } -func sortReferences(refs []itypes.ResourceReference) { - sort.Slice(refs, func(i, j int) bool { return refs[i].String() < refs[j].String() }) +func errString(err error) string { + if err == nil { + return "" + } + return err.Error() } diff --git a/internal/git/apply_trigger_test.go b/internal/git/apply_trigger_test.go index a29cd653..b74bce70 100644 --- a/internal/git/apply_trigger_test.go +++ b/internal/git/apply_trigger_test.go @@ -2,9 +2,8 @@ package git -// The branch worker's half of the apply trigger, from -// docs/design/push-notification-and-reconcile-trigger.md part 3: what the appliers are owed, and the -// rule that nothing is asked while anything live is unpublished. +// The branch worker's half of the apply trigger, from docs/design/apply-trigger.md: what the appliers +// are owed, and the rule that nothing is asked while anything live is unpublished. import ( "context" @@ -17,6 +16,7 @@ import ( "github.com/stretchr/testify/require" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + k8stypes "k8s.io/apimachinery/pkg/types" clientgoscheme "k8s.io/client-go/kubernetes/scheme" "sigs.k8s.io/controller-runtime/pkg/client" "sigs.k8s.io/controller-runtime/pkg/client/fake" @@ -76,10 +76,15 @@ func TestApplyDebt_RestorePutsBackWhatWasTaken(t *testing.T) { assert.True(t, debt.foreignOwed()) } +// applierRef is the applier testApplier(name) names for a GitTarget in team-a. +func applierRef(name string) ApplierRef { + return ApplierRef{Cluster: configv1alpha3.DefaultClusterProviderName, Namespace: "team-a", Name: name} +} + // applierTestTarget is a GitTarget on the trigger worker's provider and branch unless overridden. func applierTestTarget(name string, applier bool, mutate ...func(*configv1alpha3.GitTarget)) *configv1alpha3.GitTarget { target := &configv1alpha3.GitTarget{ - ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "team-a"}, + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "team-a", UID: k8stypes.UID("uid-" + name)}, Spec: configv1alpha3.GitTargetSpec{ GitProviderRef: meta.LocalObjectReference{Name: "test-provider"}, Branch: "main", @@ -137,14 +142,17 @@ func TestServiceAppliers_ForeignCommitAsksEveryApplierOnTheBranch(t *testing.T) loop.serviceAppliers() requests := requester.taken() - require.Len(t, requests, 1) - assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, requests[0].Reason) - assert.Equal(t, "bbb", requests[0].Commit) - assert.Equal(t, BranchKey{RepoNamespace: "team-a", RepoName: "test-provider", Branch: "main"}, requests[0].Branch) - assert.Equal(t, []itypes.ResourceReference{ - itypes.NewResourceReference("alpha", "team-a"), - itypes.NewResourceReference("bravo", "team-a"), - }, requests[0].Targets, "a suspended target stops writing, not deployments") + require.Len(t, requests, 2, "one attempt per applier") + for i, name := range []string{"alpha", "bravo"} { + assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, requests[i].Reason) + assert.Equal(t, "bbb", requests[i].Commit) + assert.Equal(t, BranchKey{RepoNamespace: "team-a", RepoName: "test-provider", Branch: "main"}, + requests[i].Branch) + assert.Equal(t, ApplierRef{Cluster: configv1alpha3.DefaultClusterProviderName, Namespace: "team-a", Name: name}, + requests[i].Applier) + assert.Equal(t, []itypes.ResourceReference{itypes.NewResourceReference(name, "team-a")}, + requests[i].TargetRefs(), "a suspended target stops writing, not deployments") + } } func TestServiceAppliers_NothingIsSentWhileAnythingLiveIsUnpublished(t *testing.T) { @@ -188,7 +196,7 @@ func TestServiceAppliers_CatchUpWaitsForTheQuietPeriod(t *testing.T) { loop.serviceAppliers() assert.Empty(t, requester.taken(), "the branch has only just become quiet") require.NotNil(t, loop.applierTimer, "the earliest catch-up is armed") - require.Len(t, loop.catchUpOwed, 2) + require.Len(t, loop.appliers, 2) // Two minutes of quiet: the one-minute applier is due, the one-hour one is not. loop.quietSince = time.Now().Add(-2 * time.Minute) @@ -197,8 +205,9 @@ func TestServiceAppliers_CatchUpWaitsForTheQuietPeriod(t *testing.T) { require.Len(t, requests, 1) assert.Equal(t, configv1alpha3.ApplyReasonCatchUp, requests[0].Reason) assert.Equal(t, "aaa", requests[0].Commit) - assert.Equal(t, []itypes.ResourceReference{itypes.NewResourceReference("soon", "team-a")}, requests[0].Targets) - assert.Contains(t, loop.catchUpOwed, itypes.NewResourceReference("late", "team-a")) + assert.Equal(t, []itypes.ResourceReference{itypes.NewResourceReference("soon", "team-a")}, + requests[0].TargetRefs()) + assert.True(t, loop.appliers[applierRef("late")].catchUp, "the one-hour applier still waits") assert.NotNil(t, loop.applierTimer) } @@ -277,7 +286,8 @@ func TestServiceAppliers_ASecondWriteInsideTheQuietPeriodRestartsIt(t *testing.T func TestServiceAppliers_AForeignCommitAnswersEverythingElseOwed(t *testing.T) { loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) - loop.oweReapply(refusalKeyFor(itypes.NewResourceReference("alpha", "team-a"), "deployments"), "refused", "obs") + loop.oweReapply(refusalKeyFor(itypes.NewResourceReference("alpha", "team-a"), "deployments"), + owedRefusal{detail: "refused", uid: "uid-alpha"}, "obs") owesForeign(loop.w) loop.serviceAppliers() @@ -286,7 +296,7 @@ func TestServiceAppliers_AForeignCommitAnswersEverythingElseOwed(t *testing.T) { require.Len(t, requests, 1, "one apply of the tip covers the refusal and the catch-up") assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, requests[0].Reason) assert.Empty(t, loop.reapplyOwed) - assert.Empty(t, loop.catchUpOwed) + assert.False(t, loop.appliers[applierRef("alpha")].catchUp) } func TestServiceAppliers_NoApplierOnTheBranchClearsTheDebt(t *testing.T) { @@ -298,7 +308,7 @@ func TestServiceAppliers_NoApplierOnTheBranchClearsTheDebt(t *testing.T) { assert.Empty(t, requester.taken()) assert.False(t, loop.w.debt.foreignOwed()) - assert.Empty(t, loop.catchUpOwed) + assert.Empty(t, loop.appliers) assert.Nil(t, loop.applierTimer) } @@ -394,7 +404,7 @@ func TestApplyTrigger_APushNoticeFindsAForeignCommit(t *testing.T) { require.Len(t, requests, 1) assert.Equal(t, gitOut(t, f.repoDir, "rev-parse", "main"), requests[0].Commit) assert.Equal(t, []itypes.ResourceReference{itypes.NewResourceReference(ledgerTargetName, "default")}, - requests[0].Targets) + requests[0].TargetRefs()) // A duplicate delivery of the same push is one more advertisement and no second request. loop.handleRefreshRequest(&RefreshRequest{PushNotice: true}) diff --git a/internal/git/branch_worker.go b/internal/git/branch_worker.go index 046c31c1..8a9f9513 100644 --- a/internal/git/branch_worker.go +++ b/internal/git/branch_worker.go @@ -141,6 +141,15 @@ type BranchWorker struct { // that do not exercise it). applyRequester ApplyRequester debt applyDebt + // applyWake wakes the loop to service the appliers without queueing a work item: an attempt + // reported back, or another worker writing this repository and branch went quiet. + applyWake chan struct{} + // siblings are the workers writing the same repository and branch as this one, under another + // GitProvider; their open windows and pending writes make the branch busy too. busyWriting is + // this worker's own, published for them at the end of every wake. Nil siblings (the CLI, tests) + // means this worker is alone. See apply_siblings.go. + siblings *branchSiblings + busyWriting atomic.Bool // Event processing eventQueue chan WorkItem @@ -192,7 +201,7 @@ type BranchWorker struct { // across every error return on the write path, and a lock there would put a second ordering // constraint on code whose only job is to record that something went wrong. // - // See the package's design note in docs/design/push-notification-and-reconcile-trigger.md §1.5. + // See the package's design note in docs/finished/push-notification-and-reconcile-trigger.md §1.5. baseTrustedState atomic.Bool // checkoutApplied is how many of the loop's retained writes the checkout holds: the checkout is @@ -409,6 +418,7 @@ func NewBranchWorker( ), contentWriter: writer, eventQueue: make(chan WorkItem, limits.QueueDepth), + applyWake: make(chan struct{}, 1), pendingResyncs: make(map[resyncKey]*resyncMarker), branchBufferMaxBytes: limits.MaxBufferBytes, } @@ -1153,21 +1163,18 @@ type branchWorkerEventLoop struct { // overwrite another's pending work: A queues, B replaces it, B is then suspended, and A's // authorized request is gone with nothing to report it. refusalPending map[refusalKey]pendingRefusalReapply - // reapplyOwed holds the refusals whose target's applier is owed a Refusal request, with the - // refusal each answers, until serviceAppliers sends it on a quiet branch. It is keyed like - // refusalPending, by target AND collection, so a collection that recovers cancels its own - // obligation and no other (refusalRecovered). See apply_trigger.go. - reapplyOwed map[refusalKey]string - // handBackCount is, per GitTarget, how many times its last request was handed back - // undelivered. The next request for the target carries it (ApplyRequest.HandBacks) and - // consumes it. See apply_trigger.go. - handBackCount map[itypes.ResourceReference]int - - // catchUpOwed holds, per GitTarget with an applier on this branch, how long the branch must stay - // quiet before that applier is asked to catch up with this worker's own pushes. quietSince is - // when the branch last became quiet (zero while it is busy), and applierTimer fires at the - // earliest catch-up still due. See apply_trigger.go. - catchUpOwed map[itypes.ResourceReference]time.Duration + // reapplyOwed holds the refusals whose target's applier is owed a Refusal request, until an + // attempt carries them on a quiet branch. It is keyed like refusalPending, by target AND + // collection, so a collection that recovers cancels its own obligation and no other + // (refusalRecovered). See apply_trigger.go. + reapplyOwed map[refusalKey]owedRefusal + + // appliers holds what each applier named on this branch is owed and how its attempts are going, + // and nextApplyID numbers the attempts. quietSince is when the branch last became quiet (zero + // while it is busy), and applierTimer fires when the earliest waiting attempt comes due. See + // apply_trigger.go. + appliers map[ApplierRef]*applierState + nextApplyID uint64 quietSince time.Time applierTimer *time.Timer // pushHinted records that a Git host reported a push while the branch was busy. The wake closes @@ -1235,6 +1242,9 @@ func (l *branchWorkerEventLoop) run() { case <-l.applierTimerC(): l.applierTimer = nil // serviceAppliers, at the end of the wake, sends what has come due. + case <-l.w.applyWake: + // An attempt reported back, or a sibling worker on this branch went quiet; + // serviceAppliers settles and sends. } l.endWake(0) } @@ -1322,8 +1332,6 @@ func (l *branchWorkerEventLoop) dispatch(item WorkItem) int64 { return charge case item.Refresh != nil: l.handleRefreshRequest(item.Refresh) - case item.ApplyWake: - // The wake is the point: endWake's serviceAppliers sends what was handed back. case item.Request == nil: case item.Request.CommitMode == CommitModeAtomic: targetName, targetNamespace := atomicRefusalTarget(item.Request) @@ -1905,7 +1913,7 @@ func (w *BranchWorker) stoppingLocked() bool { return w.stoppingState } // // The push session reads the remote's ref advertisement on a connection the cycle was making // anyway, and a cycle that commits nothing still reaches it, so a trusted base needs no fetch to -// plan against. That is the whole saving. See docs/design/push-notification-and-reconcile-trigger.md §1.3 and §1.5. +// plan against. That is the whole saving. See docs/finished/push-notification-and-reconcile-trigger.md §1.3 and §1.5. // // Only a checkout holding no retained writes may fetch here at all: a reset would destroy the // local commits those writes produced. With writes retained, the event loop's materialize has diff --git a/internal/git/refusal_reapply.go b/internal/git/refusal_reapply.go index b284e506..9ef9a871 100644 --- a/internal/git/refusal_reapply.go +++ b/internal/git/refusal_reapply.go @@ -14,6 +14,7 @@ import ( "github.com/ConfigButler/gitops-reverser/internal/manifestanalyzer" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + k8stypes "k8s.io/apimachinery/pkg/types" "github.com/ConfigButler/gitops-reverser/api/v1alpha3" itypes "github.com/ConfigButler/gitops-reverser/internal/types" @@ -69,21 +70,30 @@ func (k refusalKey) String() string { } // refusalConsent reports whether this GitTarget asked for its applier to reapply on refusal, -// ignoring rate limiting. It is re-read every time, including when a coalesced request finally -// fires, so a target that was suspended, set back to Ignore, or stripped of its applier in the -// meantime is honoured. +// ignoring rate limiting, and which incarnation of it asked. It is re-read every time, including +// when a coalesced request finally fires, so a target that was suspended, set back to Ignore, or +// stripped of its applier in the meantime is honoured. serviceAppliers reads consent once more +// before it sends. // // The GitTarget is read live rather than from the write's resolved metadata because a refusal has // no write to carry metadata: the plan was aborted before anything was built. A read failure // returns false, which is the safe direction — an unreadable target is not evidence of consent. -func (w *BranchWorker) refusalConsent(ctx context.Context, target itypes.ResourceReference) bool { +func (w *BranchWorker) refusalConsent(ctx context.Context, target itypes.ResourceReference) (k8stypes.UID, bool) { gitTarget, err := w.getGitTarget(ctx, target.Name, target.Namespace) if err != nil { w.Log.V(1).Info("Cannot read GitTarget to decide the refusal action; not requesting a reapply", "gitTarget", target.String(), "error", err.Error()) - return false + return "", false } - return ConsentsToReapply(gitTarget) + return gitTarget.UID, ConsentsToReapply(gitTarget) +} + +// owedRefusal is one refusal its target's applier is owed a Refusal request for: what was refused, +// for the logs, and the incarnation of the GitTarget that consented. A GitTarget deleted and +// recreated under the same name is another incarnation, and inherits nothing owed to the first. +type owedRefusal struct { + detail string + uid k8stypes.UID } // ConsentsToReapply reports whether a GitTarget, as it is now, asks for its applier to reapply on a @@ -143,7 +153,8 @@ func (l *branchWorkerEventLoop) reapplyForRefusal( return } target := itypes.NewResourceReference(targetName, targetNamespace) - if !l.w.refusalConsent(l.w.ctx, target) { + uid, consents := l.w.refusalConsent(l.w.ctx, target) + if !consents { return } key := newRefusalKey(target, collection) @@ -167,7 +178,7 @@ func (l *branchWorkerEventLoop) reapplyForRefusal( return } - l.oweReapply(key, detail, observation) + l.oweReapply(key, owedRefusal{detail: detail, uid: uid}, observation) } // refusalObservation digests what was refused: the objects the write would have produced, and the @@ -254,7 +265,7 @@ func (w *BranchWorker) refusalAlreadyCovered(key refusalKey, observation string) // recordRefusalObservation remembers what the request just owed covers. An empty observation is // recorded too, though it covers nothing (refusalAlreadyCovered): the entry is also the record that -// the refusal stands until it recovers (standingRefusals). +// the refusal stands until it recovers (refusalStands). func (w *BranchWorker) recordRefusalObservation(key refusalKey, observation string) { w.refusalMu.Lock() defer w.refusalMu.Unlock() @@ -308,19 +319,12 @@ func (l *branchWorkerEventLoop) refusalRecovered(target itypes.ResourceReference l.rearmRefusalTimer() } -// standingRefusals returns the refusals of target a request was owed for and that have not -// recovered since, in a stable order. -func (w *BranchWorker) standingRefusals(target itypes.ResourceReference) []refusalKey { +// refusalStands reports whether a request was owed for this refusal and it has not recovered since. +func (w *BranchWorker) refusalStands(key refusalKey) bool { w.refusalMu.Lock() defer w.refusalMu.Unlock() - var keys []refusalKey - for key := range w.coveredRefusal { - if key.target == target { - keys = append(keys, key) - } - } - sort.Slice(keys, func(i, j int) bool { return keys[i].String() < keys[j].String() }) - return keys + _, standing := w.coveredRefusal[newRefusalKey(key.target, key.collection)] + return standing } // clearRefusalObservations forgets what this collection's last reapply request covered. The ZERO collection is a @@ -401,7 +405,8 @@ func (l *branchWorkerEventLoop) flushPendingReapply() { } delete(l.refusalPending, key) - if !l.w.refusalConsent(l.w.ctx, entry.key.target) { + uid, consents := l.w.refusalConsent(l.w.ctx, entry.key.target) + if !consents { continue } // Re-checked here as well as on arrival: another request for this collection may have been owed @@ -417,7 +422,7 @@ func (l *branchWorkerEventLoop) flushPendingReapply() { l.armTrailingReapply(entry.key, entry.detail, entry.observation, wait) continue } - l.oweReapply(entry.key, entry.detail, entry.observation) + l.oweReapply(entry.key, owedRefusal{detail: entry.detail, uid: uid}, entry.observation) } l.rearmRefusalTimer() } @@ -435,18 +440,18 @@ func sortedRefusalKeys(pending map[refusalKey]pendingRefusalReapply) []refusalKe // oweReapply records that the target's applier is owed a Refusal request, which serviceAppliers // sends once the branch is quiet. Several refusals for one target before then share one request. -func (l *branchWorkerEventLoop) oweReapply(key refusalKey, detail, observation string) { +func (l *branchWorkerEventLoop) oweReapply(key refusalKey, refusal owedRefusal, observation string) { // Recorded as covered now, when the request is owed, rather than when it is delivered: the // request is the worker's decision, and a recheck of the same refusal while the branch is still - // busy must not owe a second one. A request the requester cannot deliver comes back - // (ApplyHandBack) and is owed again while the refusal stands. + // busy must not owe a second one. An attempt that does not deliver gives the refusal back while + // it stands (restoreAttempt). l.w.recordRefusalObservation(key, observation) if l.reapplyOwed == nil { - l.reapplyOwed = map[refusalKey]string{} + l.reapplyOwed = map[refusalKey]owedRefusal{} } - l.reapplyOwed[key] = detail + l.reapplyOwed[key] = refusal l.w.Log.Info("Refused edit owes its applier a reapply, sent once the branch is quiet", - "gitTarget", key.target.String(), "branch", l.w.Branch, "refused", detail) + "gitTarget", key.target.String(), "branch", l.w.Branch, "refused", refusal.detail) } // refusalIsAWriteBoundary reports whether this refusal is one where the FOLDER is accepted and diff --git a/internal/git/refusal_reapply_test.go b/internal/git/refusal_reapply_test.go index e9b3c055..e4cc5f62 100644 --- a/internal/git/refusal_reapply_test.go +++ b/internal/git/refusal_reapply_test.go @@ -3,7 +3,7 @@ package git // The reapply request that reverts a refused write, from -// docs/design/push-notification-and-reconcile-trigger.md §3.7. +// docs/design/apply-trigger.md. // // The request itself is one call; every interesting property is in the decision to make one. A // refusal is not a one-off, the action is opt-in per GitTarget and needs an applier, and an @@ -52,7 +52,7 @@ func refusalTouchWorker(t *testing.T, spec configv1alpha3.GitTargetSpec) *Branch spec.Applier = testApplier("editing") } target := &configv1alpha3.GitTarget{ - ObjectMeta: metav1.ObjectMeta{Name: "editing", Namespace: "team-a"}, + ObjectMeta: metav1.ObjectMeta{Name: "editing", Namespace: "team-a", UID: editingUID}, Spec: spec, } client := fake.NewClientBuilder().WithScheme(scheme).WithObjects(target).Build() @@ -67,6 +67,9 @@ func refusalTouchWorker(t *testing.T, spec configv1alpha3.GitTargetSpec) *Branch return w } +// editingUID is the uid of refusalTouchWorker's GitTarget. +const editingUID = "uid-editing" + func editingRef() itypes.ResourceReference { return itypes.NewResourceReference("editing", "team-a") } @@ -84,7 +87,7 @@ func editingKey() refusalKey { return refusalKeyFor(editingRef(), "deployments") func TestRefusalReapply_OffByDefault(t *testing.T) { w := refusalTouchWorker(t, configv1alpha3.GitTargetSpec{}) - assert.False(t, w.refusalConsent(w.ctx, editingRef()), + assert.False(t, consents(w, editingRef()), "a GitTarget that did not ask for a reapply must not get one") } @@ -94,7 +97,7 @@ func TestRefusalReapply_ExplicitIgnoreIsAlsoOff(t *testing.T) { OnRefusal: configv1alpha3.RefusalActionIgnore, }) - assert.False(t, w.refusalConsent(w.ctx, editingRef())) + assert.False(t, consents(w, editingRef())) } // TestRefusalReapply_AllowedWhenAskedFor is the positive case. @@ -103,7 +106,7 @@ func TestRefusalReapply_AllowedWhenAskedFor(t *testing.T) { OnRefusal: configv1alpha3.RefusalActionReapply, }) - assert.True(t, w.refusalConsent(w.ctx, editingRef())) + assert.True(t, consents(w, editingRef())) } // TestRefusalReapply_SuspendedTargetIsNeverTouched. A suspended target has stopped publishing the @@ -114,7 +117,7 @@ func TestRefusalReapply_SuspendedTargetIsNeverTouched(t *testing.T) { Suspend: true, }) - assert.False(t, w.refusalConsent(w.ctx, editingRef()), + assert.False(t, consents(w, editingRef()), "a suspended target must not drive the applier") } @@ -127,7 +130,7 @@ func TestRefusalReapply_NoApplierIsNoConsent(t *testing.T) { target.Spec.OnRefusal = configv1alpha3.RefusalActionReapply require.NoError(t, w.Client.Update(w.ctx, target)) - assert.False(t, w.refusalConsent(w.ctx, editingRef())) + assert.False(t, consents(w, editingRef())) } // TestRefusalReapply_RateLimitedPerTarget. A controller rewriting a base-owned field produces a @@ -162,7 +165,7 @@ func TestRefusalReapply_UnreadableTargetIsNotConsent(t *testing.T) { OnRefusal: configv1alpha3.RefusalActionReapply, }) - assert.False(t, w.refusalConsent(w.ctx, itypes.NewResourceReference("absent", "team-a")), + assert.False(t, consents(w, itypes.NewResourceReference("absent", "team-a")), "a GitTarget that cannot be read must not be treated as opted in") } @@ -177,7 +180,7 @@ func TestRefusalReapply_OwedRequestWaitsForAQuietBranch(t *testing.T) { t.Cleanup(loop.stopTimers) loop.openWindow = &openWindow{} - loop.oweReapply(editingKey(), "a base-owned field", "observation-1") + loop.oweReapply(editingKey(), owedRefusal{detail: "a base-owned field", uid: editingUID}, "observation-1") loop.serviceAppliers() assert.Empty(t, requester.taken(), "nothing is sent while a commit window is open") @@ -186,7 +189,7 @@ func TestRefusalReapply_OwedRequestWaitsForAQuietBranch(t *testing.T) { requests := requester.taken() require.Len(t, requests, 1) assert.Equal(t, configv1alpha3.ApplyReasonRefusal, requests[0].Reason) - assert.Equal(t, []itypes.ResourceReference{editingRef()}, requests[0].Targets, + assert.Equal(t, []itypes.ResourceReference{editingRef()}, requests[0].TargetRefs(), "only the refusing target's applier is asked") assert.True(t, w.refusalAlreadyCovered(editingKey(), "observation-1"), "the observation is covered once the request is owed") @@ -471,18 +474,30 @@ func testApplier(name string) *configv1alpha3.GitTargetApplier { return &configv1alpha3.GitTargetApplier{KustomizationRef: meta.NamespacedObjectReference{Name: name}} } -// recordingRequester records every apply request a worker sends. +// recordingRequester records every apply request a worker sends, and reports each one at once with +// outcome (Delivered unless a test sets another). With manual set it reports nothing, and the test +// reports for it. type recordingRequester struct { mu sync.Mutex requests []ApplyRequest - // counted keeps the reasons of the requests taken, so count still sees them. + // counted keeps the reasons of the requests taken, so refusals still sees them. counted map[configv1alpha3.ApplyReason]int + outcome ApplyOutcome + manual bool } func (r *recordingRequester) RequestApply(req ApplyRequest) { r.mu.Lock() - defer r.mu.Unlock() r.requests = append(r.requests, req) + outcome, manual := r.outcome, r.manual + r.mu.Unlock() + if manual { + return + } + if outcome == "" { + outcome = ApplyDelivered + } + req.Report(ApplyResult{ID: req.ID, Outcome: outcome}) } // refusals is how many Refusal requests were recorded, taken or not. @@ -512,3 +527,9 @@ func (r *recordingRequester) taken() []ApplyRequest { } return requests } + +// consents is refusalConsent's verdict alone. +func consents(w *BranchWorker, target itypes.ResourceReference) bool { + _, ok := w.refusalConsent(w.ctx, target) + return ok +} diff --git a/internal/git/types.go b/internal/git/types.go index 8748dc43..2511cef5 100644 --- a/internal/git/types.go +++ b/internal/git/types.go @@ -424,9 +424,6 @@ type WorkItem struct { // Withdraw cancels a CommitRequest the worker has not acted on yet. It rides the same FIFO as // the attach, so it is always handled after every attach sent before it. Withdraw *AttachCommitRequest - // ApplyWake wakes the loop after an apply request was handed back (ApplyHandBack). It carries - // nothing: the request is in the worker's debt, and the wake's serviceAppliers sends it. - ApplyWake bool // charge is what a write or an attach was charged against the intake budget at enqueue, and is // released once the loop has handled it. A resync's is on its marker, because coalescing can diff --git a/internal/git/worker_manager.go b/internal/git/worker_manager.go index 4182e543..1f13c872 100644 --- a/internal/git/worker_manager.go +++ b/internal/git/worker_manager.go @@ -164,6 +164,10 @@ type WorkerManager struct { // PublicationEvents. publicationEvents chan event.GenericEvent + // siblings groups the workers by the repository and branch they write, so a worker never asks + // for an apply while another worker on the same branch is busy writing. See apply_siblings.go. + siblings *branchSiblings + // applyRequester delivers every worker's apply requests. Set once at startup // (SetApplyRequester) before any worker is created; nil in the CLI and in tests, which switches // the apply trigger off. @@ -184,6 +188,7 @@ func NewWorkerManager( limits: limits.withDefaults(), sensitiveResources: sensitiveResources, workers: make(map[BranchKey]*BranchWorker), + siblings: newBranchSiblings(), commitRequests: &commitRequestOwners{}, remotes: make(map[BranchKey]RemoteObservation), replacements: make(map[BranchKey]struct{}), @@ -387,10 +392,12 @@ func (m *WorkerManager) EnsureWorker( worker.publicationReporter = func() { go m.notifyPublication(managerCtx, key) } worker.scanAcceptance = m.scanAcceptance worker.applyRequester = m.applyRequester + m.siblings.add(worker) worker.renderFidelityGate = m.renderFidelityGate worker.crOwners = m.commitRequests if err := worker.Start(m.ctx); err != nil { + m.siblings.remove(worker) return fmt.Errorf("failed to start worker for %s: %w", key.String(), err) } @@ -470,6 +477,7 @@ func (m *WorkerManager) removeWorkersLocked(keys []BranchKey, reason string) { for key, worker := range detached { m.Log.Info("Stopping branch worker", "key", key.String(), "reason", reason) worker.Stop() + m.siblings.remove(worker) // After Stop, never before: Stop waits for the loop goroutine, so until it returns the // worktree can still be written. A worker being retired takes its clone with it — the // goroutine was only half the leak, and the checkout is the half that survives a diff --git a/internal/telemetry/apply_requests.go b/internal/telemetry/apply_requests.go index 1467e5d6..4df44a34 100644 --- a/internal/telemetry/apply_requests.go +++ b/internal/telemetry/apply_requests.go @@ -4,25 +4,27 @@ package telemetry import "go.opentelemetry.io/otel/metric" -// ApplyRequestsTotal counts the apply requests GitOps Reverser decided, one per applier object per -// request, labelled by {applier_kind, reason, outcome}. +// ApplyRequestsTotal counts the apply attempts GitOps Reverser made, one per attempt to one applier +// object, labelled by {applier_kind, reason, outcome}. // -// applier_kind names the applier: "kustomization" (a Flux Kustomization). reason is the request's +// applier_kind names the applier: "kustomization" (a Flux Kustomization). reason is the attempt's // ApplyReason (ForeignCommit, Refusal, CatchUp). outcome is a frozen enum: // -// sent — the applier was asked and the ask landed: a requestedAt stamp was patched, or, for -// a CatchUp, the applier's source already held the commit and nothing was owed. -// failed — every attempt failed and the request was given up on. An ApplyRequestFailed -// Event names the GitTargets it was for. -// superseded — a newer request for the same applier replaced this one before it was sent. The -// newer request carries the newer commit, so nothing is lost; a steady stream of -// these says requests are arriving faster than the applier is answering. +// sent — the applier was asked and the ask landed: a requestedAt stamp was patched, or, for a +// CatchUp, the applier's source already held the commit and nothing was owed. +// failed — the attempt could not ask, including an applier that applies another repository or +// branch. The branch worker owns the obligation and decides a retry on a quiet branch, +// with a backoff of its own, so a failure that persists keeps counting. An +// ApplyRequestFailed Event marks the first failure of a run. +// dropped — the attempt asked nothing, because no GitTarget may still ask for its reason: each +// was deleted or recreated, moved to another branch or applier, is no longer admitted by +// its ClusterProvider, or withdrew its consent to a Refusal. // -// It counts decisions, not attempts: a request that fails twice and then lands is one "sent". The -// retries are in the logs. It says nothing about whether the applier APPLIED: a request asks and -// never reports, and the applier's own status is the record of what it did. +// It counts attempts, so a request that fails twice and then lands is two "failed" and one "sent". +// It says nothing about whether the applier APPLIED: a request asks and never reports, and the +// applier's own status is the record of what it did. // -// No GitTarget identity: one request is for one applier, which several targets can share, so a +// No GitTarget identity: one attempt is for one applier, which several targets can share, so a // target label would have to invent an owner. The ApplyRequested and ApplyRequestFailed Events on // each GitTarget carry that identity instead. var ApplyRequestsTotal metric.Int64Counter From cdb0d2107d3c5cda64b38cc5a7340ef106614498 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 21:18:13 +0000 Subject: [PATCH 18/36] test: drop the empty-commit trick and repoint design citations The bi-directional corner proved that a new GitRepository revision alone makes a Kustomization apply by pushing an empty commit. It now pushes a commit that changes only a file outside the applied folder, which proves the same property without the trick. Code and test comments cite the new apply trigger design, or the archived one for the Git-side reference. Co-Authored-By: Claude Opus 5.5 (1M context) --- api/v1alpha3/gittarget_types.go | 2 +- .../templates/applier-rbac.yaml | 2 +- internal/applier/flux.go | 2 +- internal/git/ado_multiack_test.go | 2 +- internal/git/branch_worker_base_trust_test.go | 2 +- .../git/branch_worker_fetch_metric_test.go | 2 +- internal/git/deleted_remote_branch_test.go | 2 +- internal/git/dirty_worktree_recovery_test.go | 2 +- internal/git/git_push_advertisement_test.go | 2 +- internal/git/git_request_ledger_test.go | 2 +- internal/git/git_roundtrip_ledger_test.go | 4 +- internal/git/refresh_test.go | 4 +- internal/git/replay_failure_test.go | 2 +- internal/git/reset_cleanliness_test.go | 2 +- .../git/testdata/git-roundtrip-ledger.golden | 2 +- test/e2e/flux_applier_e2e_test.go | 4 +- test/e2e/flux_bi_directional_e2e_test.go | 48 ++++++++++--------- test/e2e/refusal_reapply_e2e_test.go | 5 +- 18 files changed, 46 insertions(+), 45 deletions(-) diff --git a/api/v1alpha3/gittarget_types.go b/api/v1alpha3/gittarget_types.go index e542f354..6570972a 100644 --- a/api/v1alpha3/gittarget_types.go +++ b/api/v1alpha3/gittarget_types.go @@ -90,7 +90,7 @@ type GitTargetSpec struct { // Why a reference rather than an inference: deriving which Kustomization applies a folder would // be guessing on Flux's behalf, and the guess would drive a write. A reference the user declares - // claims nothing. See docs/design/push-notification-and-reconcile-trigger.md §3.1. + // claims nothing. See docs/design/apply-trigger.md. // Applier names the object that applies this folder back to its source cluster. With it set, // GitOps Reverser decides when that object applies: it asks for an apply only when the branch diff --git a/charts/gitops-reverser/templates/applier-rbac.yaml b/charts/gitops-reverser/templates/applier-rbac.yaml index 60f39f66..f5a5fe94 100644 --- a/charts/gitops-reverser/templates/applier-rbac.yaml +++ b/charts/gitops-reverser/templates/applier-rbac.yaml @@ -4,7 +4,7 @@ # reconcile request on them), and create SelfSubjectAccessReviews so the operator can check it holds # that access before relying on it. Separate from the manager role because it is optional and # names another project's API: an install without Flux grants nothing on it. See -# docs/design/push-notification-and-reconcile-trigger.md §4.4. +# docs/design/apply-trigger.md. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: diff --git a/internal/applier/flux.go b/internal/applier/flux.go index 0ceeed86..b0f81a11 100644 --- a/internal/applier/flux.go +++ b/internal/applier/flux.go @@ -7,7 +7,7 @@ commit is safe to apply. The worker decides WHEN; this package decides HOW, off so a slow or unreachable applier never holds a branch. Only Flux is served: spec.applier.kustomizationRef, a Kustomization whose sourceRef is a -GitRepository. See docs/design/push-notification-and-reconcile-trigger.md §3.6 and §3.8. +GitRepository. See docs/design/apply-trigger.md. A request asks and never reports. Nothing here claims an apply happened; the applier's own status (lastAppliedRevision on a Kustomization) is the record of that. diff --git a/internal/git/ado_multiack_test.go b/internal/git/ado_multiack_test.go index b2ef60f0..59db008d 100644 --- a/internal/git/ado_multiack_test.go +++ b/internal/git/ado_multiack_test.go @@ -57,7 +57,7 @@ type adoSimulator struct { // ledger tallies every request the backend sees, with bytes, so an operation's cost in // round trips to the Git host can be measured rather than guessed. See - // git_request_ledger_test.go and docs/design/push-notification-and-reconcile-trigger.md §1.6. + // git_request_ledger_test.go and docs/finished/push-notification-and-reconcile-trigger.md §1.6. ledger *gitRequestLedger // around, when set, serves each request in place of the backend and decides whether and how diff --git a/internal/git/branch_worker_base_trust_test.go b/internal/git/branch_worker_base_trust_test.go index 37800fd9..300f733f 100644 --- a/internal/git/branch_worker_base_trust_test.go +++ b/internal/git/branch_worker_base_trust_test.go @@ -2,7 +2,7 @@ package git -// The base-trust state machine from docs/design/push-notification-and-reconcile-trigger.md §1.5. +// The base-trust state machine from docs/finished/push-notification-and-reconcile-trigger.md §1.5. // // Nothing reads these flags to make a decision yet and the setter is hard-wired off, so none of // this can change behavior — which is exactly why the tests are written now. The hard part of the diff --git a/internal/git/branch_worker_fetch_metric_test.go b/internal/git/branch_worker_fetch_metric_test.go index 25fa2505..90519a4a 100644 --- a/internal/git/branch_worker_fetch_metric_test.go +++ b/internal/git/branch_worker_fetch_metric_test.go @@ -2,7 +2,7 @@ package git -// Red-first, per docs/design/push-notification-and-reconcile-trigger.md §1.7: today's behavior is pinned as a +// Red-first, per docs/finished/push-notification-and-reconcile-trigger.md §1.7: today's behavior is pinned as a // passing assertion, so the change that removes the head-of-cycle fetch is proved by this // assertion flipping rather than by argument. // diff --git a/internal/git/deleted_remote_branch_test.go b/internal/git/deleted_remote_branch_test.go index de85cadb..53619da2 100644 --- a/internal/git/deleted_remote_branch_test.go +++ b/internal/git/deleted_remote_branch_test.go @@ -16,7 +16,7 @@ package git // before the push, so the push simply re-created the branch. These tests are what keep that // recovery once the prefetch is conditional. // -// See docs/design/push-notification-and-reconcile-trigger.md §1.3 and §1.5. +// See docs/finished/push-notification-and-reconcile-trigger.md §1.3 and §1.5. import ( "context" diff --git a/internal/git/dirty_worktree_recovery_test.go b/internal/git/dirty_worktree_recovery_test.go index fda8bad5..da53f902 100644 --- a/internal/git/dirty_worktree_recovery_test.go +++ b/internal/git/dirty_worktree_recovery_test.go @@ -11,7 +11,7 @@ package git // refuses a checkout that is not the projection of the retained writes, so a path that skipped // materialize fails rather than committing leftovers. // -// See docs/design/push-notification-and-reconcile-trigger.md §1.5. +// See docs/finished/push-notification-and-reconcile-trigger.md §1.5. import ( "errors" diff --git a/internal/git/git_push_advertisement_test.go b/internal/git/git_push_advertisement_test.go index 1a25cf6d..981829b6 100644 --- a/internal/git/git_push_advertisement_test.go +++ b/internal/git/git_push_advertisement_test.go @@ -6,7 +6,7 @@ package git // the worker now listens, and that it still falls back to the network when the remote said // nothing. // -// See docs/design/push-notification-and-reconcile-trigger.md §1.3. +// See docs/finished/push-notification-and-reconcile-trigger.md §1.3. import ( "context" diff --git a/internal/git/git_request_ledger_test.go b/internal/git/git_request_ledger_test.go index 43a6a4f0..dce43f93 100644 --- a/internal/git/git_request_ledger_test.go +++ b/internal/git/git_request_ledger_test.go @@ -10,7 +10,7 @@ package git // every row carries request and response bytes beside its count, and the golden file publishes a // coarse size class for them — see sizeClass for why the class rather than the number. // -// See docs/design/push-notification-and-reconcile-trigger.md §1.6, which specifies this ledger. +// See docs/finished/push-notification-and-reconcile-trigger.md §1.6, which specifies this ledger. import ( "fmt" diff --git a/internal/git/git_roundtrip_ledger_test.go b/internal/git/git_roundtrip_ledger_test.go index 04df94eb..94100e65 100644 --- a/internal/git/git_roundtrip_ledger_test.go +++ b/internal/git/git_roundtrip_ledger_test.go @@ -2,7 +2,7 @@ package git -// This is the measurement docs/design/push-notification-and-reconcile-trigger.md §1.6 asks for, before any of +// This is the measurement docs/finished/push-notification-and-reconcile-trigger.md §1.6 asks for, before any of // the changes that page argues for. Every connection count in §4 of that page is a prediction // derived from reading code; this file replaces reading with observing. // @@ -557,7 +557,7 @@ func renderLedger(rows []ledgerRow) string { var b strings.Builder b.WriteString("# Round trips to the Git host, per operation.\n") b.WriteString("# Generated by TestGitRoundTripLedger; specified by\n") - b.WriteString("# docs/design/push-notification-and-reconcile-trigger.md §1.6.\n") + b.WriteString("# docs/finished/push-notification-and-reconcile-trigger.md §1.6.\n") b.WriteString("#\n") b.WriteString("# Regenerate with: go test ./internal/git -run TestGitRoundTripLedger -update\n") b.WriteString("#\n") diff --git a/internal/git/refresh_test.go b/internal/git/refresh_test.go index 7c294d47..f9c7c920 100644 --- a/internal/git/refresh_test.go +++ b/internal/git/refresh_test.go @@ -500,9 +500,7 @@ func TestRefresh_PublishesTheRecoveryWhenTheFolderIsFixed(t *testing.T) { // TestRefresh_WritesNothingWhileTheFolderIsRefused is the boundary, stated against the change that // made the refresher publish a refusal at all. Raising GitPathAccepted=False drives a recheck and -// a resync, and that recovery machinery is the point — but the refresher itself must not commit, -// not even the empty commit spec.onRefusal produces, which belongs to a path that had work in -// hand. +// a resync, and that recovery machinery is the point — but the refresher itself must not commit. func TestRefresh_WritesNothingWhileTheFolderIsRefused(t *testing.T) { h := newRefreshHarness(t, "refresh-refused-writes-nothing") h.publish("prime") diff --git a/internal/git/replay_failure_test.go b/internal/git/replay_failure_test.go index d0e90eac..ad6e3957 100644 --- a/internal/git/replay_failure_test.go +++ b/internal/git/replay_failure_test.go @@ -18,7 +18,7 @@ package git // worktreeDirty is false throughout: the reset cleared it, and the worktree really is clean. It is // the retained writes that are stale, which is why this needs an invariant of its own. // -// See docs/design/push-notification-and-reconcile-trigger.md §1.5. +// See docs/finished/push-notification-and-reconcile-trigger.md §1.5. import ( "context" diff --git a/internal/git/reset_cleanliness_test.go b/internal/git/reset_cleanliness_test.go index 44c9c487..71aae9e2 100644 --- a/internal/git/reset_cleanliness_test.go +++ b/internal/git/reset_cleanliness_test.go @@ -12,7 +12,7 @@ package git // reaches the server, so the compare-and-swap caveat that forces other tests onto // startRealGitServer does not apply. // -// See docs/design/push-notification-and-reconcile-trigger.md §1.5. +// See docs/finished/push-notification-and-reconcile-trigger.md §1.5. import ( "context" diff --git a/internal/git/testdata/git-roundtrip-ledger.golden b/internal/git/testdata/git-roundtrip-ledger.golden index 1084d44f..4a97bc80 100644 --- a/internal/git/testdata/git-roundtrip-ledger.golden +++ b/internal/git/testdata/git-roundtrip-ledger.golden @@ -1,6 +1,6 @@ # Round trips to the Git host, per operation. # Generated by TestGitRoundTripLedger; specified by -# docs/design/push-notification-and-reconcile-trigger.md §1.6. +# docs/finished/push-notification-and-reconcile-trigger.md §1.6. # # Regenerate with: go test ./internal/git -run TestGitRoundTripLedger -update # diff --git a/test/e2e/flux_applier_e2e_test.go b/test/e2e/flux_applier_e2e_test.go index 4de58a70..dcfd5216 100644 --- a/test/e2e/flux_applier_e2e_test.go +++ b/test/e2e/flux_applier_e2e_test.go @@ -16,7 +16,7 @@ import ( ) // GitTarget spec.applier against a real Flux, from what a request is allowed to do to how GitOps -// Reverser learns that it has to make one. See docs/design/apply-trigger-follow-ups.md. +// Reverser learns that it has to make one. See docs/design/apply-trigger.md. // // The Kustomization applies its own folder (fluxApplierFolder) into a namespace the WatchRule does // not watch, so nothing Flux applies is mirrored back and every commit on the branch is either this @@ -145,7 +145,7 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order // With no receiver and no API edit, the only thing that can find a Git-side push is the // refresher on the GitTarget's steady tick. This is the guarantee that a missing webhook costs - // latency and never an apply (docs/design/push-notification-and-reconcile-trigger.md §4.5 item 6). + // latency and never an apply (docs/design/apply-trigger.md). // The e2e install refreshes after 30s, so the wait is the control plane's reconcile cadence of // about five minutes, as in the refresh-remote spec. It("applies a Git-side push the refresher finds, with no receiver and no API edit", func() { diff --git a/test/e2e/flux_bi_directional_e2e_test.go b/test/e2e/flux_bi_directional_e2e_test.go index 4aa8b134..e9ea16e6 100644 --- a/test/e2e/flux_bi_directional_e2e_test.go +++ b/test/e2e/flux_bi_directional_e2e_test.go @@ -338,21 +338,21 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order }) // The Flux property GitOps Reverser's apply requests rest on - // (docs/design/push-notification-and-reconcile-trigger.md §3.6): **a new GitRepository revision + // (docs/design/apply-trigger.md): **a new GitRepository revision // is enough**, on its own, to make the Kustomization apply and revert live drift, even when the // manifests it builds are byte-identical to the ones it applied last time. A request that finds // the artifact behind the commit stamps only the GitRepository and relies on exactly this. // - // So this spec isolates that claim and nothing else, with a commit that changes no file pushed - // by hand. Two details carry the argument, and removing either makes the spec pass for the - // wrong reason: + // So this spec isolates that claim and nothing else, with a commit pushed by hand that changes + // only a file outside the folder the Kustomization builds. Two details carry the argument, and + // removing either makes the spec pass for the wrong reason: // // - The GitTarget is SUSPENDED first. Otherwise the operator publishes the drift to Git and // there is no drift left for Flux to correct, so the revert would prove nothing. // - Only the GitRepository is reconciled, never the Kustomization. The Kustomization is on // a 30m interval, so if it applies inside this spec it is because a new source revision // woke it. - It("reverts live drift from an empty commit, because a new revision is enough", func() { + It("reverts live drift when only the source moves, because a new revision is enough", func() { By("suspending the GitTarget so the drift stays drift instead of being published") _, err := kubectlRunInNamespace(testNs, "patch", "gittarget", run.gitTargetName, "--type=merge", "-p", `{"spec":{"suspend":true}}`) @@ -379,21 +379,24 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order Expect(err).NotTo(HaveOccurred(), "failed to drift the live IceCreamOrder") Eventually(func(g Gomega) { g.Expect(liveOrderContainer(g, run, run.secondOrderName)).To(Equal("WaffleBowl"), - "the drift must be live before the empty commit is pushed") + "the drift must be live before the source moves") }, 30*time.Second, 2*time.Second).Should(Succeed()) By("confirming the suspended operator leaves the drift alone") run.consistentlyExpectRemoteCommitCount(baselineCommitCount, biStableCountMediumWait) - By("pushing a commit that changes no file") - Expect(run.commitAllAndPush("bi-directional: empty commit to trigger a reconcile")).To(Succeed()) - emptyHead := run.gitHEAD() - Expect(run.emptyDiffAt(emptyHead)).To(BeTrue(), - "the commit under test must change no file, or this spec is measuring an ordinary sync") + By("pushing a commit that changes nothing the Kustomization builds") + Expect(os.WriteFile(filepath.Join(run.checkoutDir, "NOTES.md"), + []byte("A change outside the applied folder.\n"), 0o600)).To(Succeed()) + Expect(run.commitAllAndPush("bi-directional: a change outside the applied folder")).To(Succeed()) + sourceHead := run.gitHEAD() + Expect(run.changedFilesAt(sourceHead)).To(Equal([]string{"NOTES.md"}), + "the commit under test must change nothing the Kustomization builds, or this spec is "+ + "measuring an ordinary sync") By("waking only the source, never the Kustomization") run.reconcileFluxSource() - run.waitForFluxGitRepositoryRevision(emptyHead) + run.waitForFluxGitRepositoryRevision(sourceHead) // A longer budget than the shared helper's, and deliberately so: every other caller has // just reconciled the Kustomization by hand, while this one is waiting for Flux to notice @@ -409,8 +412,8 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order obj.Object, "status", "lastAppliedRevision") g.Expect(revErr).NotTo(HaveOccurred()) g.Expect(found).To(BeTrue()) - g.Expect(revision).To(ContainSubstring(emptyHead), - "the Kustomization must reach the empty commit without being reconciled by hand") + g.Expect(revision).To(ContainSubstring(sourceHead), + "the Kustomization must reach the new revision without being reconciled by hand") }, 3*time.Minute, biPollInterval).Should(Succeed()) Eventually(func(g Gomega) { g.Expect(liveOrderContainer(g, run, run.secondOrderName)).To(Equal("Cup"), @@ -418,10 +421,9 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order }, 2*time.Minute, 2*time.Second).Should(Succeed()) }) - // The race GitOps Reverser's apply trigger exists for, from - // docs/design/bi-directional-right.md §7: two API edits to one field a few seconds apart. With - // Flux woken by every push, the first commit is applied over the second edit and the branch - // ping-pongs. With the Kustomization named as the GitTarget's applier, Flux is asked to catch up + // The race GitOps Reverser's apply trigger exists for, from docs/design/apply-trigger.md: two + // API edits to one field a few seconds apart. With Flux woken by every push, the first commit is + // applied over the second edit and the branch ping-pongs. With the Kustomization named as the GitTarget's applier, Flux is asked to catch up // only once the branch has been quiet, so the later value survives and Flux still ends on the // tip, without anybody reconciling it by hand. It("asks Flux to apply only on a quiet branch, so the later of two quick API edits survives", func() { @@ -462,7 +464,7 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order }) // A Git-side push with an API edit on top of it, from - // docs/design/push-notification-and-reconcile-trigger.md §3.4. Nothing tells GitOps Reverser + // docs/design/apply-trigger.md. Nothing tells GitOps Reverser // about the push here, so it finds it the hard way: its own push is rejected, it replays the API // edit onto the Git-side commit, and only then asks Flux to apply, so the commit Flux applies // already carries the API side's answer. Nobody reconciles Flux by hand. @@ -501,15 +503,15 @@ var _ = Describe("Bi Directional (Flux)", Label("bi-directional", "flux"), Order }) }) -// emptyDiffAt reports whether the commit changed no file, which is the property that separates -// this from an ordinary Git-side change driving an ordinary sync. -func (r biDirectionalRun) emptyDiffAt(sha string) bool { +// changedFilesAt lists the files a commit changed, which is what separates a commit outside the +// applied folder from an ordinary Git-side change driving an ordinary sync. +func (r biDirectionalRun) changedFilesAt(sha string) []string { GinkgoHelper() cmd := exec.Command("git", "diff-tree", "--no-commit-id", "--name-only", "-r", sha) cmd.Dir = r.checkoutDir out, err := cmd.CombinedOutput() Expect(err).NotTo(HaveOccurred(), "git diff-tree: %s", out) - return strings.TrimSpace(string(out)) == "" + return strings.Fields(string(out)) } // liveOrderContainer reads one field of the live object, which is all this spec's drift needs. diff --git a/test/e2e/refusal_reapply_e2e_test.go b/test/e2e/refusal_reapply_e2e_test.go index 450f341c..9fcd9778 100644 --- a/test/e2e/refusal_reapply_e2e_test.go +++ b/test/e2e/refusal_reapply_e2e_test.go @@ -293,8 +293,9 @@ func remoteCommitCount(checkoutDir string) int { return count } -// emptyCommitCount counts the commits on the remote branch whose diff is empty. A refusal moves the -// branch for nothing, so it must stay at zero however often the refusal is answered. +// emptyCommitCount counts the commits on the remote branch whose diff is empty. A refusal is answered +// by asking its applier; nothing is committed for it, so the count must not grow however often the +// refusal is answered. func emptyCommitCount(checkoutDir string) int { GinkgoHelper() Expect(runGitIn(checkoutDir, "fetch", "origin", "main")).To(Succeed()) From 483adc5af72c680c7207f8d130737e63388c6a75 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 21:18:13 +0000 Subject: [PATCH 19/36] docs: a fresh apply trigger design, and the aim restored docs/design/apply-trigger.md replaces the trigger design (now archived in docs/finished/) with why GitOps Reverser decides when Git is applied, what building it taught, the three units that make the 1:n cases work, the apply window that is still racy and how to shrink it, and the aim beyond Flux. The follow-ups page and gitops-api's original ask are removed. architecture.md and configuration.md state the aim again (Argo CD with a pinned sync and no catch-up, a webhook applier) beside what is built, and describe the worker-owned obligations. The refusal empty commit is gone from the design docs. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/INDEX.md | 3 +- docs/api-first-publication.md | 4 +- docs/architecture.md | 73 ++++++++++++------- docs/bi-directional.md | 7 +- docs/configuration.md | 51 ++++++++----- docs/design/apply-trigger.md | 17 +++-- docs/design/branch-worker-event-model.md | 7 +- docs/design/event-pipeline-overview.md | 5 +- .../gittarget-branch-worker-pending-writes.md | 17 ++--- docs/design/gittarget-parent-branch.md | 2 +- docs/design/push-cooldown.md | 6 +- docs/design/reconcile-triggering.md | 4 +- docs/design/support-boundary/README.md | 2 +- .../support-boundary/admission-consent.md | 6 +- .../support-boundary/render-fidelity.md | 4 +- ...push-notification-and-reconcile-trigger.md | 11 ++- docs/flux.md | 41 +++++++---- docs/future/git-api-three-way-comparison.md | 2 +- docs/interpreting-metrics.md | 2 +- docs/spec/commitrequest-design.md | 2 +- 20 files changed, 154 insertions(+), 112 deletions(-) diff --git a/docs/INDEX.md b/docs/INDEX.md index 6ca67c5c..aff9878e 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -151,8 +151,7 @@ so you never have to read a proposal to find out it already shipped. | [`watch-and-catalog-architecture.md`](design/watch-and-catalog-architecture.md) | the target three-layer watch model — **needs a human call before building** | | [`metrics-observability-plan.md`](design/metrics-observability-plan.md) | **built.** the canonical metrics document, rewritten 2026-09 and shipped in the same wave. Reads the product as two ingestion paths — object state through watch, authorship through audit — joining at the resolver, and gives each stage one counter with a bounded `outcome` so the flow and its loss paths are drawable. Records what three review rounds corrected, including two reversals worth not re-litigating: the fact-loss counters must NOT be merged (an eviction counts a fact, a trim gap an occurrence, a decode error a whole batch), and a cumulative counter cannot replace a current count, which is why `status.streams` and `status.retainedDocuments` stayed. The instrument inventory lives in [`interpreting-metrics.md`](interpreting-metrics.md), not here. Open: the Grafana JSON and alert rules | | [`reconcile-triggering.md`](design/reconcile-triggering.md) | which controllers still fail to wake up | -| [`push-notification-and-reconcile-trigger.md`](design/push-notification-and-reconcile-trigger.md) | **built for Flux.** GitOps Reverser owns the apply trigger: a `GitTarget` names its applier, the applier's own short triggers are switched off, and the branch worker sends apply requests only on a quiet branch (no open commit window, no pending writes) — `ForeignCommit` after replaying pending writes onto somebody else's push, `Refusal` for `onRefusal: Reapply`, `CatchUp` after a quiet period so Flux's artifact never falls behind. Part 1 keeps the push-is-the-read invariant and the measured round trips; the receiver contract, status, metrics and tests are its reference part. Argo CD and the webhook applier are designed and not built | -| [`apply-trigger-follow-ups.md`](design/apply-trigger-follow-ups.md) | **open.** What comes after the Flux applier: the Argo CD applier and how to build it, the `PushEmptyCommit` regression for appliers GitOps Reverser cannot name, smaller loose ends, and the parts of the trigger the tests do not reach yet | +| [`apply-trigger.md`](design/apply-trigger.md) | **built for Flux; Argo CD and a webhook applier are the aim.** Why GitOps Reverser decides when Git is applied: an apply is safe only when nothing live is missing from the commit, and only the branch's writers can see that. Carries what building it taught (timers are triggers, Flux has no fetch-only mode, a decision must hold at the patch, a retry is a decision so only the worker makes one) and the three units that make the 1:n cases work: the branch owns quietness, the applier owns obligations and retries, the `GitTarget` (by uid) owns consent. Ends with the apply window that is still racy and three ways to shrink it | | [`push-cooldown.md`](design/push-cooldown.md) | **partly built.** Whether the fixed `5s` push cooldown still earns its place now that the commit window exists. Ordinary timer closures already space one author writing one target, but identity changes, saves with shorter timers and buffer-limit closures commit sooner, and serialization does not batch queued commits, so the cooldown is what batches them. Option C, the failure backoff, is built in #412 and shares the push timer; removing the success wait would now remove only `lastPushAt` and its branch, so that decision waits on the event-loop measurements in §9 | | [`multi-source-audit-ingress-hardening.md`](design/multi-source-audit-ingress-hardening.md) | how independent sources authenticate to a named audit route, when annotation routing is trustworthy, and how multi-provider ingestion remains fair | | [`release-image-reuse-plan.md`](design/release-image-reuse-plan.md) | PRs 2–5 unstarted | diff --git a/docs/api-first-publication.md b/docs/api-first-publication.md index 6edce7eb..68552831 100644 --- a/docs/api-first-publication.md +++ b/docs/api-first-publication.md @@ -28,10 +28,10 @@ common case of a competing Git writer. The push already checks the remote branch worker can plan locally and spend its network budget on publication. This guide describes the shipped behavior. The inbound Git push receiver remains a proposal: see -[how to call the receiver](design/push-notification-and-reconcile-trigger.md#83-the-wire-contract-for-whoever-calls-it) +[how to call the receiver](configuration.md#receiving-push-notifications-specreceiver) for the request shape it will accept. The one scheduling limitation to know about is that retained work has no retry timer of its own; [failure timing](#failure-timing-and-memory-limits) says what -that means in practice. The [inbound notification design](design/push-notification-and-reconcile-trigger.md) +that means in practice. The [apply trigger design](design/apply-trigger.md) contains the implementation history and the proposed receiver; [architecture](architecture.md) covers the wider operator. diff --git a/docs/architecture.md b/docs/architecture.md index 3cd1c734..8cc1c604 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -79,7 +79,7 @@ lock the parent. The parent can still move after the advertisement, so what is g parent's tip as that advertisement showed it. **The corollary is the rule to keep in your head when adding code here:** anything that resolves, commits, or concludes without reaching a push advertisement must either fetch, or refuse to conclude. See -[inbound push notification](design/push-notification-and-reconcile-trigger.md) §3. +[inbound push notification](design/apply-trigger.md). That trust belongs to **one repository**, and so does the worker holding it. A worker is keyed by `(GitProvider namespace, GitProvider name, branch)`, which names no repository, while `spec.url` is @@ -116,15 +116,24 @@ cluster snapshot: the cluster still holds the values from before the push, so a publish them over the incoming change. **GitOps Reverser decides when Git is applied to the cluster.** An apply is safe only when nothing -live is missing from the commit being applied, and only the branch worker can see the edits that are -live and not yet pushed. So a `GitTarget` names its **applier** (a Flux `Kustomization`), the -applier's own short triggers are switched off, and the branch worker sends **apply requests** -instead. A request is sent only on a **quiet branch**, with no open commit window and no pending -writes: for a commit somebody else pushed (after closing open windows and replaying the pending -writes onto it), for a refused edit the target opted to revert, and to keep Flux's artifact current -after GitOps Reverser's own pushes, because Flux's timers and restarts apply whatever artifact it -holds. A request asks and never reports; it runs off the publication path and cannot fail a -publication. See [the apply trigger design](design/push-notification-and-reconcile-trigger.md). +live is missing from the commit being applied, and only the workers writing the branch can see the +edits that are live and not yet pushed. So a `GitTarget` names its **applier**, the applier's own +triggers are switched off, and the branch worker sends **apply requests** instead. A request is sent +only on a **quiet branch**, with no open commit window and no pending writes on any worker writing +that repository and branch: + +- for a commit somebody else pushed, after closing open windows and replaying the pending writes + onto it, so the applied commit carries the API side's answer; +- for a refused edit the target opted to revert; +- for Flux only, to keep its artifact current after GitOps Reverser's own pushes, because Flux's + timers and restarts apply whatever artifact it holds. Argo CD is never asked to apply GitOps + Reverser's own commits: with automated sync off it applies nothing on its own, and a sync is pinned + to the commit the worker checked. + +The worker owns each request until it is delivered, and decides every retry the same way. A request +asks and never reports; it runs off the publication path and cannot fail a publication. **Built:** +the Flux `Kustomization` applier. **The aim:** an Argo CD `Application` and a webhook for any other +tool. See [the apply trigger design](design/apply-trigger.md). **Redis/Valkey is optional but advised.** The default configured-author mode runs without it: a plain `helm install` comes up healthy and watches initialize from current state on startup. When an endpoint @@ -659,12 +668,19 @@ older commits is foreign, because the advertisement names a tip other than the l The same quiet-branch rule sends two other requests: `Refusal`, to the refusing target's applier when `spec.onRefusal: Reapply` is set, and `CatchUp`, to every applier on the branch once it has -been quiet for `spec.applier.quietPeriod` after the worker's own pushes. Delivering them is the job -of the dispatcher in `internal/applier`, off the worker's queue: it stamps -`reconcile.fluxcd.io/requestedAt` on the `GitRepository` or the `Kustomization`, retries with -backoff, lets a newer request for the same `Kustomization` supersede one not yet sent, and records an -`ApplyRequested` Event and `status.applier`. A failed request never reaches back into the publication that -caused it. +been quiet for `spec.applier.quietPeriod` after the worker's own pushes (the longest among the +`GitTarget`s that share an applier, since its apply reaches all their folders). + +What each applier is owed lives on the worker, per applier: one attempt out at a time, and its own +retry clock. The dispatcher in `internal/applier` carries out one attempt off the worker's queue and +reports back. It reads each `GitTarget` again first (the same incarnation by uid, still naming this +applier on this branch, still admitted, still consenting to a `Refusal`), checks that the +`Kustomization`'s source fetches this repository and branch, and stamps +`reconcile.fluxcd.io/requestedAt` on the `GitRepository` or the `Kustomization`. It never retries: +an attempt that does not deliver gives the worker back what it carried, a refusal only while it still +stands, and the worker decides the retry on a quiet branch after that applier's backoff. A delivered +request records an `ApplyRequested` Event and `status.applier`. A failed request never reaches back +into the publication that caused it. *** @@ -1639,10 +1655,10 @@ as one funnel and its loss paths as one selector: [Interpreting metrics](interpreting-metrics.md#reading-git_fetches_total). **Apply requests.** `apply_requests_total{applier_kind,reason,outcome}` counts every -request the worker decided to send (`applier_kind` is `kustomization`; `reason` is +attempt the worker decided to send (`applier_kind` is `kustomization`; `reason` is `ForeignCommit`, `Refusal` or `CatchUp`), with `outcome` separating -`sent` from `failed` (handed back to the branch worker), `superseded` (a newer request for the same -applier replaced it before it went out) and `dropped` (no target may still ask). +`sent` from `failed` (the worker decides the retry) and `dropped` (no target may still ask for that +reason). It counts attempts, not obligations. `receiver_deliveries_total{type,outcome}` counts receiver deliveries: `notified`, `unmatched`, `ignored` (a ping, a tag, another event type), `rejected` (bad signature) and `invalid`. The receiver cannot tell a foreign push from an own one; the worker decides that from the @@ -1721,12 +1737,15 @@ Current limitations: triggers are off, so while the operator is down a Git-side push reaches the cluster only through a manual sync, `flux reconcile`, or Flux's long backstop interval. - **The applier must live in the cluster it applies to.** It is looked up in the source cluster - with the `ClusterProvider`'s identity. -- **Only Flux can be an applier.** An Argo CD `Application` keeps its own triggers until it can be - named in `spec.applier`. -- **An apply request cannot reserve time.** An edit made while a requested apply runs is reverted - and mirrored as the applier's commit. The rule confines that to the seconds after a foreign push, - a refusal with `Reapply`, or a quiet period on a Flux branch. + with the `ClusterProvider`'s identity, which rules out a hub Argo CD for now. +- **Only Flux can be an applier yet.** An Argo CD `Application` keeps its own triggers until + `applicationRef` lands. +- **An apply request cannot reserve time.** The request is asynchronous: an edit made while the + requested apply runs, to a field the applier sets, is reverted and mirrored as the applier's + commit. The rule confines that to the seconds after a foreign push, a refusal with `Reapply`, or a + quiet period on a Flux branch; + [the apply trigger design](design/apply-trigger.md#the-apply-window-what-is-still-racy) weighs the + options to shrink it further. - **Audit ingress uses a shared-CA trust boundary.** Named `/audit-webhook/` routes and the annotation-routed shared endpoint both require a client certificate signed by the audit CA, but that certificate is not bound to one provider. This is an accepted privileged-control-plane assumption, not @@ -1744,7 +1763,7 @@ Current limitations: | Package | Role | |---|---| | [api/v1alpha3/](../api/v1alpha3/) | CRD types | -| [internal/applier/](../internal/applier/) | apply requests: the Flux client, the dispatcher that delivers them, and the `ApplierConfigured` check | +| [internal/applier/](../internal/applier/) | apply requests: the Flux client, the dispatcher that carries out each attempt, and the `ApplierConfigured` check | | [cmd/](../cmd/) | operator entry point and server setup | | [internal/auditutil/](../internal/auditutil/) | audit identity, objectRef, and subresource helpers feeding attribution facts | | [internal/controller/](../internal/controller/) | Kubernetes reconcilers | @@ -1790,7 +1809,7 @@ Deeper dives live under [docs/design/](design/): - [Kubernetes API resource catalog](facts/kubernetes-api-resource-catalog.md) - [GitTarget status design](spec/status-conditions-guide.md) - [The support contract: what we edit, what we refuse, and why](design/support-boundary/support-contract.md) -- [Push notification and the apply trigger](design/push-notification-and-reconcile-trigger.md) +- [The apply trigger](design/apply-trigger.md) - [Git credentials interop](finished/git-credentials-interop.md) - [SOPS/age key management](finished/sops-repo-bootstrap-and-key-management-architecture.md) - [Commit signing](commit-signing.md) diff --git a/docs/bi-directional.md b/docs/bi-directional.md index 82bd7a3c..e72fa80b 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -119,8 +119,9 @@ is the [contested-cell rule](#what-happens-when-somebody-else-pushed-first) seen for an object both sides changed, the API value wins. A request asks; it never reports that the apply happened. Requests are sent off the publication -path, retried on their own, and a failed request never fails a publication that succeeded. Read the -applier's own status to see what it applied. +path, and a failed request never fails a publication that succeeded: the worker keeps what the +applier is owed and decides the retry on a quiet branch. Read the applier's own status to see what +it applied. ## Declaring the applier @@ -447,7 +448,7 @@ concurrent writers. | --- | --- | | Two API edits to one field seconds apart end at the later value with no further commit, and Flux catches up with both by itself | [Flux](../test/e2e/flux_bi_directional_e2e_test.go) | | A Git-side push with an API edit on top lands both, and Flux applies the replayed commit without being reconciled by hand | Flux | -| A new `GitRepository` revision is enough for the `Kustomization` to apply and revert drift | Flux | +| A new `GitRepository` revision, from a commit outside the applied folder, is enough for the `Kustomization` to apply and revert drift | Flux | | A refused edit with `onRefusal: Reapply` is reverted by its `Kustomization`, with no commit, and settles instead of repeating | [Refusal reapply](../test/e2e/refusal_reapply_e2e_test.go) | | A `SourceMismatch` applier is never stamped, a push the Git host reports or the refresh finds is applied, without the corner | [Manager Flux Applier](../test/e2e/flux_applier_e2e_test.go), and [the Flux guide](flux.md#what-the-tests-prove) | | A Git to cluster to Git round trip settles without a commit loop | Flux | diff --git a/docs/configuration.md b/docs/configuration.md index e26cb969..82c8bbc9 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1090,10 +1090,11 @@ Git-host configuration. Seconds-fresh is the job of the ### Who applies this folder (`spec.applier`) -When Flux applies this folder back to the cluster, `spec.applier` names the `Kustomization` that does -it, and GitOps Reverser becomes the one that decides when it applies. Flux keeps doing the fetching, -rendering, applying and pruning. The reason it has to be this way, and how to configure Flux for it, -is the [bi-directional guide](bi-directional.md). +When a GitOps tool applies this folder back to the cluster, `spec.applier` names the object that +does it, and GitOps Reverser becomes the one that decides when it applies. The tool keeps doing the +fetching, rendering, applying and pruning. The reason it has to be this way is the +[apply trigger design](design/apply-trigger.md); the [bi-directional guide](bi-directional.md) is +the model, and [Using GitOps Reverser with Flux](flux.md) the whole Flux setup. ```yaml spec: @@ -1109,9 +1110,16 @@ spec: | `kustomizationRef` | **required** | A Flux `Kustomization` whose `sourceRef` is a `GitRepository` on this target's repository and branch | | `quietPeriod` | `30s` | How long the branch must stay quiet after GitOps Reverser's own pushes before a `CatchUp` request. At most `1h` | -Argo CD (`applicationRef`) and a webhook for any other tool are the next appliers; until they -exist, an Argo CD `Application` keeps its own triggers. [Using GitOps Reverser with Flux](flux.md) -walks through the whole Flux setup, from the `GitRepository` to the Git host's webhook. +**The appliers.** A Flux `Kustomization` (`kustomizationRef`) is built. Two more are the aim, and +the API does not accept them yet: + +- **An Argo CD `Application` (`applicationRef`).** The `Application` runs with + `syncPolicy.automated.enabled: false`, and a request sets its sync operation exactly as automated + sync would, pinned to the commit GitOps Reverser checked. It is never asked to catch up with + GitOps Reverser's own commits, so `quietPeriod` will not apply to it. Until it lands, an + `Application` keeps its own triggers. +- **A webhook (`webhook`)** for any other tool: a POST of `{"repository", "branch", "commit", + "reason"}`, signed the way Flux's `generic-hmac` `Receiver` checks. **Where the reference is looked up.** In the source cluster, the cluster the `Kustomization` applies to, with the identity the [`ClusterProvider`](#clusterprovider) already uses. That identity needs `get` @@ -1122,7 +1130,8 @@ cluster, grant the same to the kubeconfig's identity. A `Kustomization` that liv than the one it applies to is not supported. **When a request is made.** Only on a quiet branch: no open commit window and no pending writes on -the branch worker, which is the moment nothing live is missing from the tip. +the branch worker, or on any other worker writing the same repository and branch through another +`GitProvider`. That is the moment nothing live is missing from the tip. | `reason` | When | Sent to | | --- | --- | --- | @@ -1133,7 +1142,9 @@ the branch worker, which is the moment nothing live is missing from the tip. Flux applies on its own timers and on a controller restart, from whatever artifact the `GitRepository` holds. `CatchUp` keeps that artifact at the tip, so such an apply applies values the cluster already holds; an artifact left behind would revert edits that are already in Git. -`quietPeriod` keeps the catch-up out of an editing session: every new write restarts it. +`quietPeriod` keeps the catch-up out of an editing session: every new write restarts it. When several +`GitTarget`s name one `Kustomization`, it waits for the longest of their quiet periods, since its +apply reaches all their folders, and it is asked once for all of them. **What a request does.** What `flux reconcile kustomization --with-source` does: stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository`, and on the `Kustomization` instead when @@ -1144,19 +1155,19 @@ A request asks; it never claims the apply happened. It runs off the publication request never fails a publication that succeeded. Read the `Kustomization`'s `status.lastAppliedRevision` for what it applied. -**Every attempt checks again.** A request is decided once, on a quiet branch, and can be attempted -later. So before every attempt, retries included, GitOps Reverser reads the `GitTarget` again and -asks nothing for it when it no longer names this applier on this branch or its `ClusterProvider` no -longer admits its namespace. A `Refusal` is asked only while the target still consents -(`onRefusal: Reapply`, not suspended); otherwise its applier is asked a `CatchUp`. And nothing is -stamped unless the `Kustomization`'s `GitRepository` fetches this repository and branch, whatever +**Every attempt checks again.** Just before it patches, GitOps Reverser reads each `GitTarget` again +and asks nothing for one that has been deleted or recreated, no longer names this applier on this +branch, is no longer admitted by its `ClusterProvider`, or, for a `Refusal`, no longer consents +(`onRefusal: Reapply`, not suspended). It never asks for something weaker in place of what it was +told; when no target still holds the request's reason it asks nothing. And nothing is stamped unless +the `Kustomization`'s `GitRepository` fetches this repository and branch, whatever `ApplierConfigured` last said. -**A request that fails** is retried for about one `quietPeriod`, or not at all when the -`Kustomization` cannot take a request, and then goes back to the branch worker, which asks again -once the branch is quiet, waiting longer each time up to a few minutes. It is not retried for -longer than that because a retry skips the quiet check: Flux applies whatever the tip is when the -retry lands, and by then live edits may be waiting to be published. +**A request that fails is decided again from scratch.** The branch worker keeps what the +applier is owed until a request delivers it. After a failure it waits out that applier's own backoff +(2 seconds doubling to 5 minutes, at least 15 seconds when the `Kustomization` cannot take a +request), then waits for a quiet branch like any first request, and asks for the tip as it is then. +A refusal that recovered in the meantime is not asked for again. **What it reports.** diff --git a/docs/design/apply-trigger.md b/docs/design/apply-trigger.md index 109843e9..b78efd76 100644 --- a/docs/design/apply-trigger.md +++ b/docs/design/apply-trigger.md @@ -57,11 +57,11 @@ already in Git. **Not every commit GitOps Reverser authors is a no-op to apply.** A replayed `F + W` carries somebody else's `F`, and a refusal needs an apply to be reverted. So the decision is state on the -worker ("this applier is owed an apply"), set by observations and cleared by a delivered request, -not a verdict on each push notification. A notification is a hint: the worker reads the remote and +worker ("this applier is owed an apply"), set by observations and cleared by a delivered request. +A push notification decides nothing by itself; it is a hint: the worker reads the remote and judges from that, which gets late, duplicate and rolled-back deliveries right. -**A decision has to hold when the patch is sent, not only when it is made.** The first review found +**A decision has to hold when the patch is sent as well as when it is made.** The first review found that a request decided once was attempted for minutes against whatever the cluster had become: a target its `ClusterProvider` no longer admitted still drove the applier, and a source that followed another branch was still stamped. Every attempt now reads each target again, and the Flux client @@ -152,9 +152,10 @@ What would make the window smaller and visible, in order of cost: is the case that can be reverted. Recording it (an Event on the `GitTarget`, a counter) turns a silent revert into a visible one. 3. **Let the API win the window.** A write by the applier's own field manager during an attempt we - made, that undoes a field edited after that attempt was sent, is our apply arriving late, not new - intent. Not mirroring it keeps the edit in Git, and the next request applies it back. This is the - rule "for an object both sides changed, the API value wins", applied to the apply window. It needs + made, that undoes a field edited after that attempt was sent, is our apply arriving late, with no + new intent behind it. Not mirroring it keeps the edit in Git, and the next request applies it + back. This is the rule "for an object both sides changed, the API value wins", applied to the + apply window. It needs the field manager from `managedFields` (no audit), and it is the only option that changes what reaches Git, so it needs its own decision. @@ -168,7 +169,7 @@ consequences: - **No `CatchUp`.** With automated sync off Argo CD applies nothing on its own, so there is no stale artifact to catch up, and every sync runs the `Application`'s hooks. -- **The commit is pinned**, so a sync applies what the worker checked, not whatever the tip became. +- **The commit is pinned**, so a sync applies exactly what the worker checked. - **A running operation is waited for, never replaced**, as Argo CD's own `SetAppOperation` does. The open question that shapes its API: a hub Argo CD keeps `Application`s in a management cluster, @@ -196,7 +197,7 @@ in-process plugin: it would run with every credential the operator holds. | --- | --- | | The decision: quiet branch, reasons, quiet period, baseline | [`apply_trigger_test.go`](../../internal/git/apply_trigger_test.go) | | Retries are decided by the worker; recovered refusals, withdrawn consent and recreated targets are not retried; each applier keeps its own clock; one attempt for a shared applier; a busy sibling worker holds the branch | [`apply_attempt_test.go`](../../internal/git/apply_attempt_test.go) | -| Every attempt rechecks targets, never asks for something weaker, and never stamps a source on another branch | [`dispatcher_test.go`](../../internal/applier/dispatcher_test.go), [`flux_test.go`](../../internal/applier/flux_test.go) | +| Every attempt rechecks targets, asks for nothing weaker than it was told, and stamps no source on another branch | [`dispatcher_test.go`](../../internal/applier/dispatcher_test.go), [`flux_test.go`](../../internal/applier/flux_test.go) | | The original race: two API edits end at the later value with no extra commit | [bi-directional Flux corner](../../test/e2e/flux_bi_directional_e2e_test.go) | | `SourceMismatch` is never stamped and is delivered once fixed; a Gitea webhook and the refresh alone each get a push applied | [Manager Flux Applier](../../test/e2e/flux_applier_e2e_test.go) | | A refused edit is reverted by its `Kustomization` | [Refusal reapply](../../test/e2e/refusal_reapply_e2e_test.go) | diff --git a/docs/design/branch-worker-event-model.md b/docs/design/branch-worker-event-model.md index 6d55bd8f..b4415873 100644 --- a/docs/design/branch-worker-event-model.md +++ b/docs/design/branch-worker-event-model.md @@ -410,7 +410,7 @@ Keep timers as wakeup mechanisms and persist the deadlines that give those wakeu Recording every periodic tick would add history without explaining a decision. Record meaningful scheduling, cancellation, and accepted firings instead. -The current worker has five timer sources: +The current worker has six timer sources: | Deadline | Existing purpose | |---|---| @@ -418,7 +418,8 @@ The current worker has five timer sources: | Push timer | Space successful publication cycles | | Retry | Retry a failed publication on its backoff, or probe for a missing parent | | Attach timeout | Settle a save still waiting for an eligible window | -| Refusal action | Recheck consent before an eligible empty commit | +| Refusal action | Recheck consent before a coalesced Refusal request is owed | +| Applier | Send an apply attempt that came due on a quiet branch | The retry schedule has its own timer; the push timer is only the success cooldown. Parent recovery has no timer of its own since #413: while its obligation is open, the retry deadline is @@ -571,6 +572,6 @@ The detailed design records continue to own adjacent policy: - [Configuration freshness](gittarget-configuration-freshness.md): watch-plan application and publication are separate observations. This design does not require new public freshness fields. - [API boundaries](gittarget-api-wave.md) and - [push notifications](push-notification-and-reconcile-trigger.md): a remote-change notification + [push notifications](apply-trigger.md): a remote-change notification requests work through the owner. It does not become another checkout writer or prove that Git changes reached the source cluster. diff --git a/docs/design/event-pipeline-overview.md b/docs/design/event-pipeline-overview.md index 4a2acea8..4e91b53f 100644 --- a/docs/design/event-pipeline-overview.md +++ b/docs/design/event-pipeline-overview.md @@ -295,7 +295,7 @@ replays current state. ## 5. Pending writes: decide, materialize, publish Every write takes one path, whatever produced it: a closed window, a resync, a save's empty -record, a refusal's empty commit, or an atomic batch. +record, or an atomic batch. ```mermaid stateDiagram-v2 @@ -522,8 +522,7 @@ remains: - **A healthy branch's FIFO is bounded by count only.** The budget applies while a retry is pending; until then, up to 1,000 queued items cost memory on top of the window and the pending writes. By design: a healthy branch drains. -- **Work the loop derives can overshoot.** A refusal's empty commit is decided from work already - accepted, and work accepted before the outage began is kept, so the held total can pass the +- **Work accepted before the outage can overshoot.** It is kept, so the held total can pass the budget by that much, and no further. ### Status and observability diff --git a/docs/design/gittarget-branch-worker-pending-writes.md b/docs/design/gittarget-branch-worker-pending-writes.md index 07d2f41b..999150fb 100644 --- a/docs/design/gittarget-branch-worker-pending-writes.md +++ b/docs/design/gittarget-branch-worker-pending-writes.md @@ -216,16 +216,14 @@ itself is terminal. Pinned by `TestDecidedWrite_SurvivesAFailedRebuildAndLandsTh `pcr.committed` reused to mean "decided", and an executor that re-entered itself through refusal handling. This step closes them, in [`pending_writes_loop.go`](../../internal/git/pending_writes_loop.go): -- Every write kind (window, atomic batch, resync, a save's empty record, a refusal's empty commit) - is decided into the pending writes and committed by `materialize`. `l.commit`, `retain` and the commit guard - are gone, with the separate resync, atomic and refusal-touch commit paths. +- Every write kind (window, atomic batch, resync, a save's empty record) is decided into the + pending writes and committed by `materialize`. `l.commit`, `retain` and the commit guard are gone, + with the separate resync and atomic commit paths. - One place classifies an attempt: committed (`settleCommitted`), failed for good (`settleFailed`, terminal for that write), or kept pending for the retry, with any waiting resync caller answered - (`replyToDeferredResyncs`). A write's origin (the resync caller, the atomic request, - the refusal a touch answers) rides with it so its outcome can be settled there. -- `materialize` is never re-entered. A refusal's empty commit decided while a refused write is being - settled is appended to the pending writes, and the running pass commits it in order. No push starts inside a - pass. + (`replyToDeferredResyncs`). A write's origin (the resync caller, the atomic request) rides + with it so its outcome can be settled there. +- `materialize` is never re-entered, and no push starts inside a pass. - Every commit re-reads its write's prune policy, as a replay does (`TestDecidedWrite_ADeferredDeleteObeysATightenedPrunePolicy`, a review finding). The read comes from the informer cache. @@ -383,8 +381,7 @@ producer wait. The intake gate is [`intake.go`](../../internal/git/intake.go). beyond its payload and message, so the byte budget also bounds the item count, empty records included. The producer sizes each event and snapshot once, off the loop, and the loop reuses the size. The charge is serialized YAML, not heap: a queued object is held as an unstructured map of - about six times that. Overshoot is limited to work accepted before the outage began and to work - the loop derives from accepted work, such as a refusal's empty commit. + about six times that. Overshoot is limited to work accepted before the outage began. - **Pause is a latch.** A payload that would cross the budget is refused and pauses the branch; every later payload is refused until the retry clears, which means a publication landed or nothing is owed. Room under the budget again, a partial replay, or a remote that can be read but diff --git a/docs/design/gittarget-parent-branch.md b/docs/design/gittarget-parent-branch.md index bbf02c0b..d891bf59 100644 --- a/docs/design/gittarget-parent-branch.md +++ b/docs/design/gittarget-parent-branch.md @@ -5,7 +5,7 @@ > records the contract as built and the reasoning behind it. Deferred: > [`gittarget-parent-empty-repository.md`](gittarget-parent-empty-repository.md) and > [`gittarget-parent-observation.md`](gittarget-parent-observation.md). -> Related: [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md), +> Related: [`push-notification-and-reconcile-trigger.md`](../finished/push-notification-and-reconcile-trigger.md), > [`../definitions.md`](../definitions.md), and `architecture.md` (Ground rules; "A branch that does > not exist yet"). diff --git a/docs/design/push-cooldown.md b/docs/design/push-cooldown.md index 690aa10d..7dd8940b 100644 --- a/docs/design/push-cooldown.md +++ b/docs/design/push-cooldown.md @@ -5,7 +5,7 @@ > unchanged. Option D needs measurement; option E remains rejected without that evidence. > Index: [`../INDEX.md`](../INDEX.md). Reviewed at `373bf8d7` on 2026-10-03. > Related: [`../api-first-publication.md`](../api-first-publication.md), -> [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md), +> [`push-notification-and-reconcile-trigger.md`](../finished/push-notification-and-reconcile-trigger.md), > [`../spec/commit-window-refactor.md`](../spec/commit-window-refactor.md) **The question.** `PushCooldown` predates the commit window. The window now groups an editing burst @@ -269,8 +269,8 @@ Ordered by how much they should worry you. 1. **Request amplification on shared branches.** The architecture explicitly supports several `GitTarget`s per branch, and multi-tenant installs are the case where the cooldown is load-bearing. Option B turns row 4's `2` into `6`. This contradicts the premise of - [`push-notification-and-reconcile-trigger.md`](push-notification-and-reconcile-trigger.md), which spent a whole change - removing two requests per publication. + [`push-notification-and-reconcile-trigger.md`](../finished/push-notification-and-reconcile-trigger.md), + which spent a whole change removing two requests per publication. 2. **Hosted Git rate limits.** With serialization the bound is one push per push-duration, which on a fast remote is several per second sustained. Measure throttling on the supported hosts before increasing publication frequency. diff --git a/docs/design/reconcile-triggering.md b/docs/design/reconcile-triggering.md index 20e770bc..81a0e717 100644 --- a/docs/design/reconcile-triggering.md +++ b/docs/design/reconcile-triggering.md @@ -280,8 +280,8 @@ flowchart LR ``` > **The middle step of that flow has since been ruled out, and the rest of this section stands.** -> [inbound push notification](push-notification-and-reconcile-trigger.md) §7 shows that `requestedAt` drives a -> full cluster-to-Git snapshot, so a receiver that patches it can publish pre-push cluster state +> [inbound push notification](../finished/push-notification-and-reconcile-trigger.md) §7 shows that +> `requestedAt` drives a full cluster-to-Git snapshot, so a receiver that patches it can publish pre-push cluster state > over the very change it was told about, before Flux or Argo CD has applied it. The receiver needs > a request shape of its own: invalidate the base, record the new tip, replay retained work, and > never snapshot. §8.3 of that page is the wire contract. Read everything below (the signature diff --git a/docs/design/support-boundary/README.md b/docs/design/support-boundary/README.md index 55cad2be..bb4d51d7 100644 --- a/docs/design/support-boundary/README.md +++ b/docs/design/support-boundary/README.md @@ -85,7 +85,7 @@ distinguishes those planned gates from the permanent construct boundary. the staged design for safely editing it. - [Git write preflight](../git-write-preflight.md): onboarding refusal and all-target admission checks, with target errors for late failures. This replaces the per-edit accounting proposal. -- [Admission consent](admission-consent.md) and [reconcile trigger](../push-notification-and-reconcile-trigger.md#option-6-trigger-a-reconcile-to-revert-a-refused-edit): +- [Admission consent](admission-consent.md) and [reconcile trigger](../../finished/push-notification-and-reconcile-trigger.md): deferred extensions; preflight does not promise automatic rollback. - Historical implementation records: [image/replica edit-through](finished/images-and-replicas-edit-through.md) and [higher-level KRM support](finished/higher-level-krm-documents.md). diff --git a/docs/design/support-boundary/admission-consent.md b/docs/design/support-boundary/admission-consent.md index d1a6ad99..e4893835 100644 --- a/docs/design/support-boundary/admission-consent.md +++ b/docs/design/support-boundary/admission-consent.md @@ -10,13 +10,13 @@ > [gittarget-granularity-and-cross-environment-edits.md](gittarget-granularity-and-cross-environment-edits.md) > — the write boundary; fan-in = 1; base read-only by L1, > [render-attribution.md](render-attribution.md) §5 — attribution may be heuristic, verification may not, -> [the reconcile trigger options](../push-notification-and-reconcile-trigger.md#part-3-the-options) +> [the reconcile trigger options](../../finished/push-notification-and-reconcile-trigger.md) > — **the sibling half: what reverts a refusal that lands anyway**, > [support-contract.md](support-contract.md) This is one half of a two-part design. This half is about turning a *refusal* into a *yes* at the moment of the edit; the other half — -[the reconcile trigger](../push-notification-and-reconcile-trigger.md#option-6-trigger-a-reconcile-to-revert-a-refused-edit) +[the reconcile trigger](../../finished/push-notification-and-reconcile-trigger.md) — is about making the *outcome* of a "no" prompt and visible. They compose, but they are separate topics and separate documents. @@ -195,7 +195,7 @@ Two limits retained from the preflight design: flush-time oracle re-verifies the *whole batch* against the declared intents — including the consented ones — and still refuses if the consented set does not actually converge. So a stale or mistaken admission-time "yes" cannot cause a bad write. Worst case it lands, the flush refuses it, - and the affected target enters error. The [reconcile trigger](../push-notification-and-reconcile-trigger.md#option-6-trigger-a-reconcile-to-revert-a-refused-edit) + and the affected target enters error. The [reconcile trigger](../../finished/push-notification-and-reconcile-trigger.md) is a separate proposed recovery mechanism; automatic rollback is not part of preflight. --- diff --git a/docs/design/support-boundary/render-fidelity.md b/docs/design/support-boundary/render-fidelity.md index c84da337..42d32cf6 100644 --- a/docs/design/support-boundary/render-fidelity.md +++ b/docs/design/support-boundary/render-fidelity.md @@ -356,7 +356,7 @@ divergence, and an incoming Git repair does not start one yet. The missing transition is intentional and visible: the controller does **not** yet observe an incoming Git revision, refresh the source, and begin a complete epoch. Therefore a human Git repair cannot by itself reopen a false gate. Adding that transition requires the retained-intent ordering barrier in -[the reconcile trigger options](../push-notification-and-reconcile-trigger.md#part-3-the-options), so a source refresh cannot +[the reconcile trigger options](../../finished/push-notification-and-reconcile-trigger.md), so a source refresh cannot discard or race an open live-edit window. The enforcement check already belongs in the branch worker, not status projection: the worker rejects a `WriteRequest` before it opens a commit window. The controller projects the same state as `Ready=False` / `Stalled=True` for `False`, and as @@ -450,7 +450,7 @@ It is recomputable only when a complete fresh epoch begins. Today that happens w installed or their scope set is replaced. Removing postBuild configuration or changing the source in Git does **not** yet start that epoch automatically, so it does not automatically reopen writes. The planned remote-revision transition must refresh safely behind the barrier in -[the reconcile trigger options](../push-notification-and-reconcile-trigger.md#part-3-the-options); +[the reconcile trigger options](../../finished/push-notification-and-reconcile-trigger.md); once that exists, a full clean replay can reopen the gate without manual acknowledgement. --- diff --git a/docs/finished/push-notification-and-reconcile-trigger.md b/docs/finished/push-notification-and-reconcile-trigger.md index 01db9456..035d5833 100644 --- a/docs/finished/push-notification-and-reconcile-trigger.md +++ b/docs/finished/push-notification-and-reconcile-trigger.md @@ -1,10 +1,9 @@ # Push notification and the apply trigger -> **design**: decided 2026-10-16; the Flux applier, the receiver and `Reapply` are built, Argo CD -> and the webhook applier are not. Index: [`../INDEX.md`](../INDEX.md) -> User-facing model: [`../bi-directional.md`](../bi-directional.md). Fields: -> [`../configuration.md`](../configuration.md#who-applies-this-folder-specapplier). The request that -> led here: [`bi-directional-right.md`](bi-directional-right.md). +> **finished**: superseded on 2026-10-16 by [`../design/apply-trigger.md`](../design/apply-trigger.md), +> which is the current design and what is still open. This page is the record of how the apply +> trigger was decided. Its Part 1 (how the branch worker reads the remote) is still cited from the +> Git code by its section numbers, which predate the last restructuring of this page. GitOps Reverser and the applier (Flux or Argo CD) share a branch. This page is how they talk: how GitOps Reverser learns that somebody else pushed, and how it decides when the applier applies. @@ -65,7 +64,7 @@ branch on the worker's authority. Any push error or rejection clears it. `checko how many pending writes the checkout holds on top of its root; unknown means a failed write left the tree dirty, and a count below the pending writes means a reset discarded their commits and they must be replayed. The loop's `materialize` is the one place that acts on it -([`gittarget-branch-worker-pending-writes.md`](gittarget-branch-worker-pending-writes.md)). +([`gittarget-branch-worker-pending-writes.md`](../design/gittarget-branch-worker-pending-writes.md)). A cycle plans straight onto the worktree only when the base is trusted and the tree is clean; otherwise it fetches, resets and replays first. diff --git a/docs/flux.md b/docs/flux.md index 479ca261..52f37a43 100644 --- a/docs/flux.md +++ b/docs/flux.md @@ -163,7 +163,11 @@ Reverser is down; a short one is a source that fetches, and so applies, on its o `GitProvider`'s repository and branch, so naming one elsewhere grants nothing that write access to the repository does not already give. - Several `GitTarget`s may name one `Kustomization`, for example one whose `path` covers several - folders. It is asked once per request, however many of them share it. + folders. It is asked once for all of them, each with its own reason, and waits for the longest of + their `quietPeriod`s, because its apply reaches every one of their folders. +- Two `GitProvider`s may name one repository, for example one per team. Each gets its own branch + worker, and the branch counts as quiet only when both are: a `Kustomization` applies the tip + whichever of them wrote it. ## The `GitTarget` @@ -248,14 +252,24 @@ already holds it, it stamps the `Kustomization` instead, and a `CatchUp` stamps **It is asynchronous.** The patch returns as soon as the API server accepts it, and nothing waits for Flux to fetch or apply. It runs off the branch worker, so a slow source cluster never holds a -publication, and a failed request never fails one. - -Before every attempt, retries included, GitOps Reverser reads the `GitTarget` again: it must still -name this `Kustomization` on this branch, its `ClusterProvider` must still admit it, and a -`Refusal` must still have consent. A request that fails is retried for about one `quietPeriod`, -because a retry skips the quiet check, and is then handed back to the branch worker, which asks -again once the branch is quiet, waiting longer each time up to five minutes. A failure that cannot -be retried, such as a `SourceMismatch`, is handed back at once and delivered once it is fixed. +publication, and a failed request never fails one. That leaves a window of a few seconds while Flux +applies: an API edit made then, to a field the `Kustomization` sets, is reverted by the apply. The +quiet-branch rule confines the window to the moments listed above; waiting for the apply would not +close it, because waiting does not stop anybody editing. +[The apply trigger design](design/apply-trigger.md#the-apply-window-what-is-still-racy) weighs the +ways to shrink it. + +Before it patches, GitOps Reverser reads each `GitTarget` again: it must still be the same +object (a deleted and recreated one inherits nothing), still name this `Kustomization` on this +branch, still be admitted by its `ClusterProvider`, and, for a `Refusal`, still consent. When none +still holds the request's reason it asks nothing; it does not substitute a weaker request. + +**A request that fails is decided again from scratch.** GitOps Reverser keeps what each +`Kustomization` is owed until a request delivers it. After a failure, that `Kustomization` waits out +its own backoff (2 seconds doubling to 5 minutes, at least 15 seconds after a `SourceMismatch`), +then waits for a quiet branch like any first request, and asks for the tip as it is then. So a +`SourceMismatch` is delivered once it is fixed, and a refusal that recovered meanwhile is not asked +for again. ## Reverting refused edits @@ -297,7 +311,7 @@ kubectl get kustomization checkout -n flux-system -o jsonpath='{.status.lastAppl | `ApplierConfigured=False`, `ApplierNotFound` | The `Kustomization` does not exist in the source cluster under that namespace and name, or Flux is not installed there | Create it, or fix `kustomizationRef`; an omitted namespace is the `GitTarget`'s | | `ApplierConfigured=False`, `SourceMismatch` | The message names the broken [`GitRepository` rule](#the-gitrepository) | Fix the `GitRepository`; the request it was holding is delivered afterward | | `ApplierConfigured=False`, `Forbidden` | The identity lacks one of the [permissions](#permissions) | Grant it, or set `applierAccess.flux: true` | -| A Warning `ApplyRequestFailed` Event | A request could not be delivered; the message says why | GitOps Reverser keeps asking on a quiet branch, at most every five minutes. It is recorded once per failing request | +| A Warning `ApplyRequestFailed` Event | A request could not be delivered; the message says why | GitOps Reverser keeps asking on a quiet branch, at most every five minutes. It is recorded for the first failure of a run | | Flux applies while edits are in flight | Something else still makes the source fetch: a short `interval`, a Flux `Receiver`, or someone running `flux reconcile` | Lengthen the intervals and remove the Flux `Receiver` | | A Git-side push takes minutes to reach the cluster | No receiver, so the refresh finds it | Configure [the webhook](#the-git-hosts-webhook) | | A refused edit is not reverted | The target is suspended, `onRefusal` is not `Reapply`, or the refusal is not eligible | See [reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal) | @@ -307,7 +321,7 @@ kubectl get kustomization checkout -n flux-system -o jsonpath='{.status.lastAppl GitOps Reverser becomes the only fast path to a deployment. While it is down, a Git-side push reaches the cluster through `flux reconcile kustomization --with-source`, which is safe then because nothing is publishing, or through the 24h backstop. When it starts again, its first look at -the remote owes every `Kustomization` a `CatchUp`, never a `ForeignCommit`: an apply on startup +the remote owes every `Kustomization` a `CatchUp` and no `ForeignCommit`: an apply on startup could revert edits made while it was down, before its watches have caught them. ## What the tests prove @@ -321,9 +335,10 @@ could revert edits made while it was down, before its watches have caught them. | With no receiver and no API edit, the refresh finds a Git-side push and Flux applies it | Manager Flux Applier | | Two API edits seconds apart end at the later value with no extra commit, and Flux catches up by itself | [Bi-directional (Flux)](../test/e2e/flux_bi_directional_e2e_test.go) | | A Git-side push with an API edit on top lands both, and Flux applies the replayed commit | Bi-directional (Flux) | -| A new `GitRepository` revision alone makes the `Kustomization` apply, which is why the source must not fetch on its own | Bi-directional (Flux) | +| A new `GitRepository` revision alone, from a commit outside the applied folder, makes the `Kustomization` apply, which is why the source must not fetch on its own | Bi-directional (Flux) | | A refused edit with `onRefusal: Reapply` is reverted by its `Kustomization`, with no commit, and settles | [Refusal reapply](../test/e2e/refusal_reapply_e2e_test.go) | -| Every attempt checks the target again (admission, branch, consent) and the source binding before it stamps; a failed request goes back to the branch worker, which asks again | [`dispatcher_test.go`](../internal/applier/dispatcher_test.go), [`apply_handback_test.go`](../internal/git/apply_handback_test.go) | +| Every attempt checks the target again (incarnation, branch, admission, consent) and the source binding before it stamps, and never asks for something weaker | [`dispatcher_test.go`](../internal/applier/dispatcher_test.go) | +| A failed request is decided again on a quiet branch; a recovered refusal, withdrawn consent or recreated target is not retried; each `Kustomization` keeps its own backoff; a shared one is asked once; a sibling worker on the same repository holds the branch | [`apply_attempt_test.go`](../internal/git/apply_attempt_test.go) | `task test-e2e` runs the Manager specs, and `task test-e2e-bi-directional` runs the bi-directional corner. Not covered end to end yet: a remote source cluster, `Forbidden` against cluster RBAC, and the diff --git a/docs/future/git-api-three-way-comparison.md b/docs/future/git-api-three-way-comparison.md index 14003681..2bb23618 100644 --- a/docs/future/git-api-three-way-comparison.md +++ b/docs/future/git-api-three-way-comparison.md @@ -479,7 +479,7 @@ needed for those forms, but neither every patch nor every update reveals an exac ## Why faster Git delivery helps but does not settle the merge Faster notification reduces how long the engine plans against stale Git state. The -[inbound notification proposal](../design/push-notification-and-reconcile-trigger.md) can improve +[apply trigger design](../design/apply-trigger.md) can improve that latency independently of this investigation. The timestamp example already gives Reverser the Git change before Alice edits. The remaining gap diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index dc8ffd35..e2c6fd17 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -822,7 +822,7 @@ and the push receiver tells it that somebody else pushed. See | Metric | Type | Labels | Meaning | | --- | --- | --- | --- | -| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per applier per request the branch worker decided to send. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not be delivered and handed back to the branch worker, which asks again once the branch is quiet; one per hand-back), `superseded` (a newer request for the same `Kustomization` replaced it before it went out) or `dropped` (no `GitTarget` may still ask: each was deleted, moved to another branch or applier, or is no longer admitted by its `ClusterProvider`) | +| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per attempt to one applier. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not ask, including a `Kustomization` that applies another repository or branch; the branch worker keeps the obligation and decides the retry on a quiet branch, so a failure that persists keeps counting at its backoff) or `dropped` (no `GitTarget` may still ask for that reason: each was deleted or recreated, moved to another branch or applier, is no longer admitted by its `ClusterProvider`, or withdrew its consent to a `Refusal`) | | `receiver_deliveries_total` | counter | `type`, `outcome` | One per verified-or-refused delivery to `/git-push/`. `type` is the `GitProvider.spec.receiver.type`; `outcome` is `notified` (handed to the branch worker), `unmatched` (no worker for that branch), `ignored` (a ping, a tag, another branch's event type, or a `generic-hmac` body naming another repository), `rejected` (bad signature or an oversized body) or `invalid` (verified but unparseable) | A delivery counted `notified` is only a hint: the worker reads the advertisement, and fetches under diff --git a/docs/spec/commitrequest-design.md b/docs/spec/commitrequest-design.md index bddcc21b..cb17dd2c 100644 --- a/docs/spec/commitrequest-design.md +++ b/docs/spec/commitrequest-design.md @@ -130,7 +130,7 @@ for a push to say. So is `NoWindow` under `Resolve`. Under `CommitEmpty` a `NoWi its message in an empty commit, and it resolves when that commit reaches the remote.) A window that produced no diff used to resolve `AlreadyPresent` at finalize, on the strength of the local plan. That was only sound while every cycle fetched before it planned. It no longer does (see -[inbound push notification](../design/push-notification-and-reconcile-trigger.md) §3), so the plan may have run +[inbound push notification](../design/apply-trigger.md)), so the plan may have run against a tree the remote has moved past, and the replay that follows a rejected push can turn the same captured object into a commit. "Already present" is a claim about the remote, so the remote settles it: a no-diff request resolves `AlreadyPresent` when the push confirms there was From 84a76c9a531d9e4af56fb611e89acc8514c7d256 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 21:27:54 +0000 Subject: [PATCH 20/36] fix(applier): never stamp an older commit over a newer push An attempt decided at one commit could land after the worker pushed a newer one and, finding the artifact at the older commit, stamp the Kustomization, re-applying it over a published edit. A request now carries a check the dispatcher makes just before it stamps: the commit is still the branch tip and nothing is waiting to be published, on this worker or a sibling. When it fails nothing is stamped (Stale), and the worker decides again at the next quiet moment, with no backoff. A ForeignCommit whose artifact already holds the commit stamps nothing: that artifact already woke the Kustomization. Only a Refusal stamps the Kustomization, to revert drift at an unchanged commit. Co-Authored-By: Claude Opus 5.5 (1M context) --- .coverage-baseline | 2 +- internal/applier/dispatcher.go | 11 ++-- internal/applier/dispatcher_test.go | 16 ++++++ internal/applier/flux.go | 26 +++++++--- internal/applier/flux_guide_test.go | 2 +- internal/applier/flux_test.go | 33 ++++++++++-- internal/git/apply_attempt_test.go | 76 ++++++++++++++++++++++++++++ internal/git/apply_request.go | 10 ++++ internal/git/apply_trigger.go | 27 +++++++++- internal/telemetry/apply_requests.go | 3 ++ 10 files changed, 187 insertions(+), 19 deletions(-) diff --git a/.coverage-baseline b/.coverage-baseline index c2add760..073f80d7 100644 --- a/.coverage-baseline +++ b/.coverage-baseline @@ -1 +1 @@ -79.9 +80.0 diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index 374e1787..a6f28b5c 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -44,6 +44,7 @@ const ( outcomeSent = "sent" outcomeFailed = "failed" outcomeDropped = "dropped" + outcomeStale = "stale" ) // shortCommitLength is how much of a commit an Event names. @@ -61,8 +62,8 @@ type SourceClients func(ctx context.Context, clusterID string) (dynamic.Interfac const DefaultAttemptTimeout = 30 * time.Second // Dispatcher carries out the apply requests the branch workers decide. It implements -// git.ApplyRequester, and runs as a leader-elected manager.Runnable, the same replica the branch -// workers run on. +// git.ApplyRequester, and runs as a manager.Runnable beside the branch workers (it asks for leader +// election as they do, which keeps the two together should it ever be switched on). // // It keeps no queue and makes no decision of its own. Each request is one attempt for one applier, // carried out on its own goroutine and reported back to the worker that decided it; the worker owns @@ -219,8 +220,12 @@ func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.Appl } flux := NewFlux(sourceClient) flux.Now = d.Now - outcome, err := flux.Request(ctx, applierName(req.Applier), want, req.Commit, req.Reason) + outcome, err := flux.Request(ctx, applierName(req.Applier), want, req.Commit, req.Reason, req.Current) switch { + case errors.Is(err, errStale): + d.count(ctx, req.Reason, outcomeStale) + log.Info("the decision no longer holds; asking nothing, the branch worker decides again") + return report(git.ApplyStale, nil) case isPermanent(err): return d.failure(ctx, req, eligible, report(git.ApplyRejected, err), log) case err != nil: diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index e3c3d6ec..ab93f6e0 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -360,6 +360,22 @@ func TestDispatcher_ATargetThatChangedAsksNothing(t *testing.T) { } } +// TestDispatcher_AStaleDecisionAsksNothing: the worker's decision is checked again just before the +// patch, and an attempt whose branch moved on or became busy reports Stale and stamps nothing. +func TestDispatcher_AStaleDecisionAsksNothing(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{reapplying(gitTarget("a", checkoutRef(), ""))}, + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+testCommit), + ) + req := request(checkout, v1alpha3.ApplyReasonRefusal, "a") + req.Current = func() bool { return false } + + assert.Equal(t, git.ApplyStale, f.attempt(req).Outcome) + assert.Empty(t, f.patches()) + assert.Empty(t, f.events(), "a stale decision is no failure") + assert.Equal(t, int64(1), f.count(v1alpha3.ApplyReasonRefusal, outcomeStale)) +} + // TestDispatcher_CatchUpAlreadyCurrent: a CatchUp the applier already satisfies counts as sent and // records nothing, since nothing was asked of it. func TestDispatcher_CatchUpAlreadyCurrent(t *testing.T) { diff --git a/internal/applier/flux.go b/internal/applier/flux.go index b0f81a11..60656798 100644 --- a/internal/applier/flux.go +++ b/internal/applier/flux.go @@ -74,8 +74,8 @@ const ( // OutcomeStampedKustomization stamped the Kustomization: the artifact already held the commit, // so a source stamp would produce no new artifact and wake nothing. OutcomeStampedKustomization Outcome = "StampedKustomization" - // OutcomeCurrent stamped nothing: a CatchUp found the artifact already at the commit, which is - // all a CatchUp asks for. + // OutcomeCurrent stamped nothing: a CatchUp or ForeignCommit found the artifact already at the + // commit, so the Kustomization was already woken by it. OutcomeCurrent Outcome = "Current" ) @@ -146,6 +146,9 @@ type permanentError struct{ err error } func (e permanentError) Error() string { return e.err.Error() } func (e permanentError) Unwrap() error { return e.err } +// errStale reports a request whose decision no longer held when the stamp was due. +var errStale = errors.New("the decision no longer holds: the branch moved on or became busy") + // isPermanent reports whether err is a permanentError. func isPermanent(err error) bool { var p permanentError @@ -162,16 +165,22 @@ func isPermanent(err error) bool { // // When the GitRepository's artifact does not hold commit, it stamps the GitRepository: the fetch // produces a new artifact and the new artifact wakes the Kustomization by itself, so a second stamp -// would only be a second apply. When the artifact already holds it, a source stamp would wake -// nothing, so it stamps the Kustomization instead, which is what a Refusal at an unchanged commit -// needs; a CatchUp needs nothing then, because a CatchUp only keeps the artifact current. An empty -// commit (the remote does not carry the branch) stamps the GitRepository, so Flux finds out itself. +// would only be a second apply. When the artifact already holds it, only a Refusal stamps the +// Kustomization, because only a Refusal asks for an apply of an unchanged commit (to revert drift). +// A ForeignCommit or CatchUp then needs nothing: the artifact that holds the commit already woke the +// Kustomization when it arrived. An empty commit (the remote does not carry the branch) stamps the +// GitRepository, so Flux finds out itself. +// +// current is asked just before the stamp. When it reports false the decision no longer holds (the +// branch moved on, or became busy) and nothing is stamped: errStale. Stamping the Kustomization of +// an artifact holding an older commit would re-apply that commit over an edit published since. func (f *Flux) Request( ctx context.Context, ref types.NamespacedName, want Binding, commit string, reason v1alpha3.ApplyReason, + current func() bool, ) (Outcome, error) { ks, err := f.client.Resource(kustomizationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) if err != nil { @@ -190,8 +199,11 @@ func (f *Flux) Request( return "", permanentError{fmt.Errorf("%s %s: %s", kindKustomization, ref, msg)} } + if current != nil && !current() { + return "", errStale + } if commit != "" && artifactHolds(source, commit) { - if reason == v1alpha3.ApplyReasonCatchUp { + if reason != v1alpha3.ApplyReasonRefusal { return OutcomeCurrent, nil } if err := f.stamp(ctx, kustomizationGVR(), ref); err != nil { diff --git a/internal/applier/flux_guide_test.go b/internal/applier/flux_guide_test.go index 7a8fbf24..369c63dd 100644 --- a/internal/applier/flux_guide_test.go +++ b/internal/applier/flux_guide_test.go @@ -89,7 +89,7 @@ func TestFluxGuide_TheCompleteExampleIsAccepted(t *testing.T) { verdict := flux.Validate(context.Background(), ref, want) assert.True(t, verdict.OK, "ApplierConfigured for the guide's example: %s: %s", verdict.Reason, verdict.Message) - outcome, err := flux.Request(context.Background(), ref, want, testCommit, v1alpha3.ApplyReasonForeignCommit) + outcome, err := flux.Request(context.Background(), ref, want, testCommit, v1alpha3.ApplyReasonForeignCommit, nil) require.NoError(t, err) assert.Equal(t, OutcomeStampedSource, outcome, "a source with no artifact yet is asked to fetch") assert.NotEmpty(t, requestedAt(t, client, gitRepositoryGVR(), repository.GetNamespace(), repository.GetName())) diff --git a/internal/applier/flux_test.go b/internal/applier/flux_test.go index a27fc2e5..ca574ce8 100644 --- a/internal/applier/flux_test.go +++ b/internal/applier/flux_test.go @@ -163,9 +163,12 @@ func TestFluxRequest_Outcomes(t *testing.T) { want: OutcomeStampedKustomization, wantKustomStamps: true, }, { - name: "foreign commit already fetched stamps the kustomization", revision: "main/" + testCommit, + // The artifact that holds the commit already woke the Kustomization when it arrived. A + // stamp here would only re-apply it, and if the tip had moved on since, re-apply an + // older commit over a published edit. + name: "foreign commit already fetched stamps nothing", revision: "main/" + testCommit, commit: testCommit, reason: v1alpha3.ApplyReasonForeignCommit, - want: OutcomeStampedKustomization, wantKustomStamps: true, + want: OutcomeCurrent, }, { name: "catch-up already current stamps nothing", revision: "sha1:" + testCommit, @@ -180,7 +183,7 @@ func TestFluxRequest_Outcomes(t *testing.T) { gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), tt.revision), ) got, err := pinnedFlux(client).Request(context.Background(), - types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding, tt.commit, tt.reason) + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding, tt.commit, tt.reason, nil) require.NoError(t, err) assert.Equal(t, tt.want, got) @@ -200,6 +203,26 @@ func TestFluxRequest_Outcomes(t *testing.T) { // TestFluxRequest_SourceRefNamespace: an omitted sourceRef namespace is the Kustomization's own; a // set one is followed across namespaces. +// TestFluxRequest_AStaleDecisionStampsNothing is the customer review's first blocker: a request +// decided at one commit, landing after a newer push, stamped the Kustomization of an artifact that +// held the older commit, re-applying it over a published edit. The decision is checked again just +// before the stamp. +func TestFluxRequest_AStaleDecisionStampsNothing(t *testing.T) { + for _, revision := range []string{"main@sha1:" + testCommit, "main@sha1:" + olderCommit} { + client := newSourceClient( + kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")), + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), revision), + ) + _, err := pinnedFlux(client).Request(context.Background(), + types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding, testCommit, + v1alpha3.ApplyReasonRefusal, func() bool { return false }) + require.ErrorIs(t, err, errStale, revision) + for _, action := range client.Actions() { + assert.NotEqual(t, "patch", action.GetVerb(), "a stale decision stamps nothing") + } + } +} + func TestFluxRequest_SourceRefNamespace(t *testing.T) { tests := []struct { name string @@ -221,7 +244,7 @@ func TestFluxRequest_SourceRefNamespace(t *testing.T) { ) kustomization := types.NamespacedName{Namespace: "apps", Name: "checkout"} got, err := pinnedFlux(client).Request(context.Background(), kustomization, testBinding, testCommit, - v1alpha3.ApplyReasonForeignCommit) + v1alpha3.ApplyReasonForeignCommit, nil) require.NoError(t, err) assert.Equal(t, OutcomeStampedSource, got) assert.NotEmpty(t, requestedAt(t, client, gitRepositoryGVR(), tt.wantNamespace, "platform")) @@ -281,7 +304,7 @@ func TestFluxRequest_Failures(t *testing.T) { client := newSourceClient(tt.objs...) _, err := pinnedFlux(client).Request(context.Background(), types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding, testCommit, - v1alpha3.ApplyReasonForeignCommit) + v1alpha3.ApplyReasonForeignCommit, nil) require.Error(t, err) assert.Equal(t, tt.wantPermanent, isPermanent(err)) for _, action := range client.Actions() { diff --git a/internal/git/apply_attempt_test.go b/internal/git/apply_attempt_test.go index 3397440f..0ab2ac24 100644 --- a/internal/git/apply_attempt_test.go +++ b/internal/git/apply_attempt_test.go @@ -331,3 +331,79 @@ func TestApplierRetryDelay_DoublesToACap(t *testing.T) { assert.Equal(t, 15*time.Second, applierRetryDelay(1, true), "a rejected attempt waits for a configuration change") assert.Equal(t, 5*time.Minute, applierRetryDelay(50, true)) } + +// TestApplier_AStaleAttemptIsDecidedAgainWithoutBackoff: an attempt whose decision no longer held +// at the patch (the branch moved on, or became busy) asked nothing, and that is no failure. What it +// carried comes back, and the next quiet moment decides again at the new tip. +func TestApplier_AStaleAttemptIsDecidedAgainWithoutBackoff(t *testing.T) { + loop, requester := manualLoop(t, applierTestTarget("alpha", true)) + owesForeign(loop.w) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1) + require.True(t, out[0].Current(), "decided at the tip on a quiet branch") + + loop.w.debt.observe(RemoteObservation{Commit: "ccc", By: ObservedByPush}) + assert.False(t, out[0].Current(), "the branch moved on") + + report(out[0], ApplyStale) + loop.serviceAppliers() + retry := requester.taken() + require.Len(t, retry, 1, "decided again at once: nothing failed") + assert.Equal(t, "ccc", retry[0].Commit) + assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, retry[0].Reason) + assert.Zero(t, retry[0].PriorFailures) +} + +// TestDecisionHolds_OnlyAtTheTipOnAQuietBranch: the check the requester makes before it patches. +func TestDecisionHolds_OnlyAtTheTipOnAQuietBranch(t *testing.T) { + w := newMetricsTestWorker() + w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByPush}) + assert.True(t, w.decisionHolds("aaa")) + assert.False(t, w.decisionHolds("older"), "another commit is not the tip") + + w.publishWriteBusy(true) + assert.False(t, w.decisionHolds("aaa"), "a window open since") + w.publishWriteBusy(false) + + w.eventQueue <- WorkItem{} + assert.False(t, w.decisionHolds("aaa"), "a queued event is a live edit not yet decided") +} + +// TestPushNotice_TheTipAlreadySeenDoesNotCutAWindow is the customer review's third blocker: the +// late webhook for the worker's own push closed the next save's window, splitting that save across +// two commits. A notice naming the tip already observed leaves the window alone; one naming +// anything else, or nothing, still closes it. +func TestPushNotice_TheTipAlreadySeenDoesNotCutAWindow(t *testing.T) { + for _, tc := range []struct { + name string + after string + wantClose bool + }{ + {name: "our own push, reported late", after: "ours"}, + {name: "somebody else's push", after: "theirs", wantClose: true}, + {name: "a notice that names no commit", after: "", wantClose: true}, + } { + t.Run(tc.name, func(t *testing.T) { + loop, _ := applyTriggerLoop(t, applierTestTarget("alpha", true), &configv1alpha3.GitProvider{ + ObjectMeta: metav1.ObjectMeta{Name: "test-provider", Namespace: "team-a"}, + Spec: configv1alpha3.GitProviderSpec{URL: "https://github.com/acme/platform"}, + }) + loop.w.recordRemoteObservation("ours", ObservedByPush) + loop.openWindow = &openWindow{} + + loop.handleRefreshRequest(&RefreshRequest{PushNotice: true, After: tc.after}) + + assert.Equal(t, tc.wantClose, loop.pushHinted) + }) + } +} + +// TestWriteBusy_ASaveAnnouncedBeforeItsWrites: a registered CommitRequest makes the branch busy, so +// a catch-up cannot land between a save's announcement and its first writes. +func TestWriteBusy_ASaveAnnouncedBeforeItsWrites(t *testing.T) { + loop, _ := applyTriggerLoop(t, applierTestTarget("alpha", true)) + assert.False(t, loop.writeBusy()) + loop.pendingCRs = map[commitRequestID]*pendingCommitRequest{{}: {}} + assert.True(t, loop.writeBusy()) +} diff --git a/internal/git/apply_request.go b/internal/git/apply_request.go index 07887817..01800d11 100644 --- a/internal/git/apply_request.go +++ b/internal/git/apply_request.go @@ -57,6 +57,12 @@ type ApplyRequest struct { // requester reports a failure only for an attempt at zero, so a standing failure is reported // once rather than on every retry. PriorFailures int + // Current reports whether the decision still holds: Commit is still the branch tip, and nothing + // live is waiting to be published on this worker or a sibling. The requester asks it just before + // it patches, and asks nothing when it is false (ApplyStale): an apply of an older commit, or + // over an edit not yet pushed, is exactly what the quiet-branch rule exists to prevent. Nil is + // always current. It does not block. + Current func() bool // Report receives the attempt's outcome. It is called exactly once, and does not block. Report func(ApplyResult) } @@ -94,6 +100,10 @@ const ( // request carried: each changed, is no longer admitted by its ClusterProvider, or withdrew its // consent to a Refusal. The worker decides again from what the targets are now. ApplyDropped ApplyOutcome = "Dropped" + // ApplyStale asked nothing, because the decision no longer held when the patch was due: the + // branch moved on, or became busy (ApplyRequest.Current). It is no failure; the worker decides + // again at the next quiet moment, at the tip as it is then. + ApplyStale ApplyOutcome = "Stale" ) // ApplyResult is the report of one attempt. diff --git a/internal/git/apply_trigger.go b/internal/git/apply_trigger.go index bc04175f..654cb9e2 100644 --- a/internal/git/apply_trigger.go +++ b/internal/git/apply_trigger.go @@ -109,6 +109,13 @@ func (d *applyDebt) foreignOwed() bool { return d.foreign } +// tipIs reports whether the last commit observed is commit. +func (d *applyDebt) tipIs(commit string) bool { + d.mu.Lock() + defer d.mu.Unlock() + return d.known && d.tip == commit +} + func (d *applyDebt) addResult(res ApplyResult) { d.mu.Lock() defer d.mu.Unlock() @@ -200,10 +207,12 @@ func (s *applierState) quietPeriod() time.Duration { } // writeBusy reports whether the loop holds anything live that is not yet on the remote, or is in a -// state where it cannot say. A request is never sent while it does. +// state where it cannot say. A request is never sent while it does. A registered CommitRequest +// counts: a save is announced before its writes arrive, and an apply landing between the two would +// revert the save's first edits before they reach Git. func (l *branchWorkerEventLoop) writeBusy() bool { return l.openWindow != nil || len(l.pendingWrites) > 0 || len(l.deferredHeals) > 0 || - l.recovery.active || l.w.worktreeDirty() + len(l.pendingCRs) > 0 || l.recovery.active || l.w.worktreeDirty() } // closeWindowForForeignCommit closes the open commit window early when the branch has moved under @@ -420,6 +429,7 @@ func (l *branchWorkerEventLoop) sendAttempt(state *applierState, tip string, ref Commit: tip, Targets: targets, PriorFailures: state.failures, + Current: func() bool { return w.decisionHolds(tip) }, Report: w.reportApplyResult, }) } @@ -440,6 +450,11 @@ func (l *branchWorkerEventLoop) settleApplyResults(now time.Time) { case ApplyDelivered, ApplyCurrent: state.failures, state.retryAt = 0, time.Time{} continue + case ApplyStale: + // The decision went stale before the patch: nothing failed, and the next quiet moment + // decides again, at the tip as it is then. + l.restoreAttempt(state, attempt) + continue case ApplyFailed, ApplyRejected, ApplyDropped: } state.failures++ @@ -480,6 +495,14 @@ func (l *branchWorkerEventLoop) applierAwaiting(id uint64) *applierState { return nil } +// decisionHolds is ApplyRequest.Current for an attempt decided at tip: the branch is still at that +// commit, and nothing live is waiting to be published on this worker or a sibling. Work still in the +// queue counts, because a queued watch event is an edit that is live and not yet decided. It is +// called from the requester's goroutine, so it reads only what is safe to read there. +func (w *BranchWorker) decisionHolds(tip string) bool { + return w.debt.tipIs(tip) && !w.busyWriting.Load() && len(w.eventQueue) == 0 && !w.siblingWriteBusy() +} + // reportApplyResult is ApplyRequest.Report: it records the report and wakes the loop to settle it. // It is called from the requester's goroutine. func (w *BranchWorker) reportApplyResult(res ApplyResult) { diff --git a/internal/telemetry/apply_requests.go b/internal/telemetry/apply_requests.go index 4df44a34..de17fe86 100644 --- a/internal/telemetry/apply_requests.go +++ b/internal/telemetry/apply_requests.go @@ -19,6 +19,9 @@ import "go.opentelemetry.io/otel/metric" // dropped — the attempt asked nothing, because no GitTarget may still ask for its reason: each // was deleted or recreated, moved to another branch or applier, is no longer admitted by // its ClusterProvider, or withdrew its consent to a Refusal. +// stale — the attempt asked nothing, because the decision no longer held when the patch was due: +// the branch had moved on or become busy. The worker decides again at the next quiet +// moment. Frequent stale attempts mean edits keep arriving within an attempt's lifetime. // // It counts attempts, so a request that fails twice and then lands is two "failed" and one "sent". // It says nothing about whether the applier APPLIED: a request asks and never reports, and the From 27dd081e721479d3739d59cc5b46810d37137f98 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 21:27:54 +0000 Subject: [PATCH 21/36] fix(git): a late notice for our own push no longer cuts a save short A push notice that reached a busy branch always closed the open window, so the Git host's late webhook for the worker's own push cut the next save's window, splitting that save across two commits. A notice naming the tip already observed now leaves the window alone; anything else still closes it. The delivery's after is still never trusted as the tip, and leaving a window open is the safe direction: it closes on its timers, and its push finds any move by the compare-and-swap. Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/git/push_notification.go | 11 ++++++++--- internal/git/refresh.go | 22 +++++++++++++++++----- 2 files changed, 25 insertions(+), 8 deletions(-) diff --git a/internal/git/push_notification.go b/internal/git/push_notification.go index ad32d2c7..edf2e269 100644 --- a/internal/git/push_notification.go +++ b/internal/git/push_notification.go @@ -4,7 +4,8 @@ package git // EnqueuePushNotification queues a Git host's report that the branch was pushed to. // -// The report is a hint and never a fact: after is logged and nothing else. It is queued as a refresh +// The report is a hint and never a fact: after never decides whether the branch moved. It is queued +// as a refresh // that reaches the remote whatever was proved recently, so the worker reads the advertisement itself // and the observation it records decides whether the branch moved (applyDebt.observe). That one // rule settles every awkward delivery without bookkeeping: a late delivery for an earlier push of @@ -12,8 +13,12 @@ package git // commits finds a tip it did not push last, and a duplicate costs one more advertisement. // // On a busy branch the refresh reads nothing (handleRefreshRequest): the worker closes its open -// window instead and publishes, and the compare-and-swap push discovers the move. +// window instead and publishes, and the compare-and-swap push discovers the move. The one thing after +// is used for is to NOT close a window: a delivery naming the tip the worker last observed is, almost +// always, the late report of the worker's own push, and closing a save's window on it would split +// that save across two commits. That direction is safe even for a wrong after: a window left open +// still closes on its timers, and its push finds any move by the compare-and-swap. func (w *BranchWorker) EnqueuePushNotification(after string) { w.Log.V(1).Info("Git host reported a push; checking where the branch is", "branch", w.Branch, "after", after) - w.EnqueueRefresh(&RefreshRequest{PushNotice: true}) + w.EnqueueRefresh(&RefreshRequest{PushNotice: true, After: after}) } diff --git a/internal/git/refresh.go b/internal/git/refresh.go index 4a1078d1..99efaa8d 100644 --- a/internal/git/refresh.go +++ b/internal/git/refresh.go @@ -40,6 +40,10 @@ type RefreshRequest struct { // remote on an idle branch, and on a busy one it closes the open window so the publication // finds the move. See EnqueuePushNotification. PushNotice bool + // After is the commit a push notice named, or empty. It is never trusted as the branch's tip; + // it only spares an open window a notice that names the tip already observed (see + // EnqueuePushNotification). + After string } // EnqueueRefresh queues a refresh, dropping it if the queue is full. @@ -71,6 +75,18 @@ func (w *BranchWorker) EnqueueRefresh(req *RefreshRequest) { } } +// noteBusyPushNotice records a push notice that reached a busy branch, so the wake closes the open +// window (closeWindowForForeignCommit): the publication it is about to make reads the remote anyway, +// closing the window makes that happen now, and its rejection, if the branch moved, starts the +// replay. A notice naming the tip already observed is the late report of a push already seen, +// usually this worker's own, and is no reason to cut a window short. +func (l *branchWorkerEventLoop) noteBusyPushNotice(req *RefreshRequest, observed RemoteObservation, known bool) { + namesSeenTip := known && req.After != "" && req.After == observed.Commit + if req.PushNotice && !namesSeenTip { + l.pushHinted = true + } +} + // handleRefreshRequest is the refresher, on the loop goroutine. // // It reads Git freely and writes no CONTENT: mirroring what it reads would turn "somebody changed @@ -112,11 +128,7 @@ func (l *branchWorkerEventLoop) handleRefreshRequest(req *RefreshRequest) { // that does not re-read the folder: a layout resolved from a half-written tree is worse than // a slightly old one. A target mid-cycle is writing, and a write publishes its own layout. if len(l.pendingWrites) > 0 || l.openWindow != nil || w.worktreeDirty() { - if req.PushNotice { - // The publication this branch is about to make reads the remote anyway; closing the - // window makes it happen now, and its rejection, if the branch moved, starts the replay. - l.pushHinted = true - } + l.noteBusyPushNotice(req, observed, known) w.Log.V(1).Info("Skipping refresh: this branch is mid-cycle", "branch", w.Branch, "gitTarget", req.Target.String()) return From af4bb65f5d023fcfd99910d2364a5a41f2650d9d Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 21:28:08 +0000 Subject: [PATCH 22/36] fix(receiver): accept GitHub's form-encoded webhooks GitHub's default content type for a webhook wraps the JSON in a payload field, so a hook left on that default got a 400 on every push, while a Flux Receiver accepts it. The signature is over the raw body, so the payload is unwrapped only after verification. The flag help no longer promises an elected replica: leader election is not switched on, and the chart runs one. Co-Authored-By: Claude Opus 5.5 (1M context) --- cmd/receiver.go | 9 +++++---- internal/webhook/receiver.go | 25 ++++++++++++++++++++++--- internal/webhook/receiver_test.go | 17 +++++++++++++++++ 3 files changed, 44 insertions(+), 7 deletions(-) diff --git a/cmd/receiver.go b/cmd/receiver.go index b2c2f689..e917bc33 100644 --- a/cmd/receiver.go +++ b/cmd/receiver.go @@ -46,8 +46,8 @@ func bindReceiverFlags(fs *flag.FlagSet, cfg *appConfig) { "points here plus a GitProvider's status.webhookPath (/git-push//), so a "+ "foreign push is noticed in seconds instead of at the next refresh. Expose it through an "+ "Ingress or Gateway that terminates TLS. Default 0 serves no receiver; foreign pushes are "+ - "then still found by the next rejected push or --git-refresh-interval. Served only on the "+ - "elected replica, because a delivery drives a branch worker.") + "then still found by the next rejected push or --git-refresh-interval. Served by the pod that "+ + "runs the branch workers, because a delivery drives one.") } // receiverEnabled reports whether a push receiver listener is configured. @@ -118,8 +118,9 @@ func newReceiverServerRunnable(addr string, handler http.Handler) *receiverServe }} } -// NeedLeaderElection keeps the receiver on the elected replica. A delivery drives a branch worker, -// and only the leader runs branch workers (WorkerManager needs leader election too): served on a +// NeedLeaderElection keeps the receiver with the branch workers: a delivery drives a branch worker, +// and the WorkerManager asks for leader election too. Leader election is not switched on today and +// the chart runs one replica, so this only keeps the two together should it ever be: served on a // standby, every delivery would find no worker and answer matched=0 for a branch that is written. func (r *receiverServerRunnable) NeedLeaderElection() bool { return true diff --git a/internal/webhook/receiver.go b/internal/webhook/receiver.go index 2b496382..39bc7092 100644 --- a/internal/webhook/receiver.go +++ b/internal/webhook/receiver.go @@ -12,7 +12,9 @@ import ( "errors" "fmt" "io" + "mime" "net/http" + "net/url" "strings" "github.com/go-logr/logr" @@ -237,7 +239,7 @@ func (h *ReceiverHandler) act( return } - delivery, err := parseDelivery(receiverType, body, provider.Spec.URL) + delivery, err := parseDelivery(receiverType, jsonBody(header, body), provider.Spec.URL) switch { case errors.Is(err, errDeliveryIgnored): log.V(1).Info("Ignored a verified push delivery", "reason", err.Error()) @@ -429,14 +431,31 @@ type genericPushPayload struct { // with the reason) for a push that is deliberately not acted on, and any other error for a body // that cannot be read. // -// after is optional in every shape. It is only ever logged, so refusing a delivery for lacking it -// would turn a field nothing depends on into a reason for a host to disable the hook. +// after is optional in every shape. It never decides whether the branch moved (it only spares an open +// window a notice for the tip already observed), so refusing a delivery for lacking it would turn a +// field nothing depends on into a reason for a host to disable the hook. // // The repository check is generic-hmac's alone. A host's own hook is configured on one repository // (or an organization), and a delivery from a sibling repository on an organization hook costs one // ref advertisement for a branch of the same name; comparing the host's repository URLs with // spec.url instead would silently drop every delivery for a provider that reaches the host by // another name (an SSH host alias, ssh.github.com:443), which is the worse failure. +// jsonBody is the JSON document a delivery carries. GitHub's default content type for a webhook is +// application/x-www-form-urlencoded, which wraps the same JSON in a payload field; the signature is +// over the raw body, so this unwrapping happens only after it was verified. Flux accepts that form +// too, so a hook moved over from a Flux Receiver keeps working. +func jsonBody(header http.Header, body []byte) []byte { + mediaType, _, err := mime.ParseMediaType(header.Get("Content-Type")) + if err != nil || mediaType != "application/x-www-form-urlencoded" { + return body + } + form, err := url.ParseQuery(string(body)) + if err != nil || form.Get("payload") == "" { + return body + } + return []byte(form.Get("payload")) +} + func parseDelivery(receiverType v1alpha3.ReceiverType, body []byte, providerURL string) (pushDelivery, error) { if receiverType == v1alpha3.ReceiverTypeGenericHMAC { return parseGenericDelivery(body, providerURL) diff --git a/internal/webhook/receiver_test.go b/internal/webhook/receiver_test.go index f81d7fa4..f015ddf0 100644 --- a/internal/webhook/receiver_test.go +++ b/internal/webhook/receiver_test.go @@ -11,6 +11,7 @@ import ( "errors" "net/http" "net/http/httptest" + "net/url" "strings" "testing" @@ -128,6 +129,9 @@ func genericBody(repository, branch string) string { return string(body) } +// formEncoded is a JSON body the way GitHub sends it with its default content type. +func formEncoded(body string) string { return url.Values{"payload": {body}}.Encode() } + func githubHeaders(event, body, token string) map[string]string { return map[string]string{headerGitHubEvent: event, headerGitHubSignature: "sha256=" + hmacHex(body, token)} } @@ -180,6 +184,19 @@ func receiverCases() map[string]receiverCase { wantCall: &fakePushCall{receiverTestNamespace, "gh", "feature/x", receiverTestCommit}, wantType: "github", wantOutcome: deliveryUnmatched, }, + // GitHub's default content type wraps the JSON in a payload field, and signs the raw body. + "github form-encoded push reaches the worker": { + path: "/git-push/team-a/gh", + headers: func() map[string]string { + h := githubHeaders("push", formEncoded(mainPush), receiverTestToken) + h["Content-Type"] = "application/x-www-form-urlencoded" + return h + }(), + body: formEncoded(mainPush), + wantStatus: http.StatusAccepted, wantMatched: 1, + wantCall: &fakePushCall{receiverTestNamespace, "gh", "main", receiverTestCommit}, + wantType: "github", wantOutcome: deliveryNotified, + }, "github wrong signature is rejected": { path: "/git-push/team-a/gh", headers: githubHeaders("push", mainPush, "not-the-token"), body: mainPush, wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, From 537a90f30d20d4784d07dd4982da3a2cf8214c58 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 21:28:08 +0000 Subject: [PATCH 23/36] docs: the stamp rule, staleness, and where the apply trigger goes next Document that only a Refusal stamps a Kustomization whose artifact holds the commit, that a stale decision stamps nothing, and the stale outcome. The apply trigger design gains its roadmap, with the customer review's remaining items: the applier in the GitTarget's own cluster, receiver hardening, closing the apply window's loop, Argo CD, the webhook applier, and more than one replica. The Flux guide notes that a namespace Role is enough. The e2e recovery assertion checks that Flux reaches the tip, since whether the retry or Flux's own fetch gets there first is a race. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 11 +++--- docs/configuration.md | 14 +++++--- docs/design/apply-trigger.md | 58 +++++++++++++++++++++++++++---- docs/flux.md | 16 ++++++++- docs/interpreting-metrics.md | 2 +- test/e2e/flux_applier_e2e_test.go | 23 ++++++------ 6 files changed, 95 insertions(+), 29 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index 8cc1c604..16c57620 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -675,8 +675,10 @@ What each applier is owed lives on the worker, per applier: one attempt out at a retry clock. The dispatcher in `internal/applier` carries out one attempt off the worker's queue and reports back. It reads each `GitTarget` again first (the same incarnation by uid, still naming this applier on this branch, still admitted, still consenting to a `Refusal`), checks that the -`Kustomization`'s source fetches this repository and branch, and stamps -`reconcile.fluxcd.io/requestedAt` on the `GitRepository` or the `Kustomization`. It never retries: +`Kustomization`'s source fetches this repository and branch, asks the worker whether its decision +still holds (the commit is still the tip and nothing is waiting to be published), and only then +stamps `reconcile.fluxcd.io/requestedAt` on the `GitRepository`, or, for a `Refusal` at an artifact +that already holds the commit, on the `Kustomization`. It never retries: an attempt that does not deliver gives the worker back what it carried, a refusal only while it still stands, and the worker decides the retry on a quiet branch after that applier's backoff. A delivered request records an `ApplyRequested` Event and `status.applier`. A failed request never reaches back @@ -1657,8 +1659,9 @@ as one funnel and its loss paths as one selector: **Apply requests.** `apply_requests_total{applier_kind,reason,outcome}` counts every attempt the worker decided to send (`applier_kind` is `kustomization`; `reason` is `ForeignCommit`, `Refusal` or `CatchUp`), with `outcome` separating -`sent` from `failed` (the worker decides the retry) and `dropped` (no target may still ask for that -reason). It counts attempts, not obligations. +`sent` from `failed` (the worker decides the retry), `dropped` (no target may still ask for that +reason) and `stale` (the decision no longer held at the patch). It counts attempts, not +obligations. `receiver_deliveries_total{type,outcome}` counts receiver deliveries: `notified`, `unmatched`, `ignored` (a ping, a tag, another event type), `rejected` (bad signature) and `invalid`. The receiver cannot tell a foreign push from an own one; the worker decides that from the diff --git a/docs/configuration.md b/docs/configuration.md index 82c8bbc9..4331628c 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1147,9 +1147,12 @@ cluster already holds; an artifact left behind would revert edits that are alrea apply reaches all their folders, and it is asked once for all of them. **What a request does.** What `flux reconcile kustomization --with-source` does: stamp -`reconcile.fluxcd.io/requestedAt` on the `GitRepository`, and on the `Kustomization` instead when -the artifact already holds the requested commit. A new artifact wakes the `Kustomization` by itself. -A `CatchUp` whose artifact already holds the commit stamps nothing. +`reconcile.fluxcd.io/requestedAt` on the `GitRepository` when its artifact does not hold the +requested commit; the new artifact wakes the `Kustomization` by itself. When the artifact already +holds the commit, only a `Refusal` stamps the `Kustomization`, to revert drift at an unchanged +commit; a `ForeignCommit` or `CatchUp` stamps nothing, since that artifact already woke it. Nothing +is stamped when, by the time of the patch, the commit is no longer the branch tip or an edit is +waiting to be published: the request is decided again at the next quiet moment. A request asks; it never claims the apply happened. It runs off the publication path, and a failed request never fails a publication that succeeded. Read the `Kustomization`'s @@ -2496,8 +2499,9 @@ Git host with an Ingress or a Gateway route of your own that terminates TLS, the `webhook-receiver` Service is exposed. Every request still has to carry a valid signature for the route it names; a route whose `GitProvider` has no `spec.receiver` does not serve. -Like everything that drives a branch worker, the listener runs on the elected replica only; a -standby refuses the connection. With the chart's default single replica that is the only pod. The +The listener runs in the pod that runs the branch workers. The chart runs exactly one replica (it +refuses `replicaCount` above `1`), so that is the only pod; running more than one needs ownership +coordination first, and a delivery would then have to reach the pod that owns the branch. The Service targets port `receiver` and is exposed on `service.ports.receiver` (`80` by default). ## Quickstart vs hand-managed resources diff --git a/docs/design/apply-trigger.md b/docs/design/apply-trigger.md index b78efd76..80973bbb 100644 --- a/docs/design/apply-trigger.md +++ b/docs/design/apply-trigger.md @@ -119,10 +119,14 @@ flowchart LR - **Attempting.** The dispatcher re-reads each target. It asks only for those that are still the same incarnation, still name this applier on this branch, are still admitted by their `ClusterProvider`, and for a `Refusal` still consent. If no target still holds the request's - reason, it asks nothing (`Dropped`). Then Flux: stamp `reconcile.fluxcd.io/requestedAt` on the - `GitRepository` when its artifact lacks the commit, else on the `Kustomization`; a `CatchUp` whose - artifact holds the commit stamps nothing. A source on another repository or branch is `Rejected`. -- **Settling.** A delivered attempt is done. Any other outcome gives back what the attempt carried, + reason, it asks nothing (`Dropped`). Just before the stamp it asks the worker whether the decision + still holds (the commit is still the tip, nothing is waiting to be published); if not, it stamps + nothing (`Stale`, no failure). Then Flux: stamp `reconcile.fluxcd.io/requestedAt` on the + `GitRepository` when its artifact lacks the commit; when the artifact holds it, only a `Refusal` + stamps the `Kustomization`, because the artifact already woke it for anything else. A source on + another repository or branch is `Rejected`. +- **Settling.** A delivered attempt is done. A stale one gives back what it carried and is decided + again at the next quiet moment. Any other outcome gives back what the attempt carried, a refusal only if it still stands, and starts that applier's backoff (2s doubling to 5m, at least 15s after a rejection). `ApplyRequestFailed` is recorded for the first failure of a run. - **Learning about pushes.** A push webhook into GitOps Reverser's receiver (seconds), a rejected @@ -185,12 +189,54 @@ in-process plugin: it would run with every credential the operator holds. - **GitOps Reverser becomes the fast path to a deployment.** While it is down, a Git-side push reaches the cluster through `flux reconcile`, a manual sync, or the long backstop interval. Bi-directional editing already stops in that state. -- **The receiver runs on the elected replica only.** A standby refuses deliveries; with more than - one replica they need routing to the leader. +- **One replica.** The receiver, the branch workers and the dispatcher run in one pod, and the chart + refuses more than one replica. Leader election is not switched on; more than one replica needs + ownership coordination first, and deliveries routed to the pod that owns the branch. - **Every applier on the branch is asked on a `ForeignCommit`**, whether or not the push touched its folder. Harmless for Flux; for Argo CD it runs hooks. Narrowing it needs the applier's own source paths, so it waits for a measured complaint. +## Where this goes next + +In order. Each step is one pull request, and the first three are what the first adopter, gitops-api, +needs before it can drop its own gate. + +1. **Merge the Flux applier (#419).** Both reviews' findings and the customer review's blockers on + stale stamps and on a late webhook cutting a save's window are fixed, each with a regression test. +2. **Find the applier in the `GitTarget`'s own cluster.** Today the `Kustomization` is looked up in + the source cluster. The usual way to run Flux against a remote cluster (gitops-api's kcp + workspaces, and every hub-and-spoke install) keeps it in the management cluster and applies + through `spec.kubeConfig`; such a target sits at `ApplierNotFound` forever. Proposed: + `spec.applier.cluster: Source | Local`, `Source` the default. `Local` resolves the reference with + the operator's own identity and only in the `GitTarget`'s own namespace, so it reaches nothing a + namespace-scoped `Role` does not grant. It is the same question as hub Argo CD, so it is settled + first and the Argo CD API follows it. +3. **Harden the receiver.** Today a route answers 401 or 503 when its `GitProvider` exists and 404 + when it does not, on a guessable `/git-push//`, so it lists namespaces; and an + unsigned request reads a Secret and buffers up to 25 MiB with no limit on concurrent requests, in + the pod that also mirrors. Proposed: a path derived from a hash of the token, namespace and name, + as Flux's `Receiver` does; one answer for an unknown route and a bad signature; a small limit on + requests in flight and a lower body cap before verification. +4. **Close the apply window's loop.** Report an attempt as delivered only once Flux has handled the + stamp (`status.lastHandledReconcileAt`), so the worker knows exactly when an apply is running, and + name an edit that lands inside one. Whether to also let the API win the window (option 3 above) + is a separate decision, because it changes what reaches Git. +5. **The Argo CD applier**, as described above, once step 2 has decided where an applier lives. +6. **The webhook applier.** +7. **More than one replica.** Leader election is not switched on, and the chart runs one replica. + Standbys need ownership coordination first (the HA plan in `docs/future/`), and deliveries then + need routing to the pod that owns the branch. + +Test gaps to close along the way: a remote source cluster, `Forbidden` against real RBAC, the receiver +and applier role installed by the Helm chart, two `GitTarget`s sharing one `Kustomization` in e2e, +and `-race` in CI. + +**What adopting it changes for gitops-api.** Its gate shrinks to "one save, one window", which +protects commit messages and has nothing to do with Flux. Its push handler relays each push to the +`GitProvider`'s receiver instead of stamping Flux. And publications trail an editing session by at +least `quietPeriod`, since Flux now applies GitOps Reverser's own commits only after the branch has +been quiet; `quietPeriod` is the knob for that trade. + ## What proves it | Claim | Test | diff --git a/docs/flux.md b/docs/flux.md index 52f37a43..1bc03439 100644 --- a/docs/flux.md +++ b/docs/flux.md @@ -208,6 +208,12 @@ For the operator's own cluster, install the chart with `applierAccess.flux: true [`config/rbac/flux-applier-role.yaml`](../config/rbac/flux-applier-role.yaml) and its binding. For a remote source cluster, grant the same to the identity in the `ClusterProvider`'s kubeconfig. +The chart's grant is cluster-wide for convenience. Requests only ever read and patch the +`Kustomization` and `GitRepository` a `GitTarget` names, so a `Role` in each namespace that holds +them, bound to the same identity, is enough when you would rather not grant cluster-wide. The access +review needs no grant of its own there: Kubernetes lets every authenticated identity create +`selfsubjectaccessreviews` (the `system:basic-user` role). + ## The Git host's webhook Point the Git host's push webhook at GitOps Reverser, so a push reaches the cluster in seconds, @@ -248,7 +254,15 @@ reconcile tick). A request is what `flux reconcile kustomization --with-source` does, minus the waiting. GitOps Reverser merge-patches `reconcile.fluxcd.io/requestedAt` on the `GitRepository` when its artifact does not hold the commit; the new artifact then wakes the `Kustomization`. When the artifact -already holds it, it stamps the `Kustomization` instead, and a `CatchUp` stamps nothing at all. +already holds it, only a `Refusal` stamps the `Kustomization` (to revert drift at an unchanged +commit); a `ForeignCommit` or `CatchUp` stamps nothing, because the artifact that holds the commit +already woke the `Kustomization` when it arrived. + +Before it stamps, GitOps Reverser checks that its decision still holds: the commit is still +the branch tip, and no edit is waiting to be published. When either changed since the request was +decided, it stamps nothing and decides again at the next quiet moment. Without that check, a +request decided at one commit could stamp the `Kustomization` after a newer push and re-apply the +older commit over a published edit. **It is asynchronous.** The patch returns as soon as the API server accepts it, and nothing waits for Flux to fetch or apply. It runs off the branch worker, so a slow source cluster never holds a diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index e2c6fd17..11ac557f 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -822,7 +822,7 @@ and the push receiver tells it that somebody else pushed. See | Metric | Type | Labels | Meaning | | --- | --- | --- | --- | -| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per attempt to one applier. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not ask, including a `Kustomization` that applies another repository or branch; the branch worker keeps the obligation and decides the retry on a quiet branch, so a failure that persists keeps counting at its backoff) or `dropped` (no `GitTarget` may still ask for that reason: each was deleted or recreated, moved to another branch or applier, is no longer admitted by its `ClusterProvider`, or withdrew its consent to a `Refusal`) | +| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per attempt to one applier. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not ask, including a `Kustomization` that applies another repository or branch; the branch worker keeps the obligation and decides the retry on a quiet branch, so a failure that persists keeps counting at its backoff) or `dropped` (no `GitTarget` may still ask for that reason: each was deleted or recreated, moved to another branch or applier, is no longer admitted by its `ClusterProvider`, or withdrew its consent to a `Refusal`) or `stale` (asked nothing because, by the time of the patch, the commit was no longer the branch tip or an edit was waiting to be published; the worker decides again at the next quiet moment) | | `receiver_deliveries_total` | counter | `type`, `outcome` | One per verified-or-refused delivery to `/git-push/`. `type` is the `GitProvider.spec.receiver.type`; `outcome` is `notified` (handed to the branch worker), `unmatched` (no worker for that branch), `ignored` (a ping, a tag, another branch's event type, or a `generic-hmac` body naming another repository), `rejected` (bad signature or an oversized body) or `invalid` (verified but unparseable) | A delivery counted `notified` is only a hint: the worker reads the advertisement, and fetches under diff --git a/test/e2e/flux_applier_e2e_test.go b/test/e2e/flux_applier_e2e_test.go index dcfd5216..aa629fd7 100644 --- a/test/e2e/flux_applier_e2e_test.go +++ b/test/e2e/flux_applier_e2e_test.go @@ -80,9 +80,8 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order cleanupNamespace(testNs) }) - // The review's findings 2 and 4 together, against the real thing: a Kustomization whose source - // follows another branch is reported, and a request for it never stamps that source, however - // it is triggered; and the request is not lost for it, but delivered once the source is fixed. + // A Kustomization whose source follows another branch is reported, and a request for it never + // stamps that source, however it is triggered; once the source is fixed, Flux reaches the tip. It("reports the applier from the source cluster, and never stamps a source on another branch", func() { By("naming a Kustomization that does not exist yet") createGitTarget(destName, testNs, providerName, gitPath, "main") @@ -126,21 +125,21 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order "a GitRepository that follows another branch must never be stamped for this one") }, 20*time.Second, 2*time.Second).Should(Succeed()) - // A request that cannot be delivered goes back to the branch worker, which asks again on a - // quiet branch. Before, it was given up on, and nothing asked again until the next commit. - By("pointing the GitRepository at main, after which the handed-back request is delivered") + // The branch worker keeps the ForeignCommit and decides it again after its backoff. Changing + // the GitRepository's branch also makes Flux fetch on its own, so which of the two brings the + // artifact to the tip is a race: when Flux wins, the worker's retry finds the artifact + // current and rightly stamps nothing. Either way Flux must end at the tip, and the worker's + // half (an obligation kept across a rejected attempt) is proven in apply_attempt_test.go. + By("pointing the GitRepository at main, after which Flux reaches the tip") + head := remoteHeadSHA(Default, repo.CheckoutDir) _, err = kubectlRunInNamespace("flux-system", "patch", "gitrepository", fluxName, "--type=merge", "-p", `{"spec":{"ref":{"branch":"main"}}}`) Expect(err).NotTo(HaveOccurred(), "failed to point the GitRepository at main") verifyResourceCondition("gittarget", destName, testNs, "ApplierConfigured", "True", "Succeeded", "") Eventually(func(g Gomega) { - g.Expect(gitTargetEventMessages(g, testNs, destName, "ApplyRequested")). - To(ContainSubstring("(ForeignCommit)"), "the ForeignCommit must not be lost to the failure") - g.Expect(fluxRequestStamps(fluxName)).NotTo(Equal("|")) - }, 5*time.Minute, 5*time.Second).Should(Succeed()) - Eventually(func(g Gomega) { + g.Expect(fluxLastAppliedRevision(g, fluxName)).To(ContainSubstring(head)) g.Expect(appliedColor(g, appliedNs)).To(Equal("green")) - }, 2*time.Minute, 2*time.Second).Should(Succeed()) + }, 5*time.Minute, 5*time.Second).Should(Succeed()) }) // With no receiver and no API edit, the only thing that can find a Git-side push is the From c326ae574e74b3d044aa0c87db2f8d971ac2d371 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Fri, 16 Oct 2026 22:13:18 +0000 Subject: [PATCH 24/36] test(e2e): share one Flux applier helper between the Flux specs The refusal-reapply and flux-applier specs each carried a copy of the helper that creates a GitRepository and Kustomization; they now share one, which takes the branch and reads the in-cluster Gitea test credential straight into the template's fields. The Reapply row of the configuration reference says the request asks and the applier's status says whether the revert happened. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/configuration.md | 2 +- test/e2e/flux_applier_e2e_test.go | 18 +++++++++-------- test/e2e/refusal_reapply_e2e_test.go | 30 +--------------------------- 3 files changed, 12 insertions(+), 38 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 4331628c..3c4632a8 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1217,7 +1217,7 @@ spec: | Value | What happens | | --- | --- | | `Ignore` (default) | The refusal is reported and nothing else happens | -| `Reapply` | Once the branch is quiet, this target's [applier](#who-applies-this-folder-specapplier) is asked to apply the current tip, with reason `Refusal`. The refused edit is reverted | +| `Reapply` | Once the branch is quiet, this target's [applier](#who-applies-this-folder-specapplier) is asked to apply the current tip, with reason `Refusal`. The refused edit is reverted when the applier applies it; the request asks, and the applier's own status says whether it did | `Reapply` requires `spec.applier`; a target without one is rejected at admission. The request goes to this target's applier alone, and it waits for a quiet branch, so an allowed edit still in a commit diff --git a/test/e2e/flux_applier_e2e_test.go b/test/e2e/flux_applier_e2e_test.go index aa629fd7..0adbd305 100644 --- a/test/e2e/flux_applier_e2e_test.go +++ b/test/e2e/flux_applier_e2e_test.go @@ -221,19 +221,21 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order const fluxApplierFolder = "flux/applier" // applyFluxApplier creates a GitRepository following branch and a Kustomization applying path from -// it, both on a 24h interval, and waits for the Kustomization's first apply. +// it, both on a 24h interval, and waits for the Kustomization's first apply. The Flux specs share it. func applyFluxApplier(testNs string, repo *RepoArtifacts, name, secretName, branch, path string) { GinkgoHelper() - // The in-cluster Gitea test credential, read from the cluster at runtime. - checkout := gitCheckout{secretNamespace: testNs, gitSecretName: repo.GitSecretHTTP} - username, password := checkout.readGitCredentialSecretDataBase64() - Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-gitrepository-http.tmpl", struct { + // The in-cluster Gitea test credential, read from the cluster at runtime, base64-encoded as the + // template's Secret expects. + fields := struct { Namespace, SecretName, Name, RepoURL, Branch, Interval, Username, Password string }{ Namespace: "flux-system", SecretName: secretName, Name: name, RepoURL: repo.RepoURLHTTP, - Branch: branch, Interval: "24h", Username: username, - Password: password, - }, "flux-system")).To(Succeed(), "failed to apply the Flux GitRepository") + Branch: branch, Interval: "24h", + } + checkout := gitCheckout{secretNamespace: testNs, gitSecretName: repo.GitSecretHTTP} + fields.Username, fields.Password = checkout.readGitCredentialSecretDataBase64() + Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-gitrepository-http.tmpl", fields, + "flux-system")).To(Succeed(), "failed to apply the Flux GitRepository") Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-kustomization.tmpl", struct { Namespace, Name, Path, SourceName, Interval, DependsOn string Prune, Wait bool diff --git a/test/e2e/refusal_reapply_e2e_test.go b/test/e2e/refusal_reapply_e2e_test.go index 9fcd9778..eebf7052 100644 --- a/test/e2e/refusal_reapply_e2e_test.go +++ b/test/e2e/refusal_reapply_e2e_test.go @@ -79,7 +79,7 @@ var _ = Describe("Manager Refusal Reapply", Label("manager", "refusal-reapply"), seedRenderedFolderIntoRepo(repo, testNs, seedFolder, "apps/checkout") By("letting Flux apply the overlay once, then leaving it on a 24h interval") - applyRefusalFluxApplier(testNs, repo, fluxName, fluxSecretName, "./"+gitPath) + applyFluxApplier(testNs, repo, fluxName, fluxSecretName, "main", "./"+gitPath) Eventually(func(g Gomega) { g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info")) }, 2*time.Minute, 2*time.Second).Should(Succeed()) @@ -170,34 +170,6 @@ var _ = Describe("Manager Refusal Reapply", Label("manager", "refusal-reapply"), }) }) -// applyRefusalFluxApplier creates a GitRepository and a Kustomization for path on the spec's -// repository, both on a 24h interval, and waits for the first apply. -func applyRefusalFluxApplier(testNs string, repo *RepoArtifacts, name, secretName, path string) { - GinkgoHelper() - // The in-cluster Gitea test credential, read from the cluster at runtime. - checkout := gitCheckout{secretNamespace: testNs, gitSecretName: repo.GitSecretHTTP} - username, password := checkout.readGitCredentialSecretDataBase64() - Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-gitrepository-http.tmpl", struct { - Namespace, SecretName, Name, RepoURL, Branch, Interval, Username, Password string - }{ - Namespace: "flux-system", SecretName: secretName, Name: name, RepoURL: repo.RepoURLHTTP, - Branch: "main", Interval: "24h", Username: username, - Password: password, - }, "flux-system")).To(Succeed(), "failed to apply the Flux GitRepository") - Expect(applyFromTemplate("test/e2e/templates/bi-directional/flux-kustomization.tmpl", struct { - Namespace, Name, Path, SourceName, Interval, DependsOn string - Prune, Wait bool - TargetNS string - HasTargetNS bool - DecryptionProvider, DecryptionSecretName string - }{ - Namespace: "flux-system", Name: name, Path: path, SourceName: name, Interval: "24h", - }, "flux-system")).To(Succeed(), "failed to apply the Flux Kustomization") - _, err := kubectlRunInNamespace("flux-system", "wait", "--for=condition=Ready", - "kustomization/"+name, "--timeout=3m") - Expect(err).NotTo(HaveOccurred(), "the Kustomization must apply the overlay once") -} - // applyRequestedMessages is every ApplyRequested Event message recorded on a GitTarget. func applyRequestedMessages(g Gomega, testNs, gitTarget string) string { return gitTargetEventMessages(g, testNs, gitTarget, "ApplyRequested") From 235ef07f941ddc537f064f7d4c6ac53c1d878d74 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 05:25:23 +0000 Subject: [PATCH 25/36] fix(receiver): derive the route from the token, and answer every miss alike The route was /git-push// and answered 401 or 503 for a GitProvider that exists and 404 for one that does not, so anybody who could reach the listener could list namespaces and providers. It is now the hex HMAC-SHA256 of "/" keyed by the receiver token, published in status.webhookPath and found through a cache index, so an unknown path costs no API read. The receiver recomputes the path from the token it reads, so a rotated token retires the old path at once. An unknown path, a path from before a rotation, a missing token and a wrong signature all answer the same 404. At most 8 deliveries are handled at once (429 above that, before anything is read), and the body cap before verification drops from 25 MiB to 2 MiB. Co-Authored-By: Claude Opus 5.5 (1M context) --- api/v1alpha3/gitprovider_types.go | 30 ++- charts/gitops-reverser/values.yaml | 2 +- cmd/receiver.go | 10 +- .../bases/configbutler.ai_gitproviders.yaml | 11 +- config/receiver-service.yaml | 2 +- docs/architecture.md | 3 +- docs/configuration.md | 21 +- docs/flux.md | 6 +- internal/controller/gitprovider_controller.go | 45 +++- .../controller/gitprovider_controller_test.go | 9 +- .../gitprovider_controller_unit_test.go | 49 +++- internal/webhook/receiver.go | 168 +++++++++---- internal/webhook/receiver_test.go | 236 ++++++++++++------ 13 files changed, 424 insertions(+), 168 deletions(-) diff --git a/api/v1alpha3/gitprovider_types.go b/api/v1alpha3/gitprovider_types.go index 649181ca..e3865ce1 100644 --- a/api/v1alpha3/gitprovider_types.go +++ b/api/v1alpha3/gitprovider_types.go @@ -3,6 +3,10 @@ package v1alpha3 import ( + "crypto/hmac" + "crypto/sha256" + "encoding/hex" + meta "github.com/fluxcd/pkg/apis/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) @@ -77,7 +81,9 @@ type GitProviderReceiver struct { // +kubebuilder:validation:Enum=github;gitlab;gitea;generic-hmac Type ReceiverType `json:"type"` - // SecretRef names a Secret in this namespace whose "token" key is the webhook secret. + // SecretRef names a Secret in this namespace whose "token" key is the webhook secret. The + // token also derives status.webhookPath, so rotating it moves the path: register the new path + // with the new secret. // +required SecretRef meta.LocalObjectReference `json:"secretRef"` } @@ -100,9 +106,20 @@ const ( // ReceiverTokenKey is the Secret key holding a receiver's token, the key a Flux Receiver Secret uses. const ReceiverTokenKey = "token" -// ReceiverPath is the path a GitProvider's receiver is served on. -func ReceiverPath(namespace, name string) string { - return "/git-push/" + namespace + "/" + name +// ReceiverPathPrefix is the prefix every receiver path starts with. +const ReceiverPathPrefix = "/git-push/" + +// A path that names the GitProvider would let anybody who can reach the listener list which +// namespaces and providers exist. Like Flux's Receiver, the path is derived from the token, so +// only whoever holds the token can find it; unlike Flux's sha256(token+name+namespace), it is an +// HMAC over "namespace/name", which no two providers sharing a token can collide on. + +// ReceiverPath is the path a GitProvider's receiver is served on: the prefix and the hex +// HMAC-SHA256 of "/", keyed by the receiver token. +func ReceiverPath(token []byte, namespace, name string) string { + mac := hmac.New(sha256.New, token) + mac.Write([]byte(namespace + "/" + name)) + return ReceiverPathPrefix + hex.EncodeToString(mac.Sum(nil)) } // KnownHostsReference points at a namespace-local ConfigMap or Secret that holds SSH known_hosts @@ -216,8 +233,9 @@ type GitProviderStatus struct { // +optional LastVerifiedAt *metav1.Time `json:"lastVerifiedAt,omitempty"` - // WebhookPath is the path spec.receiver is served on, under the receiver listener. Empty - // without spec.receiver. + // WebhookPath is the path spec.receiver is served on, under the receiver listener: + // /git-push/ and a hash of this provider's name and the receiver token. Empty without + // spec.receiver, and while the token Secret or its "token" key is missing. // +optional WebhookPath string `json:"webhookPath,omitempty"` } diff --git a/charts/gitops-reverser/values.yaml b/charts/gitops-reverser/values.yaml index 86291072..24ec452c 100644 --- a/charts/gitops-reverser/values.yaml +++ b/charts/gitops-reverser/values.yaml @@ -114,7 +114,7 @@ git: # The push receiver: the endpoint a Git host's push webhook points at, so GitOps Reverser learns in # seconds, instead of at the next refresh, that somebody else pushed. A GitProvider opts in with -# spec.receiver and publishes its path as status.webhookPath (/git-push//). +# spec.receiver and publishes its path as status.webhookPath (/git-push/). receiver: # host:port the receiver listens on (passed as --receiver-bind-address), e.g. ":9292". Empty (the # default) serves no receiver: a foreign push is then still found, by the next rejected push or by diff --git a/cmd/receiver.go b/cmd/receiver.go index e917bc33..e24ab2fb 100644 --- a/cmd/receiver.go +++ b/cmd/receiver.go @@ -19,6 +19,7 @@ import ( ctrl "sigs.k8s.io/controller-runtime" + configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" webhookhandler "github.com/ConfigButler/gitops-reverser/internal/webhook" ) @@ -43,7 +44,7 @@ const ( func bindReceiverFlags(fs *flag.FlagSet, cfg *appConfig) { fs.StringVar(&cfg.receiverBindAddress, "receiver-bind-address", receiverOff, "Address (host:port) the push receiver listens on, over plain HTTP: a Git host's push webhook "+ - "points here plus a GitProvider's status.webhookPath (/git-push//), so a "+ + "points here plus a GitProvider's status.webhookPath (/git-push/), so a "+ "foreign push is noticed in seconds instead of at the next refresh. Expose it through an "+ "Ingress or Gateway that terminates TLS. Default 0 serves no receiver; foreign pushes are "+ "then still found by the next rejected push or --git-refresh-interval. Served by the pod that "+ @@ -77,6 +78,11 @@ func setupPushReceiver(mgr ctrl.Manager, cfg appConfig, notifier webhookhandler. "next rejected push or the periodic refresh") return } + // A route is found through this index on the cached GitProviders, so a path nothing publishes + // costs no API read. + fatalIfErr(mgr.GetFieldIndexer().IndexField(context.Background(), &configbutleraiv1alpha3.GitProvider{}, + webhookhandler.GitProviderWebhookPathIndex, webhookhandler.IndexGitProviderWebhookPath), + "unable to index GitProviders by status.webhookPath") // The manager client reads GitProviders from its cache and Secrets straight from the API // server (Secrets are excluded from the cache), so a rotated token is used on the next delivery. handler, err := webhookhandler.NewReceiverHandler(webhookhandler.ReceiverHandlerConfig{ @@ -87,7 +93,7 @@ func setupPushReceiver(mgr ctrl.Manager, cfg appConfig, notifier webhookhandler. fatalIfErr(mgr.Add(newReceiverServerRunnable(cfg.receiverBindAddress, handler)), "unable to add push receiver server runnable") setupLog.Info("push receiver enabled", "address", cfg.receiverBindAddress, - "path", webhookhandler.ReceiverPathPrefix+"/") + "path", webhookhandler.ReceiverPathPrefix+"") } // buildReceiverServeMux routes only the receiver prefix to the handler. Everything else is the diff --git a/config/crd/bases/configbutler.ai_gitproviders.yaml b/config/crd/bases/configbutler.ai_gitproviders.yaml index 4adbdd93..7c714248 100644 --- a/config/crd/bases/configbutler.ai_gitproviders.yaml +++ b/config/crd/bases/configbutler.ai_gitproviders.yaml @@ -148,8 +148,10 @@ spec: the periodic refresh. properties: secretRef: - description: SecretRef names a Secret in this namespace whose - "token" key is the webhook secret. + description: |- + SecretRef names a Secret in this namespace whose "token" key is the webhook secret. The + token also derives status.webhookPath, so rotating it moves the path: register the new path + with the new secret. properties: name: description: Name of the referent. @@ -333,8 +335,9 @@ spec: type: string webhookPath: description: |- - WebhookPath is the path spec.receiver is served on, under the receiver listener. Empty - without spec.receiver. + WebhookPath is the path spec.receiver is served on, under the receiver listener: + /git-push/ and a hash of this provider's name and the receiver token. Empty without + spec.receiver, and while the token Secret or its "token" key is missing. type: string type: object required: diff --git a/config/receiver-service.yaml b/config/receiver-service.yaml index 23263159..7e3573b1 100644 --- a/config/receiver-service.yaml +++ b/config/receiver-service.yaml @@ -1,5 +1,5 @@ # The push receiver: plain HTTP for Git hosts' push webhooks (--receiver-bind-address in -# deployment.yaml). A delivery is POST /git-push//. +# deployment.yaml). A delivery is POST . apiVersion: v1 kind: Service metadata: diff --git a/docs/architecture.md b/docs/architecture.md index 16c57620..a9bb2e32 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -451,7 +451,8 @@ Represents a Git repository and the credentials/configuration used to write it. - `spec.receiver`: optional push receiver, `type` (`github`, `gitlab`, `gitea`, `generic-hmac`) and the `secretRef` holding its token, named after Flux's `Receiver`. - `status.signingPublicKey`: populated when signing is configured and key material is available. -- `status.webhookPath`: the receiver's path, `/git-push//`, when `spec.receiver` is set. +- `status.webhookPath`: the receiver's path, `/git-push/`, + when `spec.receiver` is set and its token Secret holds a token. The controller verifies repository reachability and manages the signing key lifecycle. It generates an ed25519 keypair when `signing.generateWhenMissing` is set. The portable artifact across GitOps diff --git a/docs/configuration.md b/docs/configuration.md index 3c4632a8..7bc42efa 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -408,11 +408,15 @@ spec: secretRef: name: platform-receiver-token # key: token status: - webhookPath: /git-push/team-a/platform-repo + webhookPath: /git-push/3f9a…c41e # 64 hex characters ``` Point the host's push webhook at the receiver listener plus `status.webhookPath`, with the same -token as the webhook secret. Each type verifies the request the way its host signs it: +token as the webhook secret. The path is an HMAC of the provider's namespace and name, keyed by the +token, so it names neither and only whoever holds the token can find it. Rotating the token moves +the path: the old one stops serving at once, and the new one is published at the next reconcile +(within 5 minutes, since Secrets are not watched). `status.webhookPath` stays empty while the Secret +or its `token` key is missing. Each type verifies the request the way its host signs it: | `type` | Verifies | Body | | --- | --- | --- | @@ -425,9 +429,16 @@ token as the webhook secret. Each type verifies the request the way its host sig header is the one Flux's `generic-hmac` `Receiver` checks. The signature is checked before anything in the body is read, so an unsigned request cannot probe -which repositories or branches are tracked. A verified delivery answers `202 Accepted` even when it -names a branch no `GitTarget` writes, because that is not the host's fault and an error teaches a -host to disable the hook. Pushes to other branches and tag pushes are accepted and ignored. +which repositories or branches are tracked. Every request that reaches no verified receiver gets the +same `404`, whether the path is unknown, from before a token rotation, or its signature is wrong, so +the endpoint reveals nothing about which namespaces or providers exist; the operator's log says +which it was. A verified delivery answers `202 Accepted` even when it names a branch no `GitTarget` +writes, because that is not the host's fault and an error teaches a host to disable the hook. Pushes +to other branches and tag pushes are accepted and ignored. + +The listener handles at most 8 deliveries at once and answers `429` above that, and a body over +2 MiB is refused with `413` before its signature is checked. A push event is a few kilobytes, and a +delivery refused for either reason is still found by the periodic refresh. A delivery is a hint, never a fact. It makes the branch worker read the branch's ref advertisement (one request), and what the remote says decides whether the branch gained a foreign commit. A late diff --git a/docs/flux.md b/docs/flux.md index 1bc03439..0aade3e8 100644 --- a/docs/flux.md +++ b/docs/flux.md @@ -232,9 +232,13 @@ after GitOps Reverser has replayed any pending edits onto it. ```bash kubectl get gitprovider platform -n team-a -o jsonpath='{.status.webhookPath}' - # /git-push/team-a/platform + # /git-push/3f9a…c41e ``` + The path is derived from the token, as Flux's `Receiver` path is, so it names no namespace or + provider. A wrong path and a wrong secret both answer `404`; the operator's log says which. + Rotating the token moves the path, so register the new one with the new secret. + 5. Delete the Flux `Receiver` for this repository and its webhook at the Git host. A delivery is only a hint. GitOps Reverser reads where the branch is from the remote and diff --git a/internal/controller/gitprovider_controller.go b/internal/controller/gitprovider_controller.go index 34587e21..9e7f9e91 100644 --- a/internal/controller/gitprovider_controller.go +++ b/internal/controller/gitprovider_controller.go @@ -12,6 +12,7 @@ import ( gitclient "github.com/go-git/go-git/v6/plumbing/client" corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" @@ -121,9 +122,9 @@ func (r *GitProviderReconciler) reconcileGitProvider( log.Error(err, "Could not list the GitTargets for the branch inventory; the last one stands", "name", gitProvider.Name, "namespace", gitProvider.Namespace) } - // Ahead of the gates for the same reason: the path is a fact about the spec, and an operator - // wiring a Git host's webhook needs it while the provider is still red. - publishWebhookPath(gitProvider) + // Ahead of the gates for the same reason: an operator wiring a Git host's webhook needs the + // path while the provider is still red. + r.publishWebhookPath(ctx, log, gitProvider) if err := r.validateCommitConfiguration(gitProvider); err != nil { rd.stalled(ReasonCommitConfigInvalid, err.Error()) @@ -195,14 +196,44 @@ func (r *GitProviderReconciler) commitProvider( } // publishWebhookPath sets status.webhookPath to the path spec.receiver is served on, and clears it -// when there is no receiver. It says where the route is, not that anything serves it: the listener -// is a flag (--receiver-bind-address) this object cannot see, and the path is the same either way. -func publishWebhookPath(gitProvider *configbutleraiv1alpha3.GitProvider) { +// when there is no receiver or no token to derive the path from. It says where the route is, not +// that anything serves it: the listener is a flag (--receiver-bind-address) this object cannot see. +// +// The token is read on every reconcile, so a rotated token moves the path at the next one. The +// receiver recomputes the path from the token it reads per delivery, so the old path stops serving +// at once, whatever this status still says. +func (r *GitProviderReconciler) publishWebhookPath( + ctx context.Context, + log logr.Logger, + gitProvider *configbutleraiv1alpha3.GitProvider, +) { if gitProvider.Spec.Receiver == nil { gitProvider.Status.WebhookPath = "" return } - gitProvider.Status.WebhookPath = configbutleraiv1alpha3.ReceiverPath(gitProvider.Namespace, gitProvider.Name) + secretName := gitProvider.Spec.Receiver.SecretRef.Name + secret, err := r.fetchSecret(ctx, secretName, gitProvider.Namespace) + if err != nil { + if !apierrors.IsNotFound(err) { + // A read that failed says nothing about the token: keep the published path. + log.Error(err, "Could not read the push receiver token Secret; status.webhookPath is kept", + "secret", secretName) + return + } + log.Info("Push receiver token Secret not found; status.webhookPath is empty until it exists", + "secret", secretName) + gitProvider.Status.WebhookPath = "" + return + } + token := secret.Data[configbutleraiv1alpha3.ReceiverTokenKey] + if len(token) == 0 { + log.Info("Push receiver token Secret has no token; status.webhookPath is empty until it does", + "secret", secretName, "key", configbutleraiv1alpha3.ReceiverTokenKey) + gitProvider.Status.WebhookPath = "" + return + } + gitProvider.Status.WebhookPath = configbutleraiv1alpha3.ReceiverPath(token, gitProvider.Namespace, + gitProvider.Name) } // signingKeyFailureReason classifies an ensureSigningKey failure for the Stalled reason. diff --git a/internal/controller/gitprovider_controller_test.go b/internal/controller/gitprovider_controller_test.go index de9dcee6..e72100ad 100644 --- a/internal/controller/gitprovider_controller_test.go +++ b/internal/controller/gitprovider_controller_test.go @@ -378,6 +378,12 @@ var _ = Describe("GitProvider Controller", func() { }, }, } + token := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: "receiver-token", Namespace: "default"}, + Data: map[string][]byte{configbutleraiv1alpha3.ReceiverTokenKey: []byte("receiver-test-token")}, + } + Expect(k8sClient.Create(ctx, token)).To(Succeed()) + DeferCleanup(func() { _ = k8sClient.Delete(ctx, token) }) Expect(k8sClient.Create(ctx, gitProvider)).To(Succeed()) key := types.NamespacedName{Name: gitProvider.Name, Namespace: gitProvider.Namespace} @@ -389,7 +395,8 @@ var _ = Describe("GitProvider Controller", func() { return published.Status.WebhookPath } - Eventually(webhookPath, "30s", "500ms").Should(Equal("/git-push/default/test-provider-receiver")) + Eventually(webhookPath, "30s", "500ms").Should(Equal(configbutleraiv1alpha3.ReceiverPath( + []byte("receiver-test-token"), "default", "test-provider-receiver"))) Eventually(func(g Gomega) { current := &configbutleraiv1alpha3.GitProvider{} diff --git a/internal/controller/gitprovider_controller_unit_test.go b/internal/controller/gitprovider_controller_unit_test.go index e52486d6..f7d86fa3 100644 --- a/internal/controller/gitprovider_controller_unit_test.go +++ b/internal/controller/gitprovider_controller_unit_test.go @@ -6,6 +6,8 @@ import ( "context" "testing" + "github.com/go-logr/logr" + meta "github.com/fluxcd/pkg/apis/meta" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" @@ -151,10 +153,17 @@ func newGitProviderTestClient(t *testing.T, objects ...runtime.Object) ctrlclien Build() } -// TestPublishWebhookPath_FollowsSpecReceiver pins status.webhookPath to spec.receiver: set while a -// receiver is configured, cleared the moment it is removed, so a stale path never points a Git -// host at a route that has stopped serving. -func TestPublishWebhookPath_FollowsSpecReceiver(t *testing.T) { +// TestPublishWebhookPath_FollowsSpecReceiverAndToken pins status.webhookPath to spec.receiver and +// its token: derived from the token while both exist, moved when the token rotates, and cleared when +// either goes, so a stale path never points a Git host at a route that has stopped serving. +func TestPublishWebhookPath_FollowsSpecReceiverAndToken(t *testing.T) { + ctx := context.Background() + secret := &corev1.Secret{ + ObjectMeta: metav1.ObjectMeta{Name: "receiver-token", Namespace: "team-a"}, + Data: map[string][]byte{configbutleraiv1alpha3.ReceiverTokenKey: []byte("first")}, + } + c := newGitProviderTestClient(t, secret) + reconciler := &GitProviderReconciler{Client: c} provider := &configbutleraiv1alpha3.GitProvider{ ObjectMeta: metav1.ObjectMeta{Name: "platform-repo", Namespace: "team-a"}, Spec: configbutleraiv1alpha3.GitProviderSpec{ @@ -164,11 +173,31 @@ func TestPublishWebhookPath_FollowsSpecReceiver(t *testing.T) { }, }, } - - publishWebhookPath(provider) - assert.Equal(t, "/git-push/team-a/platform-repo", provider.Status.WebhookPath) - + log := logr.Discard() + + reconciler.publishWebhookPath(ctx, log, provider) + first := provider.Status.WebhookPath + assert.Equal(t, configbutleraiv1alpha3.ReceiverPath([]byte("first"), "team-a", "platform-repo"), first) + assert.NotContains(t, first, "team-a", "the path must not name the namespace") + assert.NotContains(t, first, "platform-repo", "the path must not name the provider") + + secret.Data[configbutleraiv1alpha3.ReceiverTokenKey] = []byte("second") + require.NoError(t, c.Update(ctx, secret)) + reconciler.publishWebhookPath(ctx, log, provider) + assert.NotEqual(t, first, provider.Status.WebhookPath, "a rotated token moves the path") + + secret.Data = map[string][]byte{"password": []byte("second")} + require.NoError(t, c.Update(ctx, secret)) + reconciler.publishWebhookPath(ctx, log, provider) + assert.Empty(t, provider.Status.WebhookPath, "no token, no path") + + provider.Status.WebhookPath = "/git-push/stale" + require.NoError(t, c.Delete(ctx, secret)) + reconciler.publishWebhookPath(ctx, log, provider) + assert.Empty(t, provider.Status.WebhookPath, "no Secret, no path") + + provider.Status.WebhookPath = "/git-push/stale" provider.Spec.Receiver = nil - publishWebhookPath(provider) - assert.Empty(t, provider.Status.WebhookPath) + reconciler.publishWebhookPath(ctx, log, provider) + assert.Empty(t, provider.Status.WebhookPath, "no receiver, no path") } diff --git a/internal/webhook/receiver.go b/internal/webhook/receiver.go index 39bc7092..b237281e 100644 --- a/internal/webhook/receiver.go +++ b/internal/webhook/receiver.go @@ -23,7 +23,6 @@ import ( corev1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/types" - "k8s.io/apimachinery/pkg/util/validation" "sigs.k8s.io/controller-runtime/pkg/client" logf "sigs.k8s.io/controller-runtime/pkg/log" @@ -32,16 +31,37 @@ import ( "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) -// ReceiverPathPrefix is the path prefix the push receiver serves. A route is -// ReceiverPathPrefix + "/", as v1alpha3.ReceiverPath builds it. -const ReceiverPathPrefix = "/git-push/" +// ReceiverPathPrefix is the path prefix the push receiver serves. A route is the prefix and a hex +// HMAC-SHA256 digest, as v1alpha3.ReceiverPath builds it. +const ReceiverPathPrefix = v1alpha3.ReceiverPathPrefix -// DefaultReceiverMaxRequestBodyBytes caps a delivery's body. It is GitHub's own cap on a webhook -// payload (25 MiB), the largest of the hosts the receiver speaks to, so no genuine push event is -// refused for its size. A push event is a few kilobytes in practice; the cap exists so an -// unauthenticated client cannot make the receiver buffer an unbounded body before the signature -// over it can be checked. -const DefaultReceiverMaxRequestBodyBytes = int64(25 * 1024 * 1024) +// receiverDigestHexLength is the length of a route's digest: a hex SHA-256. +const receiverDigestHexLength = 2 * sha256.Size + +// GitProviderWebhookPathIndex is the cache index the receiver finds a route's GitProvider by. It is +// registered on the manager's cache (IndexGitProviderWebhookPath), so a request for a path nothing +// publishes costs no API read. +const GitProviderWebhookPathIndex = "status.webhookPath" + +// IndexGitProviderWebhookPath indexes a GitProvider by its published status.webhookPath. +func IndexGitProviderWebhookPath(obj client.Object) []string { + provider, ok := obj.(*v1alpha3.GitProvider) + if !ok || provider.Status.WebhookPath == "" { + return nil + } + return []string{provider.Status.WebhookPath} +} + +// DefaultReceiverMaxRequestBodyBytes caps a delivery's body before its signature is checked. A +// push event is a few kilobytes in practice: GitHub and GitLab list at most 20 commits. The cap +// keeps an unauthenticated client from making the receiver buffer much; a genuine push over it is +// refused (413), and the branch worker still finds the push at the next refresh. +const DefaultReceiverMaxRequestBodyBytes = int64(2 * 1024 * 1024) + +// DefaultReceiverMaxInFlight is how many deliveries the receiver handles at once. One is answered +// in milliseconds, so a burst of genuine pushes stays far below it; above it a request is answered +// 429 before anything is read, which bounds what the listener can cost the pod that also mirrors. +const DefaultReceiverMaxInFlight = 8 // Headers each receiver type is signed or classified by. The names are the hosts' own; the // generic-hmac header is the one Flux's generic-hmac Receiver checks, so a relay written for Flux @@ -90,9 +110,11 @@ type ReceiverHandlerConfig struct { Notifier PushNotifier // MaxRequestBodyBytes caps a delivery's body. Zero means DefaultReceiverMaxRequestBodyBytes. MaxRequestBodyBytes int64 + // MaxInFlight bounds the deliveries handled at once. Zero means DefaultReceiverMaxInFlight. + MaxInFlight int } -// ReceiverHandler serves POST /git-push//: the endpoint a Git host's push webhook +// ReceiverHandler serves POST /git-push/: the endpoint a Git host's push webhook // points at, so GitOps Reverser learns quickly that somebody pushed to a branch. // // A delivery is a hint, never a fact. The handler authenticates it, extracts the branch, and asks @@ -104,8 +126,14 @@ type ReceiverHandlerConfig struct { // Authentication comes first. The signature (or GitLab's token) is checked over the raw bytes // before the body is parsed for anything, in constant time, so an unsigned request cannot make a // target do work or probe which branches are tracked. +// +// The route does not name the GitProvider. Its digest is keyed by the receiver token, so only +// whoever holds the token can find it, and every way a request can fail to reach a verified +// receiver (no such path, a path from before the token rotated, no token, a wrong signature) +// gets the same 404: the endpoint says nothing about which namespaces or providers exist. type ReceiverHandler struct { - config ReceiverHandlerConfig + config ReceiverHandlerConfig + inFlight chan struct{} } // NewReceiverHandler builds the push receiver handler. @@ -119,7 +147,10 @@ func NewReceiverHandler(config ReceiverHandlerConfig) (*ReceiverHandler, error) if config.MaxRequestBodyBytes <= 0 { config.MaxRequestBodyBytes = DefaultReceiverMaxRequestBodyBytes } - return &ReceiverHandler{config: config}, nil + if config.MaxInFlight <= 0 { + config.MaxInFlight = DefaultReceiverMaxInFlight + } + return &ReceiverHandler{config: config, inFlight: make(chan struct{}, config.MaxInFlight)}, nil } // receiverResponse is the body of every 202. matched is 1 when a branch worker was told to look @@ -143,8 +174,9 @@ var errDeliveryIgnored = errors.New("delivery ignored") // // The status codes are a contract with Git hosts, which disable a hook that keeps failing: every // verified delivery answers 202, including one that names a branch nothing writes, because that is -// not the host's fault. Only an unauthenticated request (401), an unparseable verified one (400), -// a route that does not serve (404) and a receiver that cannot verify anything yet (503) fail. +// not the host's fault. Only an unparseable verified delivery (400), a request that reaches no +// verified receiver (404, whatever the reason), a body over the cap (413), too many requests at +// once (429) and a receiver that cannot read its own configuration (503) fail. func (h *ReceiverHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { ctx := r.Context() log := logf.Log.WithName("push-receiver") @@ -155,24 +187,42 @@ func (h *ReceiverHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { return } - key, ok := receiverRoute(r.URL.Path) + select { + case h.inFlight <- struct{}{}: + defer func() { <-h.inFlight }() + default: + log.V(1).Info("Refused a push delivery: too many in flight", "maxInFlight", h.config.MaxInFlight) + w.Header().Set("Retry-After", "1") + http.Error(w, "too many requests in flight", http.StatusTooManyRequests) + return + } + + path, ok := receiverRoute(r.URL.Path) if !ok { notFound(w) return } - log = log.WithValues("gitProvider", key.String()) - provider, ok := h.lookupProvider(ctx, log, w, key) + provider, ok := h.lookupProvider(ctx, log, w, path) if !ok { return } + key := types.NamespacedName{Namespace: provider.Namespace, Name: provider.Name} receiverType := provider.Spec.Receiver.Type - log = log.WithValues("type", receiverType) + log = log.WithValues("gitProvider", key.String(), "type", receiverType) token, ok := h.lookupToken(ctx, log, w, provider) if !ok { return } + // The index holds what status last published. A token rotated since then moves the path, and + // the old one must stop serving now, not at the next reconcile. + if subtle.ConstantTimeCompare([]byte(v1alpha3.ReceiverPath(token, key.Namespace, key.Name)), []byte(path)) != 1 { + log.Info("Push delivery on a path the current receiver token no longer derives; the token was " + + "rotated, so register the provider's new status.webhookPath") + notFound(w) + return + } body, ok := h.authenticate(ctx, log, w, r, receiverType, token) if !ok { @@ -209,9 +259,10 @@ func (h *ReceiverHandler) authenticate( if !verifyDelivery(receiverType, r.Header, body, token) { // Info, not V(1): a run of these is a misconfigured hook (a token that does not match) or // somebody probing the endpoint, and either is worth seeing at the default level. Neither - // the token nor the body is logged. + // the token nor the body is logged. The answer is the unknown route's, so a caller cannot + // tell a wrong signature from a path that serves nothing. log.Info("Rejected a push delivery: signature absent or wrong") - http.Error(w, "signature absent or wrong", http.StatusUnauthorized) + notFound(w) h.record(ctx, receiverType, deliveryRejected) return nil, false } @@ -264,55 +315,61 @@ func (h *ReceiverHandler) act( h.record(ctx, receiverType, outcome) } -// receiverRoute parses /git-push//. Anything else under the prefix, including a -// segment that cannot be a namespace or an object name, is no route: answering 404 for it without -// an API read keeps a scanner from turning into apiserver traffic. -func receiverRoute(path string) (types.NamespacedName, bool) { - rest, ok := strings.CutPrefix(path, ReceiverPathPrefix) - if !ok { - return types.NamespacedName{}, false - } - namespace, name, ok := strings.Cut(rest, "/") - if !ok || strings.Contains(name, "/") { - return types.NamespacedName{}, false +// receiverRoute returns the path when it is /git-push/. Anything else is no +// route, answered 404 without even a cache read. +func receiverRoute(path string) (string, bool) { + digest, ok := strings.CutPrefix(path, ReceiverPathPrefix) + if !ok || len(digest) != receiverDigestHexLength { + return "", false } - if len(validation.IsDNS1123Label(namespace)) > 0 || len(validation.IsDNS1123Subdomain(name)) > 0 { - return types.NamespacedName{}, false + for _, c := range digest { + if (c < '0' || c > '9') && (c < 'a' || c > 'f') { + return "", false + } } - return types.NamespacedName{Namespace: namespace, Name: name}, true + return path, true } -// lookupProvider resolves the GitProvider a route names. A missing provider and one without -// spec.receiver answer the same 404, so the endpoint does not reveal which GitProviders exist. +// lookupProvider finds the GitProvider that published path, through the cache index. A path nobody +// published and a provider without spec.receiver answer the same 404. func (h *ReceiverHandler) lookupProvider( ctx context.Context, log logr.Logger, w http.ResponseWriter, - key types.NamespacedName, + path string, ) (*v1alpha3.GitProvider, bool) { - var provider v1alpha3.GitProvider - if err := h.config.Reader.Get(ctx, key, &provider); err != nil { - if apierrors.IsNotFound(err) { - log.V(1).Info("Push delivery for a GitProvider that does not exist") - notFound(w) - return nil, false - } - log.Error(err, "Could not read the GitProvider a push delivery names") + var providers v1alpha3.GitProviderList + if err := h.config.Reader.List( + ctx, + &providers, + client.MatchingFields{GitProviderWebhookPathIndex: path}, + ); err != nil { + log.Error(err, "Could not look up the GitProvider for a push delivery") http.Error(w, "receiver not ready", http.StatusServiceUnavailable) return nil, false } + if len(providers.Items) != 1 { + if len(providers.Items) > 1 { + // Two digests keyed by tokens agree only if a status was written by hand. + log.Info("Push delivery path is published by more than one GitProvider; serving none", + "count", len(providers.Items)) + } + notFound(w) + return nil, false + } + provider := &providers.Items[0] if provider.Spec.Receiver == nil { - log.V(1).Info("Push delivery for a GitProvider without spec.receiver") + log.V(1).Info("Push delivery for a GitProvider whose spec.receiver was removed", + "gitProvider", provider.Namespace+"/"+provider.Name) notFound(w) return nil, false } - return &provider, true + return provider, true } // lookupToken reads the receiver token from the Secret spec.receiver.secretRef names. A missing -// Secret or key is the receiver not being ready, never the sender's fault, so it answers 503 and -// not 401: a 401 would tell whoever set up the hook that their secret is wrong when it is ours -// that is absent, and a host retries a 5xx where it may give up on a 4xx. +// Secret or key answers the unknown route's 404: the path was derived from a token that is gone, +// so nothing can be verified on it. Only a read that failed answers 503, so a host retries it. func (h *ReceiverHandler) lookupToken( ctx context.Context, log logr.Logger, @@ -325,16 +382,17 @@ func (h *ReceiverHandler) lookupToken( if err != nil { if apierrors.IsNotFound(err) { log.Info("Push receiver token Secret not found", "secret", secretName) + notFound(w) } else { log.Error(err, "Could not read the push receiver token Secret", "secret", secretName) + http.Error(w, "receiver not ready", http.StatusServiceUnavailable) } - http.Error(w, "receiver not ready", http.StatusServiceUnavailable) return nil, false } token := secret.Data[v1alpha3.ReceiverTokenKey] if len(token) == 0 { log.Info("Push receiver token Secret has no token", "secret", secretName, "key", v1alpha3.ReceiverTokenKey) - http.Error(w, "receiver not ready", http.StatusServiceUnavailable) + notFound(w) return nil, false } return token, true @@ -511,8 +569,8 @@ func branchFromRef(ref string) (string, error) { return branch, nil } -// notFound answers a route that does not serve. One body for every reason, so the answer does not -// say whether the GitProvider is missing or merely has no receiver. +// notFound answers every request that reaches no verified receiver. One status and body for every +// reason, so the answer does not say whether the path, the token or the signature was wrong. func notFound(w http.ResponseWriter) { http.Error(w, "no push receiver at this path", http.StatusNotFound) } diff --git a/internal/webhook/receiver_test.go b/internal/webhook/receiver_test.go index f015ddf0..39c96204 100644 --- a/internal/webhook/receiver_test.go +++ b/internal/webhook/receiver_test.go @@ -52,6 +52,8 @@ func (f *fakePushNotifier) NotifyPush(providerNamespace, providerName, branch, a return f.workers[branch] } +// receiverTestProvider is a provider whose status publishes the path receiverTestToken derives, as +// the GitProvider controller would. func receiverTestProvider(name string, receiver *v1alpha3.GitProviderReceiver) *v1alpha3.GitProvider { return &v1alpha3.GitProvider{ ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: receiverTestNamespace}, @@ -60,9 +62,22 @@ func receiverTestProvider(name string, receiver *v1alpha3.GitProviderReceiver) * AllowedBranches: []string{"*"}, Receiver: receiver, }, + Status: v1alpha3.GitProviderStatus{WebhookPath: route(name)}, } } +// route is the path receiverTestToken derives for the provider name in the test namespace. +func route(name string) string { + return v1alpha3.ReceiverPath([]byte(receiverTestToken), receiverTestNamespace, name) +} + +// rotatedProvider still publishes the path a token it no longer holds derived. +func rotatedProvider() *v1alpha3.GitProvider { + provider := receiverTestProvider("rotated", receiverOf(v1alpha3.ReceiverTypeGitHub, "receiver-token")) + provider.Status.WebhookPath = v1alpha3.ReceiverPath([]byte("the-old-token"), receiverTestNamespace, "rotated") + return provider +} + func receiverOf(receiverType v1alpha3.ReceiverType, secret string) *v1alpha3.GitProviderReceiver { return &v1alpha3.GitProviderReceiver{Type: receiverType, SecretRef: meta.LocalObjectReference{Name: secret}} } @@ -87,6 +102,7 @@ func receiverTestObjects() []client.Object { receiverTestProvider("plain", nil), receiverTestProvider("no-secret", receiverOf(v1alpha3.ReceiverTypeGitHub, "absent")), receiverTestProvider("no-key", receiverOf(v1alpha3.ReceiverTypeGitHub, "wrong-key")), + rotatedProvider(), } } @@ -100,7 +116,8 @@ func newReceiverTestHandler( scheme := runtime.NewScheme() require.NoError(t, clientgoscheme.AddToScheme(scheme)) require.NoError(t, v1alpha3.AddToScheme(scheme)) - builder := fake.NewClientBuilder().WithScheme(scheme).WithObjects(receiverTestObjects()...) + builder := fake.NewClientBuilder().WithScheme(scheme).WithObjects(receiverTestObjects()...). + WithIndex(&v1alpha3.GitProvider{}, GitProviderWebhookPathIndex, IndexGitProviderWebhookPath) if funcs != nil { builder = builder.WithInterceptorFuncs(*funcs) } @@ -174,19 +191,19 @@ func receiverCases() map[string]receiverCase { return map[string]receiverCase{ // github "github push to a worked branch is notified": { - path: "/git-push/team-a/gh", headers: githubHeaders("push", mainPush, receiverTestToken), body: mainPush, + path: route("gh"), headers: githubHeaders("push", mainPush, receiverTestToken), body: mainPush, wantStatus: http.StatusAccepted, wantMatched: 1, wantCall: ghCall, wantType: "github", wantOutcome: deliveryNotified, }, "github push to a branch nothing writes is still a success": { - path: "/git-push/team-a/gh", headers: githubHeaders("push", otherPush, receiverTestToken), body: otherPush, + path: route("gh"), headers: githubHeaders("push", otherPush, receiverTestToken), body: otherPush, wantStatus: http.StatusAccepted, wantMatched: 0, wantCall: &fakePushCall{receiverTestNamespace, "gh", "feature/x", receiverTestCommit}, wantType: "github", wantOutcome: deliveryUnmatched, }, // GitHub's default content type wraps the JSON in a payload field, and signs the raw body. "github form-encoded push reaches the worker": { - path: "/git-push/team-a/gh", + path: route("gh"), headers: func() map[string]string { h := githubHeaders("push", formEncoded(mainPush), receiverTestToken) h["Content-Type"] = "application/x-www-form-urlencoded" @@ -198,111 +215,111 @@ func receiverCases() map[string]receiverCase { wantType: "github", wantOutcome: deliveryNotified, }, "github wrong signature is rejected": { - path: "/git-push/team-a/gh", headers: githubHeaders("push", mainPush, "not-the-token"), body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + path: route("gh"), headers: githubHeaders("push", mainPush, "not-the-token"), body: mainPush, + wantStatus: http.StatusNotFound, wantType: "github", wantOutcome: deliveryRejected, }, "github missing signature is rejected": { - path: "/git-push/team-a/gh", headers: map[string]string{headerGitHubEvent: "push"}, body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + path: route("gh"), headers: map[string]string{headerGitHubEvent: "push"}, body: mainPush, + wantStatus: http.StatusNotFound, wantType: "github", wantOutcome: deliveryRejected, }, "github signature without the sha256 prefix is rejected": { - path: "/git-push/team-a/gh", + path: route("gh"), headers: map[string]string{ headerGitHubEvent: "push", headerGitHubSignature: hmacHex(mainPush, receiverTestToken), }, body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + wantStatus: http.StatusNotFound, wantType: "github", wantOutcome: deliveryRejected, }, "github signature that is not hex is rejected": { - path: "/git-push/team-a/gh", + path: route("gh"), headers: map[string]string{ headerGitHubEvent: "push", headerGitHubSignature: "sha256=zz", }, body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "github", wantOutcome: deliveryRejected, + wantStatus: http.StatusNotFound, wantType: "github", wantOutcome: deliveryRejected, }, "github ping is accepted and does nothing": { - path: "/git-push/team-a/gh", headers: githubHeaders("ping", `{"zen":"hi"}`, receiverTestToken), + path: route("gh"), headers: githubHeaders("ping", `{"zen":"hi"}`, receiverTestToken), body: `{"zen":"hi"}`, wantStatus: http.StatusAccepted, wantType: "github", wantOutcome: deliveryIgnored, }, "github non-push event is ignored": { - path: "/git-push/team-a/gh", headers: githubHeaders("issues", `{}`, receiverTestToken), body: `{}`, + path: route("gh"), headers: githubHeaders("issues", `{}`, receiverTestToken), body: `{}`, wantStatus: http.StatusAccepted, wantType: "github", wantOutcome: deliveryIgnored, }, "github tag push is ignored": { - path: "/git-push/team-a/gh", headers: githubHeaders("push", tagPush, receiverTestToken), body: tagPush, + path: route("gh"), headers: githubHeaders("push", tagPush, receiverTestToken), body: tagPush, wantStatus: http.StatusAccepted, wantType: "github", wantOutcome: deliveryIgnored, }, "github verified body that is not JSON is invalid": { - path: "/git-push/team-a/gh", headers: githubHeaders("push", "not json", receiverTestToken), + path: route("gh"), headers: githubHeaders("push", "not json", receiverTestToken), body: "not json", wantStatus: http.StatusBadRequest, wantType: "github", wantOutcome: deliveryInvalid, }, "github verified push without a ref is invalid": { - path: "/git-push/team-a/gh", headers: githubHeaders("push", `{"after":"abc"}`, receiverTestToken), + path: route("gh"), headers: githubHeaders("push", `{"after":"abc"}`, receiverTestToken), body: `{"after":"abc"}`, wantStatus: http.StatusBadRequest, wantType: "github", wantOutcome: deliveryInvalid, }, "github ref naming no branch is invalid": { - path: "/git-push/team-a/gh", headers: githubHeaders("push", `{"ref":"refs/heads/"}`, receiverTestToken), + path: route("gh"), headers: githubHeaders("push", `{"ref":"refs/heads/"}`, receiverTestToken), body: `{"ref":"refs/heads/"}`, wantStatus: http.StatusBadRequest, wantType: "github", wantOutcome: deliveryInvalid, }, // gitlab "gitlab push with the token is notified": { - path: "/git-push/team-a/gl", headers: gitlabHeaders("Push Hook", receiverTestToken), body: mainPush, + path: route("gl"), headers: gitlabHeaders("Push Hook", receiverTestToken), body: mainPush, wantStatus: http.StatusAccepted, wantMatched: 1, wantCall: &fakePushCall{receiverTestNamespace, "gl", "main", receiverTestCommit}, wantType: "gitlab", wantOutcome: deliveryNotified, }, "gitlab wrong token is rejected": { - path: "/git-push/team-a/gl", headers: gitlabHeaders("Push Hook", "nope"), body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "gitlab", wantOutcome: deliveryRejected, + path: route("gl"), headers: gitlabHeaders("Push Hook", "nope"), body: mainPush, + wantStatus: http.StatusNotFound, wantType: "gitlab", wantOutcome: deliveryRejected, }, "gitlab missing token is rejected": { - path: "/git-push/team-a/gl", headers: map[string]string{headerGitLabEvent: "Push Hook"}, body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "gitlab", wantOutcome: deliveryRejected, + path: route("gl"), headers: map[string]string{headerGitLabEvent: "Push Hook"}, body: mainPush, + wantStatus: http.StatusNotFound, wantType: "gitlab", wantOutcome: deliveryRejected, }, "gitlab tag push hook is ignored": { - path: "/git-push/team-a/gl", headers: gitlabHeaders("Tag Push Hook", receiverTestToken), body: tagPush, + path: route("gl"), headers: gitlabHeaders("Tag Push Hook", receiverTestToken), body: tagPush, wantStatus: http.StatusAccepted, wantType: "gitlab", wantOutcome: deliveryIgnored, }, // gitea "gitea push is notified": { - path: "/git-push/team-a/gt", headers: giteaHeaders("push", mainPush, receiverTestToken), body: mainPush, + path: route("gt"), headers: giteaHeaders("push", mainPush, receiverTestToken), body: mainPush, wantStatus: http.StatusAccepted, wantMatched: 1, wantCall: &fakePushCall{receiverTestNamespace, "gt", "main", receiverTestCommit}, wantType: "gitea", wantOutcome: deliveryNotified, }, "gitea wrong signature is rejected": { - path: "/git-push/team-a/gt", headers: giteaHeaders("push", mainPush, "nope"), body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "gitea", wantOutcome: deliveryRejected, + path: route("gt"), headers: giteaHeaders("push", mainPush, "nope"), body: mainPush, + wantStatus: http.StatusNotFound, wantType: "gitea", wantOutcome: deliveryRejected, }, "gitea github-style prefixed signature is rejected": { - path: "/git-push/team-a/gt", + path: route("gt"), headers: map[string]string{ headerGiteaEvent: "push", headerGiteaSignature: "sha256=" + hmacHex(mainPush, receiverTestToken), }, body: mainPush, - wantStatus: http.StatusUnauthorized, wantType: "gitea", wantOutcome: deliveryRejected, + wantStatus: http.StatusNotFound, wantType: "gitea", wantOutcome: deliveryRejected, }, "gitea non-push event is ignored": { - path: "/git-push/team-a/gt", headers: giteaHeaders("create", `{}`, receiverTestToken), body: `{}`, + path: route("gt"), headers: giteaHeaders("create", `{}`, receiverTestToken), body: `{}`, wantStatus: http.StatusAccepted, wantType: "gitea", wantOutcome: deliveryIgnored, }, // generic-hmac "generic-hmac push for the same repository spelled differently is notified": { - path: "/git-push/team-a/gen", headers: genericHeaders(genMain, receiverTestToken), body: genMain, + path: route("gen"), headers: genericHeaders(genMain, receiverTestToken), body: genMain, wantStatus: http.StatusAccepted, wantMatched: 1, wantCall: &fakePushCall{receiverTestNamespace, "gen", "main", receiverTestCommit}, wantType: "generic-hmac", wantOutcome: deliveryNotified, }, "generic-hmac full ref is read as its branch": { - path: "/git-push/team-a/gen", + path: route("gen"), headers: genericHeaders(genericBody("", "refs/heads/main"), receiverTestToken), body: genericBody("", "refs/heads/main"), wantStatus: http.StatusAccepted, wantMatched: 1, @@ -310,64 +327,74 @@ func receiverCases() map[string]receiverCase { wantType: "generic-hmac", wantOutcome: deliveryNotified, }, "generic-hmac another repository is ignored, not refused": { - path: "/git-push/team-a/gen", + path: route("gen"), headers: genericHeaders(genericBody("https://github.com/acme/other", "main"), receiverTestToken), body: genericBody("https://github.com/acme/other", "main"), wantStatus: http.StatusAccepted, wantType: "generic-hmac", wantOutcome: deliveryIgnored, }, "generic-hmac tag ref is ignored": { - path: "/git-push/team-a/gen", + path: route("gen"), headers: genericHeaders(genericBody("", "refs/tags/v1"), receiverTestToken), body: genericBody("", "refs/tags/v1"), wantStatus: http.StatusAccepted, wantType: "generic-hmac", wantOutcome: deliveryIgnored, }, "generic-hmac wrong signature is rejected": { - path: "/git-push/team-a/gen", headers: genericHeaders(genMain, "nope"), body: genMain, - wantStatus: http.StatusUnauthorized, wantType: "generic-hmac", wantOutcome: deliveryRejected, + path: route("gen"), headers: genericHeaders(genMain, "nope"), body: genMain, + wantStatus: http.StatusNotFound, wantType: "generic-hmac", wantOutcome: deliveryRejected, }, "generic-hmac without a branch is invalid": { - path: "/git-push/team-a/gen", + path: route("gen"), headers: genericHeaders(`{"repository":"x"}`, receiverTestToken), body: `{"repository":"x"}`, wantStatus: http.StatusBadRequest, wantType: "generic-hmac", wantOutcome: deliveryInvalid, }, - // routes that do not serve, and a receiver that cannot verify yet - "unknown provider is 404": { - path: "/git-push/team-a/missing", + // every way of reaching no verified receiver is the same 404 + "a path nothing publishes is 404": { + path: v1alpha3.ReceiverPath([]byte(receiverTestToken), receiverTestNamespace, "missing"), headers: githubHeaders("push", mainPush, receiverTestToken), body: mainPush, wantStatus: http.StatusNotFound, }, - "provider without spec.receiver is 404": { - path: "/git-push/team-a/plain", headers: githubHeaders("push", mainPush, receiverTestToken), body: mainPush, + "a provider without spec.receiver is 404": { + path: route("plain"), headers: githubHeaders("push", mainPush, receiverTestToken), body: mainPush, + wantStatus: http.StatusNotFound, + }, + "a path from before the token rotated is 404": { + path: rotatedProvider().Status.WebhookPath, + headers: githubHeaders("push", mainPush, receiverTestToken), + body: mainPush, wantStatus: http.StatusNotFound, }, - "route without a name is 404": { - path: "/git-push/team-a", body: mainPush, wantStatus: http.StatusNotFound, + "a missing token Secret is 404": { + path: route("no-secret"), headers: githubHeaders("push", mainPush, receiverTestToken), + body: mainPush, wantStatus: http.StatusNotFound, }, - "route with an extra segment is 404": { - path: "/git-push/team-a/gh/extra", body: mainPush, wantStatus: http.StatusNotFound, + "a token Secret without the token key is 404": { + path: route("no-key"), headers: githubHeaders("push", mainPush, receiverTestToken), + body: mainPush, wantStatus: http.StatusNotFound, }, - "route with an empty name is 404": { - path: "/git-push/team-a/", body: mainPush, wantStatus: http.StatusNotFound, + "the old namespace/name route is 404": { + path: "/git-push/team-a/" + "gh", + headers: githubHeaders("push", mainPush, receiverTestToken), + body: mainPush, + wantStatus: http.StatusNotFound, }, - "route with an invalid namespace is 404": { - path: "/git-push/Team_A/gh", body: mainPush, wantStatus: http.StatusNotFound, + "an uppercase digest is 404": { + path: strings.ToUpper(route("gh")), body: mainPush, wantStatus: http.StatusNotFound, }, - "path outside the prefix is 404": { - path: "/other/team-a/gh", body: mainPush, wantStatus: http.StatusNotFound, + "a short digest is 404": { + path: route("gh")[:len(route("gh"))-1], body: mainPush, wantStatus: http.StatusNotFound, }, - "missing token Secret is 503, not 401": { - path: "/git-push/team-a/no-secret", headers: githubHeaders("push", mainPush, receiverTestToken), - body: mainPush, wantStatus: http.StatusServiceUnavailable, + "a digest with an extra segment is 404": { + path: route("gh") + "/extra", body: mainPush, wantStatus: http.StatusNotFound, }, - "token Secret without the token key is 503": { - path: "/git-push/team-a/no-key", headers: githubHeaders("push", mainPush, receiverTestToken), - body: mainPush, wantStatus: http.StatusServiceUnavailable, + "path outside the prefix is 404": { + path: "/other/" + strings.TrimPrefix(route("gh"), ReceiverPathPrefix), body: mainPush, + wantStatus: http.StatusNotFound, }, "GET is not allowed": { - method: http.MethodGet, path: "/git-push/team-a/gh", wantStatus: http.StatusMethodNotAllowed, + method: http.MethodGet, path: route("gh"), wantStatus: http.StatusMethodNotAllowed, }, } } @@ -431,7 +458,7 @@ func TestReceiverHandler_OversizedBodyIsRefusedBeforeVerification(t *testing.T) body := hostPushBody("refs/heads/main") require.Greater(t, len(body), 64) - req := httptest.NewRequest(http.MethodPost, "/git-push/team-a/gh", strings.NewReader(body)) + req := httptest.NewRequest(http.MethodPost, route("gh"), strings.NewReader(body)) for k, v := range githubHeaders("push", body, receiverTestToken) { req.Header.Set(k, v) } @@ -446,26 +473,87 @@ func TestReceiverHandler_OversizedBodyIsRefusedBeforeVerification(t *testing.T) assert.Equal(t, int64(1), got) } -// TestReceiverHandler_UnreadableProviderIs503 separates "no such route" from "cannot tell yet": an -// API error reading the GitProvider must not answer 404, which a host may treat as a dead hook. -func TestReceiverHandler_UnreadableProviderIs503(t *testing.T) { - notifier := &fakePushNotifier{} - h := newReceiverTestHandler(t, notifier, 0, &interceptor.Funcs{ - Get: func(_ context.Context, _ client.WithWatch, _ client.ObjectKey, _ client.Object, _ ...client.GetOption) error { - return errors.New("apiserver unavailable") +// TestReceiverHandler_UnreadableConfigurationIs503 separates "no such route" from "cannot tell +// yet": an API error reading the GitProvider or its token must not answer 404, which a host may treat +// as a dead hook. +func TestReceiverHandler_UnreadableConfigurationIs503(t *testing.T) { + failing := errors.New("apiserver unavailable") + for name, funcs := range map[string]*interceptor.Funcs{ + "provider lookup fails": { + List: func(_ context.Context, _ client.WithWatch, _ client.ObjectList, _ ...client.ListOption) error { + return failing + }, }, - }) + "token read fails": { + Get: func(_ context.Context, _ client.WithWatch, _ client.ObjectKey, _ client.Object, _ ...client.GetOption) error { + return failing + }, + }, + } { + t.Run(name, func(t *testing.T) { + notifier := &fakePushNotifier{} + h := newReceiverTestHandler(t, notifier, 0, funcs) + + body := hostPushBody("refs/heads/main") + req := httptest.NewRequest(http.MethodPost, route("gh"), strings.NewReader(body)) + for k, v := range githubHeaders("push", body, receiverTestToken) { + req.Header.Set(k, v) + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + + assert.Equal(t, http.StatusServiceUnavailable, rec.Code) + assert.Empty(t, notifier.calls) + }) + } +} + +// TestReceiverHandler_TooManyInFlightIs429 pins the in-flight cap: a request over it is answered +// before anything is read, and a request after the others finished is served again. +func TestReceiverHandler_TooManyInFlightIs429(t *testing.T) { + notifier := &fakePushNotifier{workers: map[string]bool{"main": true}} + h := newReceiverTestHandler(t, notifier, 0, nil) + for range DefaultReceiverMaxInFlight { + h.inFlight <- struct{}{} + } body := hostPushBody("refs/heads/main") - req := httptest.NewRequest(http.MethodPost, "/git-push/team-a/gh", strings.NewReader(body)) - for k, v := range githubHeaders("push", body, receiverTestToken) { - req.Header.Set(k, v) + post := func() *httptest.ResponseRecorder { + req := httptest.NewRequest(http.MethodPost, route("gh"), strings.NewReader(body)) + for k, v := range githubHeaders("push", body, receiverTestToken) { + req.Header.Set(k, v) + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + return rec } - rec := httptest.NewRecorder() - h.ServeHTTP(rec, req) - assert.Equal(t, http.StatusServiceUnavailable, rec.Code) + rec := post() + assert.Equal(t, http.StatusTooManyRequests, rec.Code) + assert.Equal(t, "1", rec.Header().Get("Retry-After")) assert.Empty(t, notifier.calls) + + for range DefaultReceiverMaxInFlight { + <-h.inFlight + } + assert.Equal(t, http.StatusAccepted, post().Code) + assert.Len(t, notifier.calls, 1) +} + +// TestReceiverPath_DoesNotNameTheProviderAndNeverCollides pins what the path is for: it reveals +// neither namespace nor name, it moves with the token, and two providers sharing a token whose +// names concatenate alike (Flux's sha256(token+name+namespace) would collide on these) get two paths. +func TestReceiverPath_DoesNotNameTheProviderAndNeverCollides(t *testing.T) { + token := []byte(receiverTestToken) + path := v1alpha3.ReceiverPath(token, "team-a", "platform") + assert.True(t, strings.HasPrefix(path, ReceiverPathPrefix)) + assert.NotContains(t, path, "team-a") + assert.NotContains(t, path, "platform") + _, ok := receiverRoute(path) + assert.True(t, ok, "the receiver must serve the path the API builds") + + assert.NotEqual(t, path, v1alpha3.ReceiverPath([]byte("another"), "team-a", "platform")) + assert.NotEqual(t, v1alpha3.ReceiverPath(token, "ab", "c"), v1alpha3.ReceiverPath(token, "a", "bc")) } func TestNewReceiverHandler_RequiresReaderAndNotifier(t *testing.T) { From c6e1da78aecac3f07415f1f8025fc447c8705274 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 05:29:35 +0000 Subject: [PATCH 26/36] feat(applier): find the applier through its own ClusterProvider spec.applier.clusterProviderRef names the ClusterProvider whose cluster holds the Kustomization; omitted, it is the GitTarget's own spec.clusterProviderRef, so nothing that exists changes meaning. It covers Flux run from a management cluster (gitops-api's kcp workspaces, every hub-and-spoke install), which used to sit at ApplierNotFound forever, and it is the shape the Argo CD applier will take for a hub Argo CD. The applier's provider is gated exactly as a source's is: its accessFrom must admit the GitTarget's namespace (authz.ApplierAdmitted), checked by ApplierConfigured and again by the dispatcher before every attempt. A ClusterProvider change re-checks the GitTargets using it as an applier's. An applier-only cluster has no cluster context, because nothing would release one: ApplierDynamicClient reuses a mirrored cluster's client, and otherwise builds one per call from the provider as it resolves. Co-Authored-By: Claude Opus 5.5 (1M context) --- api/v1alpha3/gittarget_types.go | 27 +++++++- api/v1alpha3/helpers_test.go | 18 +++++ api/v1alpha3/zz_generated.deepcopy.go | 5 ++ cmd/main.go | 4 +- .../crd/bases/configbutler.ai_gittargets.yaml | 20 +++++- docs/configuration.md | 37 +++++++--- docs/flux.md | 17 +++-- internal/applier/dispatcher.go | 21 +++--- internal/applier/dispatcher_test.go | 42 ++++++++++++ internal/authz/clusterprovider_admission.go | 52 ++++++++++---- .../authz/clusterprovider_admission_test.go | 68 +++++++++++++++++++ internal/controller/gittarget_applier.go | 25 +++++-- internal/controller/gittarget_applier_test.go | 54 +++++++++++++++ internal/controller/gittarget_controller.go | 6 +- internal/git/apply_request.go | 4 +- internal/git/apply_trigger_test.go | 18 +++++ internal/watch/cluster_context.go | 39 +++++++++-- internal/watch/cluster_model_test.go | 6 +- internal/watch/source_dynamic_client_test.go | 34 ++++++++++ 19 files changed, 437 insertions(+), 60 deletions(-) diff --git a/api/v1alpha3/gittarget_types.go b/api/v1alpha3/gittarget_types.go index 6570972a..cbd7ed81 100644 --- a/api/v1alpha3/gittarget_types.go +++ b/api/v1alpha3/gittarget_types.go @@ -98,7 +98,8 @@ type GitTargetSpec struct { // refused edit (spec.onRefusal: Reapply), and, once the branch has been quiet for quietPeriod, // to keep the applier current with its own pushes. Switch off the applier's own triggers (short // intervals, Git-host webhooks into it), or they go on applying without that check. The - // reference is looked up in the source cluster, with the identity the ClusterProvider uses. + // reference is looked up in the cluster spec.applier.clusterProviderRef names, by default the + // source cluster, with the identity that ClusterProvider uses. // +optional Applier *GitTargetApplier `json:"applier,omitempty"` @@ -674,6 +675,15 @@ func (g *GitTarget) SourceCluster() string { return g.Spec.ClusterProviderRef.Name } +// ApplierCluster is the ClusterProvider whose cluster holds spec.applier: its clusterProviderRef, +// or the source cluster (SourceCluster) when that is omitted. Call it only with spec.applier set. +func (g *GitTarget) ApplierCluster() string { + if ref := g.Spec.Applier.ClusterProviderRef; ref != nil && ref.Name != "" { + return ref.Name + } + return g.SourceCluster() +} + // IsLocalSource reports whether this GitTarget references the "default" ClusterProvider, which the // watch data plane maps to its local cluster context. It is a NAME test, not a claim about the // physical cluster: a "default" provider may carry a kubeConfig. It only supplies the pre-discovery @@ -709,6 +719,21 @@ const ( // GitTargetApplier names the object that applies a GitTarget's folder to its source cluster. type GitTargetApplier struct { + // The same reference spec.clusterProviderRef is, for the same reason: a ClusterProvider is how + // GitOps Reverser reaches any cluster, with its credential and its accessFrom gate. Flux run + // from a management cluster is the usual way to deploy to a remote one, so the applier's + // cluster cannot be assumed to be the source cluster. Unlike the source, it may change: moving + // an applier does not change what the folder means. + + // ClusterProviderRef names the ClusterProvider whose cluster holds the Kustomization, such as + // a management cluster running Flux for the source cluster. Omitted, it is the GitTarget's + // own spec.clusterProviderRef. The provider must admit this GitTarget's namespace + // (spec.accessFrom), and its identity needs get on the Kustomization and the GitRepository and + // patch on both. A provider without spec.kubeConfig is the operator's own cluster. + // +optional + // +kubebuilder:validation:XValidation:rule="self.name != ''",message="spec.applier.clusterProviderRef.name must not be empty" + ClusterProviderRef *meta.LocalObjectReference `json:"clusterProviderRef,omitempty"` + // KustomizationRef is a Flux Kustomization whose spec.sourceRef is a GitRepository on this // target's repository and branch. Its namespace defaults to the GitTarget's namespace. GitOps // Reverser asks it to apply the way `flux reconcile kustomization --with-source` does: it diff --git a/api/v1alpha3/helpers_test.go b/api/v1alpha3/helpers_test.go index 94a9be5c..9f7d10fc 100644 --- a/api/v1alpha3/helpers_test.go +++ b/api/v1alpha3/helpers_test.go @@ -321,3 +321,21 @@ func TestIsLocalSource_IsANameTest(t *testing.T) { }) } } + +// TestApplierCluster_DefaultsToTheSourceCluster: an omitted spec.applier.clusterProviderRef is the +// GitTarget's own source cluster, never the "default" provider, so a remote-source target keeps its +// meaning when the field is added to the API. +func TestApplierCluster_DefaultsToTheSourceCluster(t *testing.T) { + target := &GitTarget{Spec: GitTargetSpec{ + ClusterProviderRef: &meta.LocalObjectReference{Name: "prod-eu-1"}, + Applier: &GitTargetApplier{}, + }} + assert.Equal(t, "prod-eu-1", target.ApplierCluster()) + + target.Spec.Applier.ClusterProviderRef = &meta.LocalObjectReference{Name: "hub"} + assert.Equal(t, "hub", target.ApplierCluster()) + + target.Spec.ClusterProviderRef = nil + target.Spec.Applier.ClusterProviderRef = nil + assert.Equal(t, DefaultClusterProviderName, target.ApplierCluster()) +} diff --git a/api/v1alpha3/zz_generated.deepcopy.go b/api/v1alpha3/zz_generated.deepcopy.go index 1220014b..a43deb76 100644 --- a/api/v1alpha3/zz_generated.deepcopy.go +++ b/api/v1alpha3/zz_generated.deepcopy.go @@ -798,6 +798,11 @@ func (in *GitTarget) DeepCopyObject() runtime.Object { // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *GitTargetApplier) DeepCopyInto(out *GitTargetApplier) { *out = *in + if in.ClusterProviderRef != nil { + in, out := &in.ClusterProviderRef, &out.ClusterProviderRef + *out = new(meta.LocalObjectReference) + **out = **in + } out.KustomizationRef = in.KustomizationRef if in.QuietPeriod != nil { in, out := &in.QuietPeriod, &out.QuietPeriod diff --git a/cmd/main.go b/cmd/main.go index d4863e15..4b4a5bf6 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -184,7 +184,7 @@ func main() { // The branch workers decide when a GitTarget's applier may apply; the dispatcher delivers it, // off their queues, to the Flux objects in each target's source cluster. applyDispatcher := applier.NewDispatcher( - mgr.GetClient(), watchMgr.SourceDynamicClient, mgr.GetEventRecorderFor("gittarget"), + mgr.GetClient(), watchMgr.ApplierDynamicClient, mgr.GetEventRecorderFor("gittarget"), ctrl.Log.WithName("applier")) fatalIfErr(mgr.Add(applyDispatcher), "unable to add apply request dispatcher to manager") workerManager.SetApplyRequester(applyDispatcher) @@ -354,7 +354,7 @@ func main() { Recorder: mgr.GetEventRecorderFor("gittarget"), GitRefreshInterval: cfg.gitRefreshInterval, - ApplierClients: watchMgr.SourceDynamicClient, + ApplierClients: watchMgr.ApplierDynamicClient, ApplyRequests: applyDispatcher, }).SetupWithManager(mgr); err != nil { setupLog.Error(err, "unable to create controller", "controller", "GitTarget") diff --git a/config/crd/bases/configbutler.ai_gittargets.yaml b/config/crd/bases/configbutler.ai_gittargets.yaml index 15b1b78e..6e4f27b2 100644 --- a/config/crd/bases/configbutler.ai_gittargets.yaml +++ b/config/crd/bases/configbutler.ai_gittargets.yaml @@ -120,8 +120,26 @@ spec: refused edit (spec.onRefusal: Reapply), and, once the branch has been quiet for quietPeriod, to keep the applier current with its own pushes. Switch off the applier's own triggers (short intervals, Git-host webhooks into it), or they go on applying without that check. The - reference is looked up in the source cluster, with the identity the ClusterProvider uses. + reference is looked up in the cluster spec.applier.clusterProviderRef names, by default the + source cluster, with the identity that ClusterProvider uses. properties: + clusterProviderRef: + description: |- + ClusterProviderRef names the ClusterProvider whose cluster holds the Kustomization, such as + a management cluster running Flux for the source cluster. Omitted, it is the GitTarget's + own spec.clusterProviderRef. The provider must admit this GitTarget's namespace + (spec.accessFrom), and its identity needs get on the Kustomization and the GitRepository and + patch on both. A provider without spec.kubeConfig is the operator's own cluster. + properties: + name: + description: Name of the referent. + type: string + required: + - name + type: object + x-kubernetes-validations: + - message: spec.applier.clusterProviderRef.name must not be empty + rule: self.name != '' kustomizationRef: description: |- KustomizationRef is a Flux Kustomization whose spec.sourceRef is a GitRepository on this diff --git a/docs/configuration.md b/docs/configuration.md index 7bc42efa..e37e4767 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1132,13 +1132,30 @@ the API does not accept them yet: - **A webhook (`webhook`)** for any other tool: a POST of `{"repository", "branch", "commit", "reason"}`, signed the way Flux's `generic-hmac` `Receiver` checks. -**Where the reference is looked up.** In the source cluster, the cluster the `Kustomization` applies -to, with the identity the [`ClusterProvider`](#clusterprovider) already uses. That identity needs `get` -and `patch` on `kustomizations` (`kustomize.toolkit.fluxcd.io`) and `gitrepositories` -(`source.toolkit.fluxcd.io`), and `create` on `selfsubjectaccessreviews` to check it. For the -operator's own cluster the chart grants them with `applierAccess.flux: true`; for a remote source -cluster, grant the same to the kubeconfig's identity. A `Kustomization` that lives in a cluster other -than the one it applies to is not supported. +**Where the reference is looked up.** In the cluster `spec.applier.clusterProviderRef` names, with +the identity that [`ClusterProvider`](#clusterprovider) uses. Omitted, it is the `GitTarget`'s own +`spec.clusterProviderRef`, the source cluster. Name another provider when Flux runs in a management +cluster and applies to the source cluster from there; a provider without `spec.kubeConfig` is the +operator's own cluster. That provider's `spec.accessFrom` must admit the `GitTarget`'s namespace, +as for mirroring, or `ApplierConfigured` is `False` with `NamespaceNotAuthorized` (or +`ClusterProviderNotFound`) and nothing is asked. + +```yaml +spec: + clusterProviderRef: + name: prod-eu-1 # mirrored from here + applier: + clusterProviderRef: + name: management # Flux runs here + kustomizationRef: + namespace: flux-system + name: prod-eu-1-apps +``` + +The identity needs `get` and `patch` on `kustomizations` (`kustomize.toolkit.fluxcd.io`) and +`gitrepositories` (`source.toolkit.fluxcd.io`), and `create` on `selfsubjectaccessreviews` to check +it. For the operator's own cluster the chart grants them with `applierAccess.flux: true`; for a +remote cluster, grant the same to the kubeconfig's identity. **When a request is made.** Only on a quiet branch: no open commit window and no pending writes on the branch worker, or on any other worker writing the same repository and branch through another @@ -1202,9 +1219,11 @@ status: | Reason | Means | | --- | --- | -| `ApplierNotFound` | No such `Kustomization` in the source cluster, or Flux is not installed there | +| `ApplierNotFound` | No such `Kustomization` in the applier's cluster, or Flux is not installed there | | `SourceMismatch` | The `Kustomization` applies a different repository or branch, or a source that is not a `GitRepository` | -| `Forbidden` | The source-cluster identity may not patch the `Kustomization` or its `GitRepository` | +| `Forbidden` | The applier cluster's identity may not patch the `Kustomization` or its `GitRepository` | +| `NamespaceNotAuthorized` | `spec.applier.clusterProviderRef` names a provider whose `accessFrom` does not admit this namespace | +| `ClusterProviderNotFound` | `spec.applier.clusterProviderRef` names a provider that does not exist | It is checked on the target's reconcile tick rather than watched, so a cluster without the Flux CRDs installed costs nothing. Each delivered request also records an `ApplyRequested` Event on the diff --git a/docs/flux.md b/docs/flux.md index 0aade3e8..aebb1ac1 100644 --- a/docs/flux.md +++ b/docs/flux.md @@ -178,8 +178,14 @@ spec: namespace: flux-system # default: this GitTarget's namespace name: checkout quietPeriod: 30s # default 30s, at most 1h + clusterProviderRef: # default: this GitTarget's spec.clusterProviderRef + name: management ``` +`clusterProviderRef` is for Flux running in another cluster than the one it applies to, such as a +management cluster with a `Kustomization` per workload cluster. Leave it out when Flux runs in the +source cluster. + `quietPeriod` is how long the branch must stay quiet after GitOps Reverser's own pushes before Flux is asked to catch up with them. Every new write restarts it. A short one keeps Flux's artifact current; a long one makes it less likely that an edit arrives while that catch-up apply runs. @@ -189,14 +195,15 @@ The target reports `ApplierConfigured`, and `Ready` includes it: | Reason | Means | Readiness | | --- | --- | --- | | `Succeeded` | The `Kustomization` exists, applies this repository and branch, and may be patched | ready | -| `ApplierNotFound` | No such `Kustomization` in the source cluster, or Flux is not installed there | progressing: Flux objects are often created after the `GitTarget` | +| `ApplierNotFound` | No such `Kustomization` in the applier's cluster, or Flux is not installed there | progressing: Flux objects are often created after the `GitTarget` | | `SourceMismatch` | One of the [`GitRepository` rules](#the-gitrepository) is broken, or the source is not a `GitRepository` | stalled: needs a human | -| `Forbidden` | The source-cluster identity may not patch the `Kustomization` or its `GitRepository` | stalled: needs a human | +| `Forbidden` | The applier cluster's identity may not patch the `Kustomization` or its `GitRepository` | stalled: needs a human | +| `NamespaceNotAuthorized`, `ClusterProviderNotFound` | `spec.applier.clusterProviderRef` names a provider that does not admit this namespace, or does not exist | stalled: needs a human | ## Permissions -GitOps Reverser looks the `Kustomization` up in the **source cluster** (the cluster it applies to) -with the identity of the target's `ClusterProvider`. That identity needs: +GitOps Reverser looks the `Kustomization` up in the cluster of `spec.applier.clusterProviderRef`, +by default the **source cluster**, with the identity of that `ClusterProvider`. That identity needs: | API group | Resource | Verbs | | --- | --- | --- | @@ -206,7 +213,7 @@ with the identity of the target's `ClusterProvider`. That identity needs: For the operator's own cluster, install the chart with `applierAccess.flux: true`, or apply [`config/rbac/flux-applier-role.yaml`](../config/rbac/flux-applier-role.yaml) and its binding. For -a remote source cluster, grant the same to the identity in the `ClusterProvider`'s kubeconfig. +a remote cluster, grant the same to the identity in the `ClusterProvider`'s kubeconfig. The chart's grant is cluster-wide for convenience. Requests only ever read and patch the `Kustomization` and `GitRepository` a `GitTarget` names, so a `Role` in each namespace that holds diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index a6f28b5c..7fc8fb7c 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -53,8 +53,8 @@ const shortCommitLength = 8 // errNotRunning fails an attempt made before the dispatcher started or after it stopped. var errNotRunning = errors.New("the apply request dispatcher is not running") -// SourceClients resolves a source cluster's dynamic client by ClusterProvider name -// (GitTarget.SourceCluster()). In production it is (*watch.Manager).SourceDynamicClient. +// SourceClients resolves the dynamic client of the cluster an applier lives in, by ClusterProvider +// name (GitTarget.ApplierCluster()). In production it is (*watch.Manager).ApplierDynamicClient. type SourceClients func(ctx context.Context, clusterID string) (dynamic.Interface, error) // DefaultAttemptTimeout bounds one attempt: a source cluster that does not answer fails it, and the @@ -305,9 +305,10 @@ func (d *Dispatcher) recheck(ctx context.Context, req git.ApplyRequest) ([]eligi // recheckTarget reads one GitTarget again and returns it when it may still ask the applier for its // reason, or nil when it may not. // -// A ClusterProvider's source client is shared by every GitTarget that declared the cluster, so a -// client being available is no permission to use it: a GitTarget its ClusterProvider does not admit -// would otherwise drive the applier on the strength of another GitTarget's admission. +// A ClusterProvider's client is shared by every GitTarget that uses the cluster, so a client being +// available is no permission to use it: a GitTarget its ClusterProviders do not admit (its own, and +// the applier's when that is another) would otherwise drive the applier on the strength of another +// GitTarget's admission. func (d *Dispatcher) recheckTarget( ctx context.Context, req git.ApplyRequest, candidate git.ApplyTarget, ) (*v1alpha3.GitTarget, error) { @@ -324,13 +325,13 @@ func (d *Dispatcher) recheckTarget( log.V(1).Info("GitTarget is no longer the one that named this applier on this branch; not asking for it") return nil, nil //nolint:nilnil // as above } - decision, err := authz.GitTargetAdmitted(ctx, d.reader, target) + decision, err := authz.ApplierAdmitted(ctx, d.reader, target) if err != nil { - return nil, fmt.Errorf("check GitTarget %s against its ClusterProvider: %w", candidate.Ref, err) + return nil, fmt.Errorf("check GitTarget %s against its ClusterProviders: %w", candidate.Ref, err) } if !decision.Allowed { - log.Info("GitTarget is not admitted by its ClusterProvider; not asking its applier", - "reason", decision.Reason) + log.Info("GitTarget is not admitted by its ClusterProviders; not asking its applier", + "reason", decision.Reason, "message", decision.Message) return nil, nil //nolint:nilnil // as above } if candidate.Reason == v1alpha3.ApplyReasonRefusal && !git.ConsentsToReapply(target) { @@ -351,7 +352,7 @@ func namesApplier(target *v1alpha3.GitTarget, req git.ApplyRequest) bool { if namespace == "" { namespace = target.Namespace } - return target.SourceCluster() == req.Applier.Cluster && namespace == req.Applier.Namespace && + return target.ApplierCluster() == req.Applier.Cluster && namespace == req.Applier.Namespace && ref.Name == req.Applier.Name } diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index ab93f6e0..d33a8127 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -249,6 +249,48 @@ func TestDispatcher_LooksTheApplierUpInItsCluster(t *testing.T) { assert.Equal(t, []string{"gitrepositories " + targetNS}, f.patches()) } +// appliedFrom puts a GitTarget's applier in the cluster another ClusterProvider names. +func appliedFrom(target *v1alpha3.GitTarget, cluster string) *v1alpha3.GitTarget { + target.Spec.Applier.ClusterProviderRef = &fluxmeta.LocalObjectReference{Name: cluster} + return target +} + +// TestDispatcher_AsksAnApplierInItsOwnCluster: a GitTarget mirroring prod-eu-1 whose Kustomization +// lives in a management cluster is asked through that cluster's client, never the source's. +func TestDispatcher_AsksAnApplierInItsOwnCluster(t *testing.T) { + hub := git.ApplierRef{Cluster: "hub", Namespace: fluxSystemNS, Name: "checkout"} + f := newDispatcherFixture(t, + []client.Object{appliedFrom(gitTarget("a", checkoutRef(), "prod-eu-1"), "hub"), clusterProvider("hub", targetNS)}, + behindSource()..., + ) + + assert.Equal(t, git.ApplyDelivered, f.attempt(request(hub, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) + assert.Equal(t, []string{"hub"}, f.source.clusters) + + // The same Kustomization asked as if it lived in the source cluster is not this target's applier. + f.source.clusters = nil + sourceSide := git.ApplierRef{Cluster: "prod-eu-1", Namespace: fluxSystemNS, Name: "checkout"} + assert.Equal(t, git.ApplyDropped, f.attempt(request(sourceSide, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) + assert.Empty(t, f.source.clusters) +} + +// TestDispatcher_AnApplierClusterThatDoesNotAdmitAsksNothing: the applier's ClusterProvider gates who +// may wield its credential exactly as a source's does, so a GitTarget its accessFrom does not admit +// asks nothing, though its own source admits it. +func TestDispatcher_AnApplierClusterThatDoesNotAdmitAsksNothing(t *testing.T) { + hub := git.ApplierRef{Cluster: "hub", Namespace: fluxSystemNS, Name: "checkout"} + f := newDispatcherFixture(t, + []client.Object{ + appliedFrom(gitTarget("a", checkoutRef(), "prod-eu-1"), "hub"), + clusterProvider("hub", "some-other-namespace"), + }, + behindSource()..., + ) + + assert.Equal(t, git.ApplyDropped, f.attempt(request(hub, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) + assert.Empty(t, f.source.clusters, "the hub's client is never even asked for") +} + // TestDispatcher_AFailureIsReportedAndNeverRetried: retrying is the branch worker's decision, on a // quiet branch. The dispatcher reports the failure, and records ApplyRequestFailed only for the // first failure of a run. diff --git a/internal/authz/clusterprovider_admission.go b/internal/authz/clusterprovider_admission.go index e4e5e210..9f50124b 100644 --- a/internal/authz/clusterprovider_admission.go +++ b/internal/authz/clusterprovider_admission.go @@ -68,33 +68,61 @@ func GitTargetAdmitted( target *configv1alpha3.GitTarget, ) (Decision, error) { providerName := target.SourceCluster() + return namespaceAdmitted(ctx, reader, target.Namespace, providerName, fmt.Sprintf( + "referenced ClusterProvider %q was not found; a GitTarget may mirror a source cluster "+ + "only through an existing ClusterProvider. The operator never creates one: declare it "+ + "yourself, or let the chart render %q via clusterProvider.createDefault", + providerName, configv1alpha3.DefaultClusterProviderName)) +} + +// ApplierAdmitted reports whether target may ask its applier to apply: its namespace is admitted +// by its own ClusterProvider (GitTargetAdmitted) and, when spec.applier.clusterProviderRef names +// another one, by that one too. The applier is reached with that provider's credential, so the +// provider's accessFrom decides who may wield it, exactly as for mirroring. Call it only with +// spec.applier set. +func ApplierAdmitted( + ctx context.Context, + reader client.Reader, + target *configv1alpha3.GitTarget, +) (Decision, error) { + decision, err := GitTargetAdmitted(ctx, reader, target) + if err != nil || !decision.Allowed { + return decision, err + } + providerName := target.ApplierCluster() + if providerName == target.SourceCluster() { + return decision, nil + } + return namespaceAdmitted(ctx, reader, target.Namespace, providerName, fmt.Sprintf( + "ClusterProvider %q named by spec.applier.clusterProviderRef was not found", providerName)) +} +// namespaceAdmitted is the check both admissions share: the provider exists, and its accessFrom +// admits namespace. notFound is the denial message for a provider that does not exist. +func namespaceAdmitted( + ctx context.Context, + reader client.Reader, + namespace, providerName, notFound string, +) (Decision, error) { var provider configv1alpha3.ClusterProvider if err := reader.Get(ctx, k8stypes.NamespacedName{Name: providerName}, &provider); err != nil { if apierrors.IsNotFound(err) { - return Decision{ - Reason: ReasonClusterProviderNotFound, - Message: fmt.Sprintf( - "referenced ClusterProvider %q was not found; a GitTarget may mirror a source cluster "+ - "only through an existing ClusterProvider. The operator never creates one: declare it "+ - "yourself, or let the chart render %q via clusterProvider.createDefault", - providerName, configv1alpha3.DefaultClusterProviderName), - }, nil + return Decision{Reason: ReasonClusterProviderNotFound, Message: notFound}, nil } return Decision{}, fmt.Errorf("read ClusterProvider %q: %w", providerName, err) } nsLabels := map[string]string{} var ns corev1.Namespace - if err := reader.Get(ctx, k8stypes.NamespacedName{Name: target.Namespace}, &ns); err != nil { + if err := reader.Get(ctx, k8stypes.NamespacedName{Name: namespace}, &ns); err != nil { if !apierrors.IsNotFound(err) { - return Decision{}, fmt.Errorf("read namespace %q: %w", target.Namespace, err) + return Decision{}, fmt.Errorf("read namespace %q: %w", namespace, err) } } else { nsLabels = ns.Labels } - allowed, selErr := provider.AllowsNamespace(target.Namespace, nsLabels) + allowed, selErr := provider.AllowsNamespace(namespace, nsLabels) if selErr != nil { return Decision{ Reason: ReasonNamespaceNotAuthorized, @@ -107,7 +135,7 @@ func GitTargetAdmitted( Reason: ReasonNamespaceNotAuthorized, Message: fmt.Sprintf( "namespace %q is not permitted to reference ClusterProvider %q (spec.accessFrom)", - target.Namespace, providerName), + namespace, providerName), }, nil } diff --git a/internal/authz/clusterprovider_admission_test.go b/internal/authz/clusterprovider_admission_test.go index a74242e6..deb835c0 100644 --- a/internal/authz/clusterprovider_admission_test.go +++ b/internal/authz/clusterprovider_admission_test.go @@ -300,3 +300,71 @@ func TestGitTargetAdmitted_ReadErrorsSurfaceAsError(t *testing.T) { }) } } + +// TestApplierAdmitted: an applier in another cluster is reached with that ClusterProvider's +// credential, so its accessFrom must admit the GitTarget too, and the source's denial comes first. +func TestApplierAdmitted(t *testing.T) { + admits := func(name string, namespaces ...string) *configv1alpha3.ClusterProvider { + return providerNamed(name, &configv1alpha3.NamespaceMatcher{Names: namespaces}) + } + withApplier := func(source, applierCluster string) *configv1alpha3.GitTarget { + target := targetIn(source) + target.Spec.Applier = &configv1alpha3.GitTargetApplier{} + if applierCluster != "" { + target.Spec.Applier.ClusterProviderRef = &meta.LocalObjectReference{Name: applierCluster} + } + return target + } + tests := []struct { + name string + objects []client.Object + target *configv1alpha3.GitTarget + wantAllowed bool + wantReason string + wantMessage string + }{ + { + name: "omitted, the applier's provider is the source's", + objects: []client.Object{admits("prod-eu-1", testNS)}, + target: withApplier("prod-eu-1", ""), + wantAllowed: true, + }, + { + name: "another provider that admits the namespace", + objects: []client.Object{admits("prod-eu-1", testNS), admits("hub", testNS)}, + target: withApplier("prod-eu-1", "hub"), + wantAllowed: true, + }, + { + name: "another provider that does not admit the namespace", + objects: []client.Object{admits("prod-eu-1", testNS), admits("hub", "elsewhere")}, + target: withApplier("prod-eu-1", "hub"), + wantReason: ReasonNamespaceNotAuthorized, + wantMessage: `ClusterProvider "hub"`, + }, + { + name: "another provider that does not exist", + objects: []client.Object{admits("prod-eu-1", testNS)}, + target: withApplier("prod-eu-1", "hub"), + wantReason: ReasonClusterProviderNotFound, + wantMessage: "spec.applier.clusterProviderRef", + }, + { + name: "the source's denial comes first", + objects: []client.Object{admits("prod-eu-1", "elsewhere"), admits("hub", testNS)}, + target: withApplier("prod-eu-1", "hub"), + wantReason: ReasonNamespaceNotAuthorized, + wantMessage: `ClusterProvider "prod-eu-1"`, + }, + } + for _, tc := range tests { + t.Run(tc.name, func(t *testing.T) { + reader := fake.NewClientBuilder().WithScheme(admissionScheme(t)).WithObjects(tc.objects...).Build() + decision, err := ApplierAdmitted(context.Background(), reader, tc.target) + require.NoError(t, err) + assert.Equal(t, tc.wantAllowed, decision.Allowed, decision.Message) + assert.Equal(t, tc.wantReason, decision.Reason) + assert.Contains(t, decision.Message, tc.wantMessage) + }) + } +} diff --git a/internal/controller/gittarget_applier.go b/internal/controller/gittarget_applier.go index 8fedba22..ce99bb8b 100644 --- a/internal/controller/gittarget_applier.go +++ b/internal/controller/gittarget_applier.go @@ -12,11 +12,12 @@ import ( configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" "github.com/ConfigButler/gitops-reverser/internal/applier" + "github.com/ConfigButler/gitops-reverser/internal/authz" "github.com/ConfigButler/gitops-reverser/internal/types" ) // GitTargetConditionApplierConfigured reports whether spec.applier names an object GitOps Reverser -// can ask to apply: it exists in the source cluster, applies this target's repository and branch, +// can ask to apply: it exists in the applier's cluster, applies this target's repository and branch, // and may be patched. Present only while spec.applier is set. const GitTargetConditionApplierConfigured = "ApplierConfigured" @@ -65,9 +66,9 @@ func (r *GitTargetReconciler) observeApplier( return value, true } -// applierVerdict checks the applier in the source cluster. A check that could not be made keeps the -// verdict already published, since a blip reading the source cluster says nothing about the -// applier; with nothing published it is Unknown. +// applierVerdict checks the applier in its cluster (spec.applier.clusterProviderRef, by default the +// source cluster). A check that could not be made keeps the verdict already published, since a blip +// reading that cluster says nothing about the applier; with nothing published it is Unknown. func (r *GitTargetReconciler) applierVerdict( ctx context.Context, target *configbutleraiv1alpha3.GitTarget, @@ -80,13 +81,23 @@ func (r *GitTargetReconciler) applierVerdict( return conditionValue{ Status: metav1.ConditionUnknown, Reason: GitTargetReasonNotChecked, - Message: fmt.Sprintf("Kustomization %s has not been checked: no source-cluster client is wired", ref), + Message: fmt.Sprintf("Kustomization %s has not been checked: no cluster client is wired", ref), } } - client, err := r.ApplierClients(ctx, target.SourceCluster()) + cluster := target.ApplierCluster() + if cluster != target.SourceCluster() { + decision, err := authz.ApplierAdmitted(ctx, r.Client, target) + if err != nil { + return keptOrUnknown(target, fmt.Sprintf("Cannot check ClusterProvider %q: %v", cluster, err)) + } + if !decision.Allowed { + return conditionValue{Status: metav1.ConditionFalse, Reason: decision.Reason, Message: decision.Message} + } + } + client, err := r.ApplierClients(ctx, cluster) if err != nil { return keptOrUnknown(target, - fmt.Sprintf("Cannot reach the source cluster to check Kustomization %s: %v", ref, err)) + fmt.Sprintf("Cannot reach cluster %q to check Kustomization %s: %v", cluster, ref, err)) } newFlux := r.newFlux if newFlux == nil { diff --git a/internal/controller/gittarget_applier_test.go b/internal/controller/gittarget_applier_test.go index da836d59..b6a9aaa6 100644 --- a/internal/controller/gittarget_applier_test.go +++ b/internal/controller/gittarget_applier_test.go @@ -22,6 +22,7 @@ import ( configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" "github.com/ConfigButler/gitops-reverser/internal/applier" + "github.com/ConfigButler/gitops-reverser/internal/authz" "github.com/ConfigButler/gitops-reverser/internal/types" ) @@ -240,3 +241,56 @@ func TestApplierReadiness_StallsOnlyWhatNeedsAHuman(t *testing.T) { }) } } + +// TestObserveApplier_AnApplierInAnotherCluster: spec.applier.clusterProviderRef moves the check to +// that cluster, behind that provider's accessFrom, exactly as the dispatcher asks. +func TestObserveApplier_AnApplierInAnotherCluster(t *testing.T) { + admitting := func(name string, namespaces ...string) *configbutleraiv1alpha3.ClusterProvider { + return &configbutleraiv1alpha3.ClusterProvider{ + ObjectMeta: metav1.ObjectMeta{Name: name}, + Spec: configbutleraiv1alpha3.ClusterProviderSpec{ + AccessFrom: &configbutleraiv1alpha3.NamespaceMatcher{Names: namespaces}, + }, + } + } + cases := []struct { + name string + hub *configbutleraiv1alpha3.ClusterProvider + wantStatus metav1.ConditionStatus + wantReason string + wantClients []string + }{ + {name: "a provider that admits the namespace is asked", hub: admitting("hub", "team-a"), + wantStatus: metav1.ConditionTrue, wantReason: ReasonSucceeded, wantClients: []string{"hub"}}, + {name: "a provider that does not admit it is never asked", hub: admitting("hub", "team-b"), + wantStatus: metav1.ConditionFalse, wantReason: authz.ReasonNamespaceNotAuthorized}, + {name: "a provider that does not exist is never asked", + wantStatus: metav1.ConditionFalse, wantReason: authz.ReasonClusterProviderNotFound}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + objects := []runtime.Object{admitting(configbutleraiv1alpha3.DefaultClusterProviderName, "team-a")} + if tc.hub != nil { + objects = append(objects, tc.hub) + } + target := applierTarget(true) + target.Spec.Applier.ClusterProviderRef = &meta.LocalObjectReference{Name: "hub"} + r := applierReconciler(applierFakeSource(applierFluxObjects("main")...), nil, true) + r.Client = newGitProviderTestClient(t, objects...) + var asked []string + source := r.ApplierClients + r.ApplierClients = func(ctx context.Context, cluster string) (dynamic.Interface, error) { + asked = append(asked, cluster) + return source(ctx, cluster) + } + + value, present := r.observeApplier(context.Background(), beginStatus(nil, nil, target), target, + applierProvider()) + + require.True(t, present) + assert.Equal(t, tc.wantStatus, value.Status, value.Message) + assert.Equal(t, tc.wantReason, value.Reason) + assert.Equal(t, tc.wantClients, asked) + }) + } +} diff --git a/internal/controller/gittarget_controller.go b/internal/controller/gittarget_controller.go index 35497478..e177daeb 100644 --- a/internal/controller/gittarget_controller.go +++ b/internal/controller/gittarget_controller.go @@ -1706,7 +1706,8 @@ func (r *GitTargetReconciler) gitProviderToGitTargets( return requests } -// clusterProviderToGitTargets maps a ClusterProvider event to every GitTarget that references it, +// clusterProviderToGitTargets maps a ClusterProvider event to every GitTarget that references it, as +// its source or as its applier's cluster, // across ALL namespaces (the provider is cluster-scoped). It re-enqueues dependents when the // provider's Ready flips or its accessFrom policy changes, so the projected // ClusterProviderReady and the namespace-authorization refusal converge without waiting for the @@ -1724,7 +1725,8 @@ func (r *GitTargetReconciler) clusterProviderToGitTargets( var requests []ctrlreconcile.Request for i := range targets.Items { t := &targets.Items[i] - if t.SourceCluster() != obj.GetName() { + usesApplier := t.Spec.Applier != nil && t.ApplierCluster() == obj.GetName() + if t.SourceCluster() != obj.GetName() && !usesApplier { continue } requests = append(requests, ctrlreconcile.Request{ diff --git a/internal/git/apply_request.go b/internal/git/apply_request.go index 01800d11..4f3c11fa 100644 --- a/internal/git/apply_request.go +++ b/internal/git/apply_request.go @@ -15,7 +15,7 @@ import ( // ApplierRef is one applier, as the GitTargets on a branch name it: a Flux Kustomization in a // source cluster. Several GitTargets may name one applier, and it is asked once for all of them. type ApplierRef struct { - // Cluster is the source cluster the applier lives in (GitTarget.SourceCluster()). + // Cluster is the ClusterProvider whose cluster the applier lives in (GitTarget.ApplierCluster()). Cluster string // Namespace and Name are the Kustomization's, the namespace defaulted to the GitTarget's own. Namespace string @@ -31,7 +31,7 @@ func applierOf(target *v1alpha3.GitTarget) ApplierRef { if namespace == "" { namespace = target.Namespace } - return ApplierRef{Cluster: target.SourceCluster(), Namespace: namespace, Name: ref.Name} + return ApplierRef{Cluster: target.ApplierCluster(), Namespace: namespace, Name: ref.Name} } // ApplyRequest is one attempt, decided by a branch worker on a quiet branch, to have one applier diff --git a/internal/git/apply_trigger_test.go b/internal/git/apply_trigger_test.go index b74bce70..a53f8e86 100644 --- a/internal/git/apply_trigger_test.go +++ b/internal/git/apply_trigger_test.go @@ -432,3 +432,21 @@ func TestApplyTrigger_ARejectedPushOwesTheReplayedTip(t *testing.T) { assert.Equal(t, head, requests[0].Commit, "the replayed tip, not the foreign commit under it") assert.Equal(t, "from-another-writer", gitOut(t, f.repoDir, "show", head+":OUTSIDE.md")) } + +// TestApplierOf_KeysTheApplierByItsOwnCluster: one Kustomization name in two clusters is two +// appliers, and spec.applier.clusterProviderRef, not the source, decides which cluster. +func TestApplierOf_KeysTheApplierByItsOwnCluster(t *testing.T) { + target := &configv1alpha3.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Namespace: "team-a", Name: "apps"}, + Spec: configv1alpha3.GitTargetSpec{ + ClusterProviderRef: &meta.LocalObjectReference{Name: "prod-eu-1"}, + Applier: &configv1alpha3.GitTargetApplier{ + KustomizationRef: meta.NamespacedObjectReference{Name: "apps"}, + }, + }, + } + assert.Equal(t, ApplierRef{Cluster: "prod-eu-1", Namespace: "team-a", Name: "apps"}, applierOf(target)) + + target.Spec.Applier.ClusterProviderRef = &meta.LocalObjectReference{Name: "hub"} + assert.Equal(t, ApplierRef{Cluster: "hub", Namespace: "team-a", Name: "apps"}, applierOf(target)) +} diff --git a/internal/watch/cluster_context.go b/internal/watch/cluster_context.go index 6cd81e32..cecb35ca 100644 --- a/internal/watch/cluster_context.go +++ b/internal/watch/cluster_context.go @@ -494,7 +494,7 @@ func (m *Manager) clusterRESTConfigLocked(ctx context.Context, cc *clusterContex cc.configVersion = inClusterConfigVersion return cfg, nil } - cfg, version, inCluster, err := m.resolveSourceConfig(ctx, cc) + cfg, version, inCluster, err := m.resolveSourceConfig(ctx, cc.id) if err != nil { return nil, err } @@ -524,15 +524,15 @@ func inClusterRESTConfig() (*rest.Config, error) { // opaque version token, whether the provider resolved in-cluster, and the error. func (m *Manager) resolveSourceConfig( ctx context.Context, - cc *clusterContext, + clusterID string, ) (*rest.Config, string, bool, error) { if m.SourceClusters == nil { return nil, "", false, fmt.Errorf( - "cannot reach source cluster %q: no source-cluster resolver configured", cc.id) + "cannot reach source cluster %q: no source-cluster resolver configured", clusterID) } - resolved, version, err := m.SourceClusters.ResolveSourceCluster(ctx, cc.id) + resolved, version, err := m.SourceClusters.ResolveSourceCluster(ctx, clusterID) if err != nil { - return nil, "", false, fmt.Errorf("resolve source cluster %q: %w", cc.id, err) + return nil, "", false, fmt.Errorf("resolve source cluster %q: %w", clusterID, err) } if resolved == nil { // The provider omits kubeConfig: it names the operator's own cluster. @@ -558,7 +558,7 @@ func (m *Manager) refreshClusterCredentials(ctx context.Context, cc *clusterCont if cc.isLocal() { return } - cfg, version, inCluster, err := m.resolveSourceConfig(ctx, cc) + cfg, version, inCluster, err := m.resolveSourceConfig(ctx, cc.id) if err != nil { if isDefinitiveCredentialFailure(err) && m.dropClusterClients(cc) { m.invalidateClusterWatches(cc.id) @@ -720,6 +720,33 @@ func (m *Manager) SourceDynamicClient(ctx context.Context, clusterID string) (dy return m.clusterDynamicClient(ctx, clusterID) } +// ApplierDynamicClient returns the client an applier in clusterID is asked through, with that +// ClusterProvider's identity, as SourceDynamicClient's is. An applier may live in a cluster no +// GitTarget mirrors from, such as a management cluster that runs Flux for remote ones. No context +// is created for such a cluster, because nothing would release it (see SourceDynamicClient): the +// client is built for this call from the provider as it resolves now, and dropped with it. +func (m *Manager) ApplierDynamicClient(ctx context.Context, clusterID string) (dynamic.Interface, error) { + if clusterID == configPlaneClusterID { + return nil, errors.New("no cluster named: the config plane is not a ClusterProvider") + } + if m.sourceClusterDeclared(clusterID) { + return m.clusterDynamicClient(ctx, clusterID) + } + cfg, _, inCluster, err := m.resolveSourceConfig(ctx, clusterID) + if err != nil { + return nil, err + } + // Tests inject a fake client for the local cluster without a REST config at all. + if inCluster && m.dynamicClient != nil { + return m.dynamicClient, nil + } + dc, err := dynamic.NewForConfig(cfg) + if err != nil { + return nil, fmt.Errorf("build dynamic client for cluster %q: %w", describeCluster(clusterID), err) + } + return dc, nil +} + // sourceClusterDeclared reports whether any GitTarget's Declare captured this cluster id. func (m *Manager) sourceClusterDeclared(clusterID string) bool { for _, id := range m.watchPlane().clusters { diff --git a/internal/watch/cluster_model_test.go b/internal/watch/cluster_model_test.go index 4f5d2715..5eaa3764 100644 --- a/internal/watch/cluster_model_test.go +++ b/internal/watch/cluster_model_test.go @@ -68,7 +68,7 @@ func TestResolveSourceConfig_InClusterIsResolvedFromTheProvider(t *testing.T) { m := &Manager{Log: logr.Discard(), SourceClusters: tt.resolver} cc := m.cluster(tt.providerName) - cfg, version, inCluster, err := m.resolveSourceConfig(context.Background(), cc) + cfg, version, inCluster, err := m.resolveSourceConfig(context.Background(), cc.id) if tt.wantInCluster { // There is no cluster under a unit test, so the in-cluster branch surfaces // ctrl.GetConfig's failure. Either way it must have taken the in-cluster path @@ -95,7 +95,7 @@ func TestResolveSourceConfig_InClusterIsResolvedFromTheProvider(t *testing.T) { func TestResolveSourceConfig_Failures(t *testing.T) { t.Run("no resolver configured", func(t *testing.T) { m := &Manager{Log: logr.Discard()} - _, _, inCluster, err := m.resolveSourceConfig(context.Background(), m.cluster("prod-eu-1")) + _, _, inCluster, err := m.resolveSourceConfig(context.Background(), "prod-eu-1") require.Error(t, err) assert.Contains(t, err.Error(), "no source-cluster resolver configured") assert.False(t, inCluster) @@ -103,7 +103,7 @@ func TestResolveSourceConfig_Failures(t *testing.T) { t.Run("resolver error is propagated, not defaulted to in-cluster", func(t *testing.T) { m := &Manager{Log: logr.Discard(), SourceClusters: stubResolver{err: errors.New("boom")}} - _, _, inCluster, err := m.resolveSourceConfig(context.Background(), m.cluster("prod-eu-1")) + _, _, inCluster, err := m.resolveSourceConfig(context.Background(), "prod-eu-1") require.Error(t, err) assert.Contains(t, err.Error(), "resolve source cluster") assert.False(t, inCluster) diff --git a/internal/watch/source_dynamic_client_test.go b/internal/watch/source_dynamic_client_test.go index ebd5bba4..74c54112 100644 --- a/internal/watch/source_dynamic_client_test.go +++ b/internal/watch/source_dynamic_client_test.go @@ -57,3 +57,37 @@ func TestSourceDynamicClient_ServesADeclaredCluster(t *testing.T) { require.NotNil(t, cc) assert.Same(t, first, cc.dynamicClient, "it is the client the cluster's watches run on") } + +// TestApplierDynamicClient_ServesAnUndeclaredClusterWithoutKeepingIt: an applier may live in a +// cluster no GitTarget mirrors from. It is served, from the provider as it resolves now, and no +// cluster context is created, so nothing outlives the call. +func TestApplierDynamicClient_ServesAnUndeclaredClusterWithoutKeepingIt(t *testing.T) { + m := &Manager{ + Log: logr.Discard(), + SourceClusters: stubSourceClusterResolver{cfg: &rest.Config{Host: "https://hub.example"}, version: "v1"}, + } + + dc, err := m.ApplierDynamicClient(context.Background(), "hub") + require.NoError(t, err) + require.NotNil(t, dc) + assert.Nil(t, m.clusterContextByID("hub"), "an applier-only cluster creates no cluster context") +} + +// TestApplierDynamicClient_ReusesADeclaredClustersClient: a cluster some GitTarget mirrors from +// already has a client, with the same identity; the applier is asked through that one. +func TestApplierDynamicClient_ReusesADeclaredClustersClient(t *testing.T) { + m := &Manager{ + Log: logr.Discard(), + SourceClusters: stubSourceClusterResolver{cfg: &rest.Config{Host: "https://prod.example"}, version: "v1"}, + } + m.rememberGitTargetCluster(gd("checkout"), "prod-eu-1") + + source, err := m.SourceDynamicClient(context.Background(), "prod-eu-1") + require.NoError(t, err) + applier, err := m.ApplierDynamicClient(context.Background(), "prod-eu-1") + require.NoError(t, err) + assert.Same(t, source, applier) + + _, err = m.ApplierDynamicClient(context.Background(), configPlaneClusterID) + require.Error(t, err, "the config plane is not a ClusterProvider") +} From e24d2428162ea8cf169a1a8afcfaa2f350ea152f Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 05:40:52 +0000 Subject: [PATCH 27/36] feat(applier): the API wins an edit made while Flux applies An apply runs a few seconds after its stamp. An API edit in those seconds, to a field the Kustomization applies, was reverted by the apply and mirrored as Flux's commit, so the edit survived only in history. For an object both sides changed the API value wins; this applies that rule to the apply window. The window is now known: the dispatcher reports a delivered attempt only once Flux finished it (lastHandledReconcileAt carries the stamp, or the Kustomization attempted a new revision after a source stamp), at most 5 minutes, and the worker keeps it open 10 seconds more for the watch. Inside it, a write whose latest managedFields entry is kustomize-controller alone, to an object in the applier's folders that the API side edited after the attempt was sent, is withheld. Git keeps the edit and the CatchUp its push owes applies it back. Only proof withholds: a same-second tie with another manager could be the edit itself and is mirrored as before, and a DELETE never is. Every collision is logged and counted in apply_window_collisions_total{outcome="withheld"|"ambiguous"}. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/architecture.md | 15 +- docs/bi-directional.md | 5 +- docs/design/apply-trigger.md | 125 ++++++++-------- docs/flux.md | 24 ++-- docs/interpreting-metrics.md | 5 + internal/applier/dispatcher.go | 79 +++++++++-- internal/applier/dispatcher_test.go | 46 +++++- internal/applier/flux.go | 108 +++++++++++--- internal/applier/flux_guide_test.go | 4 +- internal/applier/flux_test.go | 76 +++++++++- internal/git/apply_trigger.go | 5 + internal/git/apply_window.go | 157 +++++++++++++++++++++ internal/git/apply_window_test.go | 154 ++++++++++++++++++++ internal/git/branch_worker.go | 3 + internal/git/types.go | 8 ++ internal/telemetry/apply_window.go | 23 +++ internal/telemetry/exporter.go | 1 + internal/watch/last_field_managers_test.go | 67 +++++++++ internal/watch/target_watch.go | 24 ++++ 19 files changed, 808 insertions(+), 121 deletions(-) create mode 100644 internal/git/apply_window.go create mode 100644 internal/git/apply_window_test.go create mode 100644 internal/telemetry/apply_window.go create mode 100644 internal/watch/last_field_managers_test.go diff --git a/docs/architecture.md b/docs/architecture.md index a9bb2e32..1f4330dc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1740,16 +1740,13 @@ Current limitations: - **GitOps Reverser is the fast path to a deployment.** With `spec.applier` set, the applier's own triggers are off, so while the operator is down a Git-side push reaches the cluster only through a manual sync, `flux reconcile`, or Flux's long backstop interval. -- **The applier must live in the cluster it applies to.** It is looked up in the source cluster - with the `ClusterProvider`'s identity, which rules out a hub Argo CD for now. - **Only Flux can be an applier yet.** An Argo CD `Application` keeps its own triggers until - `applicationRef` lands. -- **An apply request cannot reserve time.** The request is asynchronous: an edit made while the - requested apply runs, to a field the applier sets, is reverted and mirrored as the applier's - commit. The rule confines that to the seconds after a foreign push, a refusal with `Reapply`, or a - quiet period on a Flux branch; - [the apply trigger design](design/apply-trigger.md#the-apply-window-what-is-still-racy) weighs the - options to shrink it further. + `applicationRef` lands. The `Kustomization` may live in any cluster a `ClusterProvider` reaches + (`spec.applier.clusterProviderRef`), such as a management cluster. +- **An apply request cannot reserve time.** An edit made while the requested apply runs, to a field + the applier sets, is reverted on the cluster. Its value stays in Git, because the applier's write + over it is not mirrored, and the next apply restores it; until then the cluster runs the older + value. See [the apply window](design/apply-trigger.md#the-apply-window). - **Audit ingress uses a shared-CA trust boundary.** Named `/audit-webhook/` routes and the annotation-routed shared endpoint both require a client certificate signed by the audit CA, but that certificate is not bound to one provider. This is an accepted privileged-control-plane assumption, not diff --git a/docs/bi-directional.md b/docs/bi-directional.md index e72fa80b..60b04e9f 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -57,7 +57,10 @@ flowchart LR An apply is safe only when nothing live is missing from the commit being applied. Between an edit and its push, the edit is live in the cluster and absent from Git: it sits in an open commit window or in the pending writes. An apply at that moment writes Git's older value over it. GitOps Reverser -then mirrors that revert faithfully, as a commit authored by the applier, and the edit is gone. +would then mirror that revert faithfully, as a commit authored by the applier, and the edit would be +gone. (An edit that lands while an apply GitOps Reverser asked for is running keeps its value in +Git: the applier's write over it is not mirrored. See +[the Flux guide](flux.md#what-gitops-reverser-asks-and-when).) Only GitOps Reverser can see those pending edits. The applier cannot. So an apply nobody asked GitOps Reverser about is an apply nobody checked, and every trigger the applier runs by itself is diff --git a/docs/design/apply-trigger.md b/docs/design/apply-trigger.md index 80973bbb..49796063 100644 --- a/docs/design/apply-trigger.md +++ b/docs/design/apply-trigger.md @@ -1,7 +1,8 @@ # The apply trigger -> **design**: decided 2026-10-16; the Flux applier, the push receiver and `onRefusal: Reapply` are -> built, Argo CD and the webhook applier are the aim. Index: [`../INDEX.md`](../INDEX.md). +> **design**: decided 2026-10-16, extended 2026-10-17; the Flux applier (in any cluster), the push +> receiver, `onRefusal: Reapply` and the apply window are built, Argo CD and the webhook applier are +> the aim. Index: [`../INDEX.md`](../INDEX.md). > How to use it: [`../flux.md`](../flux.md) and [`../bi-directional.md`](../bi-directional.md). > Fields: [`../configuration.md`](../configuration.md#who-applies-this-folder-specapplier). @@ -104,7 +105,7 @@ flowchart LR OBS --> OWE --> QUIET QUIET -- "yes, and due" --> EXEC - EXEC -- "recheck targets, check source,
stamp requestedAt" --> FLUX + EXEC -- "recheck targets, check source,
stamp requestedAt, wait for it" --> FLUX EXEC -. "report: Delivered · Current ·
Failed · Rejected · Dropped" .-> OWE ``` @@ -118,13 +119,15 @@ flowchart LR reason. - **Attempting.** The dispatcher re-reads each target. It asks only for those that are still the same incarnation, still name this applier on this branch, are still admitted by their - `ClusterProvider`, and for a `Refusal` still consent. If no target still holds the request's + `ClusterProvider`s (their own, and the applier's when `spec.applier.clusterProviderRef` names + another), and for a `Refusal` still consent. If no target still holds the request's reason, it asks nothing (`Dropped`). Just before the stamp it asks the worker whether the decision still holds (the commit is still the tip, nothing is waiting to be published); if not, it stamps nothing (`Stale`, no failure). Then Flux: stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository` when its artifact lacks the commit; when the artifact holds it, only a `Refusal` stamps the `Kustomization`, because the artifact already woke it for anything else. A source on - another repository or branch is `Rejected`. + another repository or branch is `Rejected`. A delivered attempt is reported only once Flux has + finished the apply, or after 5 minutes, so the attempt is out for exactly the apply window. - **Settling.** A delivered attempt is done. A stale one gives back what it carried and is decided again at the next quiet moment. Any other outcome gives back what the attempt carried, a refusal only if it still stands, and starts that applier's backoff (2s doubling to 5m, at least @@ -133,35 +136,36 @@ flowchart LR compare-and-swap push (at the next publication), or the periodic refresh (minutes). All three feed the same observation point. -## The apply window: what is still racy - -**A request is asynchronous, and that leaves a window.** The patch returns at once; Flux fetches and -applies a few seconds later. An API edit that lands in those seconds, to a field the -`Kustomization` applies, is reverted by the apply, and GitOps Reverser mirrors the revert as Flux's -commit. The edit survives in Git history only if its own commit was pushed first. - -The quiet-branch rule confines that window to three moments: right after somebody else's push, right -after a refusal the target asked to have reverted, and right after `quietPeriod` of silence on a -branch whose artifact is behind. It does not close it, and making the request synchronous would not -either: waiting for the apply to finish does not stop a user from editing during it. Only refusing -edits would, which halts the API for everyone to protect a few seconds, so we do not. - -What would make the window smaller and visible, in order of cost: - -1. **Know when the window closes.** Flux copies the requested stamp into the `Kustomization`'s - `status.lastHandledReconcileAt` once it has handled it. The dispatcher can report `Delivered` - only then, so an attempt stays "out" for exactly the apply's duration and nothing is stacked on - an apply still running. -2. **Name the collision.** An edit observed while an attempt is out, on an object the applier owns, - is the case that can be reverted. Recording it (an Event on the `GitTarget`, a counter) turns a - silent revert into a visible one. -3. **Let the API win the window.** A write by the applier's own field manager during an attempt we - made, that undoes a field edited after that attempt was sent, is our apply arriving late, with no - new intent behind it. Not mirroring it keeps the edit in Git, and the next request applies it - back. This is the rule "for an object both sides changed, the API value wins", applied to the - apply window. It needs - the field manager from `managedFields` (no audit), and it is the only option that changes what - reaches Git, so it needs its own decision. +## The apply window + +**A request is asynchronous, and that leaves a window.** Flux fetches and applies a few seconds +after the stamp. An API edit that lands in those seconds, to a field the `Kustomization` applies, is +reverted by the apply, because the commit Flux was asked for does not hold it yet. Making the +request synchronous would not help: waiting for the apply does not stop a user from editing during +it, and refusing edits would halt the API for everyone to protect a few seconds. + +The quiet-branch rule confines the window to three moments: right after somebody else's push, right +after a refusal the target asked to have reverted, and after `quietPeriod` of silence on a branch +whose artifact is behind. Inside it, **the API wins**, as it does for any object both sides changed: + +- **The window is known.** The dispatcher reports a delivered attempt only once Flux has finished + the apply: `status.lastHandledReconcileAt` carries the stamp of a stamped `Kustomization`, or the + `Kustomization` attempted a new revision after a source stamp (or the source handled the stamp + with no new artifact). At most 5 minutes; the worker keeps the window open 10 seconds more, + because Flux's writes reach it through the watch. +- **The late apply is withheld.** A write whose latest `managedFields` entry is + `kustomize-controller` alone, inside the window, to an object in the applier's folders that the + API side edited after the attempt was sent, is not mirrored. Git keeps the edit, and the + `CatchUp` its push owes applies it back. The cluster runs the reverted value until then. +- **Only proof withholds.** `managedFields` times resolve to the second. When `kustomize-controller` + ties with another manager, the write could be the edit itself, and withholding an edit loses it + outright; it is mirrored as before. A DELETE carries nothing about its deleter and is never + withheld. Both are logged. + +Every collision is logged with the object, the applier and the requested commit, and counted in +`apply_window_collisions_total` (`withheld` or `ambiguous`). What remains: an edit reverted on the +cluster for up to `quietPeriod` until the `CatchUp`, an ambiguous tie mirrored as the applier's +commit, and a Flux write that reaches the watch more than 10 seconds after Flux reported done. ## The aim beyond Flux @@ -176,8 +180,10 @@ consequences: - **The commit is pinned**, so a sync applies exactly what the worker checked. - **A running operation is waited for, never replaced**, as Argo CD's own `SetAppOperation` does. -The open question that shapes its API: a hub Argo CD keeps `Application`s in a management cluster, -not the cluster they deploy to, and today an applier is looked up in the source cluster. +A hub Argo CD keeps `Application`s in a management cluster, not the cluster they deploy to. That is +answered as for Flux: `spec.applier.clusterProviderRef` names the cluster the `Application` lives +in, and the `Application`'s own field manager (`argocd-controller`) takes `kustomize-controller`'s +place in the apply window. **A webhook applier** for any other tool: POST `{"repository", "branch", "commit", "reason"}`, signed `X-Signature: sha256=` the way Flux's `generic-hmac` `Receiver` checks. It gets @@ -192,44 +198,30 @@ in-process plugin: it would run with every credential the operator holds. - **One replica.** The receiver, the branch workers and the dispatcher run in one pod, and the chart refuses more than one replica. Leader election is not switched on; more than one replica needs ownership coordination first, and deliveries routed to the pod that owns the branch. +- **A request holds its applier while Flux applies.** The next request to that applier waits for + the running apply (up to 5 minutes), which is the point: nothing is stacked on an apply still + running. A Flux that never reports done costs those 5 minutes per request. - **Every applier on the branch is asked on a `ForeignCommit`**, whether or not the push touched its folder. Harmless for Flux; for Argo CD it runs hooks. Narrowing it needs the applier's own source paths, so it waits for a measured complaint. ## Where this goes next -In order. Each step is one pull request, and the first three are what the first adopter, gitops-api, -needs before it can drop its own gate. - -1. **Merge the Flux applier (#419).** Both reviews' findings and the customer review's blockers on - stale stamps and on a late webhook cutting a save's window are fixed, each with a regression test. -2. **Find the applier in the `GitTarget`'s own cluster.** Today the `Kustomization` is looked up in - the source cluster. The usual way to run Flux against a remote cluster (gitops-api's kcp - workspaces, and every hub-and-spoke install) keeps it in the management cluster and applies - through `spec.kubeConfig`; such a target sits at `ApplierNotFound` forever. Proposed: - `spec.applier.cluster: Source | Local`, `Source` the default. `Local` resolves the reference with - the operator's own identity and only in the `GitTarget`'s own namespace, so it reaches nothing a - namespace-scoped `Role` does not grant. It is the same question as hub Argo CD, so it is settled - first and the Argo CD API follows it. -3. **Harden the receiver.** Today a route answers 401 or 503 when its `GitProvider` exists and 404 - when it does not, on a guessable `/git-push//`, so it lists namespaces; and an - unsigned request reads a Secret and buffers up to 25 MiB with no limit on concurrent requests, in - the pod that also mirrors. Proposed: a path derived from a hash of the token, namespace and name, - as Flux's `Receiver` does; one answer for an unknown route and a bad signature; a small limit on - requests in flight and a lower body cap before verification. -4. **Close the apply window's loop.** Report an attempt as delivered only once Flux has handled the - stamp (`status.lastHandledReconcileAt`), so the worker knows exactly when an apply is running, and - name an edit that lands inside one. Whether to also let the API win the window (option 3 above) - is a separate decision, because it changes what reaches Git. -5. **The Argo CD applier**, as described above, once step 2 has decided where an applier lives. -6. **The webhook applier.** -7. **More than one replica.** Leader election is not switched on, and the chart runs one replica. +The Flux applier PR (#419) carries what gitops-api, the first adopter, needs before it can drop its +own gate: both reviews' findings and the customer review's three blockers (stale stamps, a late +webhook cutting a save's window, a `Kustomization` in the management cluster via +`spec.applier.clusterProviderRef`), the hardened receiver, and the apply window. After it, in order, +each one pull request: + +1. **The Argo CD applier**, as described above. +2. **The webhook applier.** +3. **More than one replica.** Leader election is not switched on, and the chart runs one replica. Standbys need ownership coordination first (the HA plan in `docs/future/`), and deliveries then need routing to the pod that owns the branch. -Test gaps to close along the way: a remote source cluster, `Forbidden` against real RBAC, the receiver -and applier role installed by the Helm chart, two `GitTarget`s sharing one `Kustomization` in e2e, -and `-race` in CI. +Test gaps to close along the way: a remote source cluster and an applier in another cluster in e2e, +`Forbidden` against real RBAC, the receiver and applier role installed by the Helm chart, two +`GitTarget`s sharing one `Kustomization` in e2e, an apply-window collision in e2e, and `-race` in CI. **What adopting it changes for gitops-api.** Its gate shrinks to "one save, one window", which protects commit messages and has nothing to do with Flux. Its push handler relays each push to the @@ -244,6 +236,9 @@ been quiet; `quietPeriod` is the knob for that trade. | The decision: quiet branch, reasons, quiet period, baseline | [`apply_trigger_test.go`](../../internal/git/apply_trigger_test.go) | | Retries are decided by the worker; recovered refusals, withdrawn consent and recreated targets are not retried; each applier keeps its own clock; one attempt for a shared applier; a busy sibling worker holds the branch | [`apply_attempt_test.go`](../../internal/git/apply_attempt_test.go) | | Every attempt rechecks targets, asks for nothing weaker than it was told, and stamps no source on another branch | [`dispatcher_test.go`](../../internal/applier/dispatcher_test.go), [`flux_test.go`](../../internal/applier/flux_test.go) | +| An applier in another cluster is reached through its own `ClusterProvider`, behind that provider's `accessFrom` | [`dispatcher_test.go`](../../internal/applier/dispatcher_test.go), [`clusterprovider_admission_test.go`](../../internal/authz/clusterprovider_admission_test.go), [`gittarget_applier_test.go`](../../internal/controller/gittarget_applier_test.go) | +| An attempt stays out until Flux finished the apply; the applier's late write over a newer edit is withheld, and nothing that could be the edit is | [`flux_test.go`](../../internal/applier/flux_test.go), [`apply_window_test.go`](../../internal/git/apply_window_test.go), [`last_field_managers_test.go`](../../internal/watch/last_field_managers_test.go) | +| The receiver's path names no provider, and every miss answers the same | [`receiver_test.go`](../../internal/webhook/receiver_test.go) | | The original race: two API edits end at the later value with no extra commit | [bi-directional Flux corner](../../test/e2e/flux_bi_directional_e2e_test.go) | | `SourceMismatch` is never stamped and is delivered once fixed; a Gitea webhook and the refresh alone each get a push applied | [Manager Flux Applier](../../test/e2e/flux_applier_e2e_test.go) | | A refused edit is reverted by its `Kustomization` | [Refusal reapply](../../test/e2e/refusal_reapply_e2e_test.go) | diff --git a/docs/flux.md b/docs/flux.md index aebb1ac1..5565510b 100644 --- a/docs/flux.md +++ b/docs/flux.md @@ -275,14 +275,22 @@ decided, it stamps nothing and decides again at the next quiet moment. Without t request decided at one commit could stamp the `Kustomization` after a newer push and re-apply the older commit over a published edit. -**It is asynchronous.** The patch returns as soon as the API server accepts it, and nothing waits -for Flux to fetch or apply. It runs off the branch worker, so a slow source cluster never holds a -publication, and a failed request never fails one. That leaves a window of a few seconds while Flux -applies: an API edit made then, to a field the `Kustomization` sets, is reverted by the apply. The -quiet-branch rule confines the window to the moments listed above; waiting for the apply would not -close it, because waiting does not stop anybody editing. -[The apply trigger design](design/apply-trigger.md#the-apply-window-what-is-still-racy) weighs the -ways to shrink it. +**An edit made while Flux applies keeps its value.** The patch returns at once, and Flux fetches and +applies a few seconds later. Until Flux reports it finished (`status.lastHandledReconcileAt` for a +stamped `Kustomization`, a new `status.lastAttemptedRevision` after a source stamp, at most +5 minutes), GitOps Reverser treats the apply as running. An API edit made in those seconds, to a +field the `Kustomization` sets, is reverted on the cluster by that apply, because the commit Flux +was asked for does not hold it yet. GitOps Reverser recognizes that write by its field manager, +`kustomize-controller`, and does not mirror it: the API's value stays in Git, and the `CatchUp` its +push owes applies it back once the branch has been quiet for `quietPeriod`. Each such write is +logged and counted in `apply_window_collisions_total{outcome="withheld"}`. + +Only a write proven to be Flux's is withheld. When `kustomize-controller` and another manager wrote +the object in the same second, the write could be the edit itself, so it is mirrored as before and +counted as `ambiguous`. A delete is never withheld. + +The request runs off the branch worker, so a slow source cluster never holds a publication, and a +failed request never fails one. Before it patches, GitOps Reverser reads each `GitTarget` again: it must still be the same object (a deleted and recreated one inherits nothing), still name this `Kustomization` on this diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index 11ac557f..ce745f43 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -823,6 +823,7 @@ and the push receiver tells it that somebody else pushed. See | Metric | Type | Labels | Meaning | | --- | --- | --- | --- | | `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per attempt to one applier. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not ask, including a `Kustomization` that applies another repository or branch; the branch worker keeps the obligation and decides the retry on a quiet branch, so a failure that persists keeps counting at its backoff) or `dropped` (no `GitTarget` may still ask for that reason: each was deleted or recreated, moved to another branch or applier, is no longer admitted by its `ClusterProvider`, or withdrew its consent to a `Refusal`) or `stale` (asked nothing because, by the time of the patch, the commit was no longer the branch tip or an edit was waiting to be published; the worker decides again at the next quiet moment) | +| `apply_window_collisions_total` | counter | `provider_namespace`, `provider_name`, `branch`, `outcome` | One per write by the applier, while an apply it was asked for runs, to an object the API side edited after the request was sent: the apply arriving late over a newer edit. `outcome` is `withheld` (proven to be the applier's write, so not mirrored: Git keeps the API's value and the next apply restores it on the cluster) or `ambiguous` (the applier and another field manager wrote in the same second, so it was mirrored as ever; if it was the apply, the revert is in Git as the applier's commit) | | `receiver_deliveries_total` | counter | `type`, `outcome` | One per verified-or-refused delivery to `/git-push/`. `type` is the `GitProvider.spec.receiver.type`; `outcome` is `notified` (handed to the branch worker), `unmatched` (no worker for that branch), `ignored` (a ping, a tag, another branch's event type, or a `generic-hmac` body naming another repository), `rejected` (bad signature or an oversized body) or `invalid` (verified but unparseable) | A delivery counted `notified` is only a hint: the worker reads the advertisement, and fetches under @@ -836,6 +837,10 @@ followed by `CatchUp` requests and never by `ForeignCommit`. sum by (reason) (rate(gitopsreverser_apply_requests_total{outcome="failed"}[15m])) # A receiver a Git host is calling with the wrong secret. sum by (type) (rate(gitopsreverser_receiver_deliveries_total{outcome="rejected"}[15m])) > 0 +# Edits landing while an apply runs, by branch. Withheld ones are reverted on the cluster until the +# next apply; ambiguous ones may have put the applier's revert in Git. +sum by (provider_namespace, provider_name, branch, outcome) ( + increase(gitopsreverser_apply_window_collisions_total[1h])) > 0 ``` ## Audit attribution (optional) diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index 7fc8fb7c..6209cecd 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -61,6 +61,15 @@ type SourceClients func(ctx context.Context, clusterID string) (dynamic.Interfac // branch worker decides when to try again. const DefaultAttemptTimeout = 30 * time.Second +// DefaultApplyWindowTimeout bounds how long a delivered attempt stays out waiting for Flux to finish +// the apply. An apply that waits on health checks can take minutes; one Flux never finishes (the +// controller is down, the fetch failed) must not hold the applier forever, so the window then +// closes by time. +const DefaultApplyWindowTimeout = 5 * time.Minute + +// DefaultHandledPollInterval is how often a delivered attempt reads whether Flux finished. +const DefaultHandledPollInterval = 2 * time.Second + // Dispatcher carries out the apply requests the branch workers decide. It implements // git.ApplyRequester, and runs as a manager.Runnable beside the branch workers (it asks for leader // election as they do, which keeps the two together should it ever be switched on). @@ -83,8 +92,13 @@ type Dispatcher struct { recorder record.EventRecorder log logr.Logger - // AttemptTimeout bounds one attempt. Set before Start. + // AttemptTimeout bounds one attempt up to its stamp. Set before Start. AttemptTimeout time.Duration + // ApplyWindowTimeout bounds the wait, after a stamp, for Flux to finish the apply it asked for; + // zero does not wait. Set before Start. + ApplyWindowTimeout time.Duration + // HandledPollInterval is how often that wait reads Flux's status. Set before Start. + HandledPollInterval time.Duration // Now is the clock requestedAt stamps and the ledger read. Now func() time.Time @@ -111,13 +125,15 @@ func NewDispatcher( log logr.Logger, ) *Dispatcher { return &Dispatcher{ - reader: reader, - sourceClients: sourceClients, - recorder: recorder, - log: log, - AttemptTimeout: DefaultAttemptTimeout, - Now: time.Now, - ledger: map[string]ledgerEntry{}, + reader: reader, + sourceClients: sourceClients, + recorder: recorder, + log: log, + AttemptTimeout: DefaultAttemptTimeout, + ApplyWindowTimeout: DefaultApplyWindowTimeout, + HandledPollInterval: DefaultHandledPollInterval, + Now: time.Now, + ledger: map[string]ledgerEntry{}, } } @@ -193,8 +209,11 @@ type eligibleTarget struct { reason v1alpha3.ApplyReason } -// attempt carries out one request and returns its report. +// attempt carries out one request and returns its report. A delivered one is reported once Flux has +// finished the apply (awaitHandled), so the branch worker's attempt stays out for exactly the apply +// window: the seconds in which an edit can be reverted by it. func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.ApplyResult { + run := ctx ctx, cancel := context.WithTimeout(ctx, d.AttemptTimeout) defer cancel() log := d.log.WithValues("applier", req.Applier.String(), "branch", req.Branch.String(), @@ -220,7 +239,7 @@ func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.Appl } flux := NewFlux(sourceClient) flux.Now = d.Now - outcome, err := flux.Request(ctx, applierName(req.Applier), want, req.Commit, req.Reason, req.Current) + stamped, err := flux.Request(ctx, applierName(req.Applier), want, req.Commit, req.Reason, req.Current) switch { case errors.Is(err, errStale): d.count(ctx, req.Reason, outcomeStale) @@ -233,15 +252,51 @@ func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.Appl } d.count(ctx, req.Reason, outcomeSent) - if outcome == OutcomeCurrent { + if stamped.Outcome == OutcomeCurrent { log.V(1).Info("applier already holds the commit; nothing to request") return report(git.ApplyCurrent, nil) } - log.Info("apply requested", "outcome", outcome) + log.Info("apply requested", "outcome", stamped.Outcome) d.delivered(req, eligible) + d.awaitHandled(run, flux, applierName(req.Applier), stamped, log) return report(git.ApplyDelivered, nil) } +// awaitHandled waits, up to ApplyWindowTimeout, for Flux to finish the apply stamped asked for. It +// decides nothing: the request was delivered either way, and a wait that ends by time or by +// shutdown only closes the window sooner than Flux did. +func (d *Dispatcher) awaitHandled( + ctx context.Context, flux *Flux, ref types.NamespacedName, stamped Stamped, log logr.Logger, +) { + if d.ApplyWindowTimeout <= 0 { + return + } + ctx, cancel := context.WithTimeout(ctx, d.ApplyWindowTimeout) + defer cancel() + started := d.Now() + ticker := time.NewTicker(d.HandledPollInterval) + defer ticker.Stop() + for { + handled, err := flux.Handled(ctx, ref, stamped) + switch { + case handled: + log.V(1).Info("Flux finished the requested apply", "after", d.Now().Sub(started).String()) + return + case err != nil && ctx.Err() == nil: + log.V(1).Info("could not read whether Flux finished the requested apply", "error", err.Error()) + } + select { + case <-ctx.Done(): + if errors.Is(ctx.Err(), context.DeadlineExceeded) { + log.Info("Flux did not report the requested apply finished in time; the apply window closes now", + "waited", d.Now().Sub(started).String(), "timeout", d.ApplyWindowTimeout.String()) + } + return + case <-ticker.C: + } + } +} + // applierName is the Kustomization an applier reference names. func applierName(ref git.ApplierRef) types.NamespacedName { return types.NamespacedName{Namespace: ref.Namespace, Name: ref.Name} diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index d33a8127..6d889cc6 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -149,6 +149,7 @@ func newDispatcherFixture(t *testing.T, targets []client.Object, sourceObjs ...r recorder := record.NewFakeRecorder(64) d := NewDispatcher(controlClient, source.get, recorder, logr.Discard()) d.Now = func() time.Time { return stampTime } + d.ApplyWindowTimeout = 0 // the tests that wait for Flux set their own return &dispatcherFixture{dispatcher: d, control: controlClient, source: source, recorder: recorder, reader: reader} } @@ -259,8 +260,12 @@ func appliedFrom(target *v1alpha3.GitTarget, cluster string) *v1alpha3.GitTarget // lives in a management cluster is asked through that cluster's client, never the source's. func TestDispatcher_AsksAnApplierInItsOwnCluster(t *testing.T) { hub := git.ApplierRef{Cluster: "hub", Namespace: fluxSystemNS, Name: "checkout"} - f := newDispatcherFixture(t, - []client.Object{appliedFrom(gitTarget("a", checkoutRef(), "prod-eu-1"), "hub"), clusterProvider("hub", targetNS)}, + f := newDispatcherFixture( + t, + []client.Object{ + appliedFrom(gitTarget("a", checkoutRef(), "prod-eu-1"), "hub"), + clusterProvider("hub", targetNS), + }, behindSource()..., ) @@ -509,3 +514,40 @@ func TestDescribeCommit(t *testing.T) { assert.Equal(t, "abc", describeCommit("abc")) assert.Equal(t, "the branch, which the remote does not carry", describeCommit("")) } + +// TestDispatcher_ADeliveredAttemptStaysOutUntilFluxFinished: the worker's attempt is the apply +// window, so a delivered one is reported only once Flux has finished the apply, or by time. +func TestDispatcher_ADeliveredAttemptStaysOutUntilFluxFinished(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{gitTarget("a", checkoutRef(), "")}, behindSource()...) + f.dispatcher.ApplyWindowTimeout = time.Minute + f.dispatcher.HandledPollInterval = 5 * time.Millisecond + + done := make(chan git.ApplyResult, 1) + go func() { done <- f.attempt(request(checkout, v1alpha3.ApplyReasonForeignCommit, "a")) }() + + require.Eventually(t, func() bool { return len(f.patches()) == 1 }, 5*time.Second, 5*time.Millisecond) + select { + case res := <-done: + t.Fatalf("reported %s before Flux finished the apply", res.Outcome) + case <-time.After(50 * time.Millisecond): + } + + setStatus(t, f.source.client, kustomizationGVR(), types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, + "main@sha1:"+testCommit, "lastAttemptedRevision") + select { + case res := <-done: + assert.Equal(t, git.ApplyDelivered, res.Outcome) + case <-time.After(5 * time.Second): + t.Fatal("not reported after Flux finished the apply") + } +} + +// TestDispatcher_AnApplyFluxNeverFinishesClosesByTime: Flux down, or a fetch that never lands, +// must not hold the applier's attempt out forever. +func TestDispatcher_AnApplyFluxNeverFinishesClosesByTime(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{gitTarget("a", checkoutRef(), "")}, behindSource()...) + f.dispatcher.ApplyWindowTimeout = 30 * time.Millisecond + f.dispatcher.HandledPollInterval = 5 * time.Millisecond + + assert.Equal(t, git.ApplyDelivered, f.attempt(request(checkout, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) +} diff --git a/internal/applier/flux.go b/internal/applier/flux.go index 60656798..2ca43c57 100644 --- a/internal/applier/flux.go +++ b/internal/applier/flux.go @@ -109,6 +109,21 @@ type Verdict struct { Err error } +// Stamped is what one Request did, with what Flux reported before it, so Handled can tell when +// Flux has finished the apply it asked for. +type Stamped struct { + // Outcome is what was stamped, if anything. + Outcome Outcome + // Token is the reconcile.fluxcd.io/requestedAt value stamped; empty when nothing was. + Token string + // Source is the GitRepository the Kustomization applies from. + Source types.NamespacedName + // PriorAttemptedRevision is the Kustomization's status.lastAttemptedRevision before the stamp. + PriorAttemptedRevision string + // PriorArtifactRevision is the GitRepository's status.artifact.revision before the stamp. + PriorArtifactRevision string +} + // Binding is the repository and branch an applier must apply from: the GitProvider's spec.url and // the GitTarget's spec.branch. Request checks it before it writes anything, not only Validate, // because a published ApplierConfigured verdict can be older than the applier it describes. @@ -156,7 +171,7 @@ func isPermanent(err error) bool { } // Request asks the Kustomization ref to apply commit, the way `flux reconcile kustomization -// --with-source` does, minus the waiting: a request asks and never waits to see the apply. +// --with-source` does, minus the waiting: Request only asks, and Handled tells when Flux is done. // // It first checks that the Kustomization applies want, and refuses with a permanent error when it // does not: stamping a GitRepository that follows another branch would deploy that branch and @@ -181,40 +196,92 @@ func (f *Flux) Request( commit string, reason v1alpha3.ApplyReason, current func() bool, -) (Outcome, error) { +) (Stamped, error) { ks, err := f.client.Resource(kustomizationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) if err != nil { - return "", fmt.Errorf("get %s %s: %w", kindKustomization, ref, err) + return Stamped{}, fmt.Errorf("get %s %s: %w", kindKustomization, ref, err) } sourceRef, err := gitRepositoryRef(ks) if err != nil { - return "", permanentError{err} + return Stamped{}, permanentError{err} } source, err := f.client.Resource(gitRepositoryGVR()).Namespace(sourceRef.Namespace). Get(ctx, sourceRef.Name, metav1.GetOptions{}) if err != nil { - return "", fmt.Errorf("get %s %s: %w", kindGitRepository, sourceRef, err) + return Stamped{}, fmt.Errorf("get %s %s: %w", kindGitRepository, sourceRef, err) } if msg := sourceMismatch(source, sourceRef, want.RepositoryURL, want.Branch); msg != "" { - return "", permanentError{fmt.Errorf("%s %s: %s", kindKustomization, ref, msg)} + return Stamped{}, permanentError{fmt.Errorf("%s %s: %s", kindKustomization, ref, msg)} } if current != nil && !current() { - return "", errStale + return Stamped{}, errStale + } + stamped := Stamped{ + Source: sourceRef, + PriorAttemptedRevision: statusString(ks, "lastAttemptedRevision"), + PriorArtifactRevision: statusString(source, "artifact", "revision"), } if commit != "" && artifactHolds(source, commit) { if reason != v1alpha3.ApplyReasonRefusal { - return OutcomeCurrent, nil + stamped.Outcome = OutcomeCurrent + return stamped, nil } - if err := f.stamp(ctx, kustomizationGVR(), ref); err != nil { - return "", fmt.Errorf("stamp %s %s: %w", kindKustomization, ref, err) + token, err := f.stamp(ctx, kustomizationGVR(), ref) + if err != nil { + return Stamped{}, fmt.Errorf("stamp %s %s: %w", kindKustomization, ref, err) } - return OutcomeStampedKustomization, nil + stamped.Outcome, stamped.Token = OutcomeStampedKustomization, token + return stamped, nil + } + token, err := f.stamp(ctx, gitRepositoryGVR(), sourceRef) + if err != nil { + return Stamped{}, fmt.Errorf("stamp %s %s: %w", kindGitRepository, sourceRef, err) + } + stamped.Outcome, stamped.Token = OutcomeStampedSource, token + return stamped, nil +} + +// Handled reports whether Flux has finished the apply s asked for, which is when the apply window +// closes: until then an edit to an object the Kustomization applies can be reverted by it. +// +// Flux records a request it handled in status.lastHandledReconcileAt, and does so when the +// reconcile ends (external-sources/flux/pkg/runtime/reconcile/result.go). A stamped Kustomization +// has therefore finished once that field carries the stamp. A stamped GitRepository wakes the +// Kustomization through its new artifact, so that apply has finished once the Kustomization +// attempted a revision other than the one it had attempted before; and if the fetch produced no new +// artifact, there is nothing for it to apply, and the GitRepository's own handled request says so. +func (f *Flux) Handled(ctx context.Context, ref types.NamespacedName, s Stamped) (bool, error) { + if s.Outcome != OutcomeStampedKustomization && s.Outcome != OutcomeStampedSource { + return true, nil } - if err := f.stamp(ctx, gitRepositoryGVR(), sourceRef); err != nil { - return "", fmt.Errorf("stamp %s %s: %w", kindGitRepository, sourceRef, err) + ks, err := f.client.Resource(kustomizationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + if err != nil { + return false, fmt.Errorf("get %s %s: %w", kindKustomization, ref, err) + } + if s.Outcome == OutcomeStampedKustomization { + return statusString(ks, "lastHandledReconcileAt") == s.Token, nil + } + if attempted := statusString( + ks, + "lastAttemptedRevision", + ); attempted != "" && + attempted != s.PriorAttemptedRevision { + return true, nil + } + source, err := f.client.Resource(gitRepositoryGVR()).Namespace(s.Source.Namespace). + Get(ctx, s.Source.Name, metav1.GetOptions{}) + if err != nil { + return false, fmt.Errorf("get %s %s: %w", kindGitRepository, s.Source, err) } - return OutcomeStampedSource, nil + return statusString(source, "lastHandledReconcileAt") == s.Token && + statusString(source, "artifact", "revision") == s.PriorArtifactRevision, nil +} + +// statusString reads a string under an object's status. +func statusString(obj *unstructured.Unstructured, fields ...string) string { + value, _, _ := unstructured.NestedString(obj.Object, append([]string{"status"}, fields...)...) + return value } // stamp sets reconcile.fluxcd.io/requestedAt to now, which is the whole of what `flux reconcile` @@ -222,20 +289,21 @@ func (f *Flux) Request( // (@1ed703a3) merge-patches that annotation, and only that, with time.Now().Format(time.RFC3339Nano). // Flux reconciles when the value differs from the status.lastHandledReconcileAt it last answered, // so any new value asks; no read-modify-write is needed and no conflict is possible. -func (f *Flux) stamp(ctx context.Context, gvr schema.GroupVersionResource, ref types.NamespacedName) error { +// +// It returns the value stamped, which Flux copies to status.lastHandledReconcileAt once handled. +func (f *Flux) stamp(ctx context.Context, gvr schema.GroupVersionResource, ref types.NamespacedName) (string, error) { + token := f.Now().Format(time.RFC3339Nano) patch, err := json.Marshal(map[string]any{ "metadata": map[string]any{ - "annotations": map[string]string{ - fluxmeta.ReconcileRequestAnnotation: f.Now().Format(time.RFC3339Nano), - }, + "annotations": map[string]string{fluxmeta.ReconcileRequestAnnotation: token}, }, }) if err != nil { - return err + return "", err } _, err = f.client.Resource(gvr).Namespace(ref.Namespace). Patch(ctx, ref.Name, types.MergePatchType, patch, metav1.PatchOptions{}) - return err + return token, err } // gitRepositoryRef reads a Kustomization's spec.sourceRef, which must name a GitRepository. Its diff --git a/internal/applier/flux_guide_test.go b/internal/applier/flux_guide_test.go index 369c63dd..d6bbe737 100644 --- a/internal/applier/flux_guide_test.go +++ b/internal/applier/flux_guide_test.go @@ -89,9 +89,9 @@ func TestFluxGuide_TheCompleteExampleIsAccepted(t *testing.T) { verdict := flux.Validate(context.Background(), ref, want) assert.True(t, verdict.OK, "ApplierConfigured for the guide's example: %s: %s", verdict.Reason, verdict.Message) - outcome, err := flux.Request(context.Background(), ref, want, testCommit, v1alpha3.ApplyReasonForeignCommit, nil) + stamped, err := flux.Request(context.Background(), ref, want, testCommit, v1alpha3.ApplyReasonForeignCommit, nil) require.NoError(t, err) - assert.Equal(t, OutcomeStampedSource, outcome, "a source with no artifact yet is asked to fetch") + assert.Equal(t, OutcomeStampedSource, stamped.Outcome, "a source with no artifact yet is asked to fetch") assert.NotEmpty(t, requestedAt(t, client, gitRepositoryGVR(), repository.GetNamespace(), repository.GetName())) } diff --git a/internal/applier/flux_test.go b/internal/applier/flux_test.go index ca574ce8..ee6bdb2b 100644 --- a/internal/applier/flux_test.go +++ b/internal/applier/flux_test.go @@ -185,7 +185,7 @@ func TestFluxRequest_Outcomes(t *testing.T) { got, err := pinnedFlux(client).Request(context.Background(), types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"}, testBinding, tt.commit, tt.reason, nil) require.NoError(t, err) - assert.Equal(t, tt.want, got) + assert.Equal(t, tt.want, got.Outcome) wantStamp := func(want bool) string { if want { @@ -246,7 +246,7 @@ func TestFluxRequest_SourceRefNamespace(t *testing.T) { got, err := pinnedFlux(client).Request(context.Background(), kustomization, testBinding, testCommit, v1alpha3.ApplyReasonForeignCommit, nil) require.NoError(t, err) - assert.Equal(t, OutcomeStampedSource, got) + assert.Equal(t, OutcomeStampedSource, got.Outcome) assert.NotEmpty(t, requestedAt(t, client, gitRepositoryGVR(), tt.wantNamespace, "platform")) }) } @@ -547,3 +547,75 @@ func TestSelfSubjectAccessCheck_AsksTheSourceCluster(t *testing.T) { _, err = check(context.Background(), authorizationv1.ResourceAttributes{Verb: "patch"}) require.Error(t, err) } + +// setStatus writes fields under an object's status in the fake source cluster, as Flux would. +func setStatus(t *testing.T, client *dynamicfake.FakeDynamicClient, gvr schema.GroupVersionResource, + ref types.NamespacedName, value string, fields ...string, +) { + t.Helper() + resource := client.Resource(gvr).Namespace(ref.Namespace) + obj, err := resource.Get(context.Background(), ref.Name, metav1.GetOptions{}) + require.NoError(t, err) + require.NoError(t, unstructured.SetNestedField(obj.Object, value, append([]string{"status"}, fields...)...)) + _, err = resource.Update(context.Background(), obj, metav1.UpdateOptions{}) + require.NoError(t, err) +} + +// TestFlux_Handled pins when an apply window closes: when Flux reports it finished the apply the +// stamp asked for, and not before, for each thing a request can stamp. +func TestFlux_Handled(t *testing.T) { + checkout := types.NamespacedName{Namespace: fluxSystemNS, Name: "checkout"} + platform := types.NamespacedName{Namespace: fluxSystemNS, Name: "platform"} + newClient := func() *dynamicfake.FakeDynamicClient { + ks := kustomizationObj(fluxSystemNS, "checkout", gitRepoSourceRef("platform")) + _ = unstructured.SetNestedField(ks.Object, "main@sha1:"+olderCommit, "status", "lastAttemptedRevision") + return newSourceClient(ks, + gitRepositoryObj(fluxSystemNS, "platform", testRepoURL, branchRef(testBranch), "main@sha1:"+olderCommit)) + } + request := func(t *testing.T, client *dynamicfake.FakeDynamicClient, reason v1alpha3.ApplyReason, commit string) Stamped { + t.Helper() + stamped, err := pinnedFlux(client).Request(context.Background(), checkout, testBinding, commit, reason, nil) + require.NoError(t, err) + return stamped + } + handled := func(t *testing.T, client *dynamicfake.FakeDynamicClient, stamped Stamped) bool { + t.Helper() + ok, err := pinnedFlux(client).Handled(context.Background(), checkout, stamped) + require.NoError(t, err) + return ok + } + + t.Run("a stamped Kustomization, once it handled the stamp", func(t *testing.T) { + client := newClient() + stamped := request(t, client, v1alpha3.ApplyReasonRefusal, olderCommit) + require.Equal(t, OutcomeStampedKustomization, stamped.Outcome) + assert.False(t, handled(t, client, stamped)) + setStatus(t, client, kustomizationGVR(), checkout, "an older stamp", "lastHandledReconcileAt") + assert.False(t, handled(t, client, stamped), "another request's answer is not this one's") + setStatus(t, client, kustomizationGVR(), checkout, stamped.Token, "lastHandledReconcileAt") + assert.True(t, handled(t, client, stamped)) + }) + + t.Run("a stamped source, once the Kustomization attempted the new revision", func(t *testing.T) { + client := newClient() + stamped := request(t, client, v1alpha3.ApplyReasonForeignCommit, testCommit) + require.Equal(t, OutcomeStampedSource, stamped.Outcome) + assert.False(t, handled(t, client, stamped)) + setStatus(t, client, kustomizationGVR(), checkout, "main@sha1:"+testCommit, "lastAttemptedRevision") + assert.True(t, handled(t, client, stamped)) + }) + + t.Run("a stamped source whose fetch brought nothing new, once the source handled the stamp", func(t *testing.T) { + client := newClient() + stamped := request(t, client, v1alpha3.ApplyReasonForeignCommit, testCommit) + setStatus(t, client, gitRepositoryGVR(), platform, stamped.Token, "lastHandledReconcileAt") + assert.True(t, handled(t, client, stamped), "no new artifact: nothing for the Kustomization to apply") + + setStatus(t, client, gitRepositoryGVR(), platform, "main@sha1:"+testCommit, "artifact", "revision") + assert.False(t, handled(t, client, stamped), "a new artifact: the Kustomization's apply is still to come") + }) + + t.Run("nothing stamped is nothing to wait for", func(t *testing.T) { + assert.True(t, handled(t, newClient(), Stamped{Outcome: OutcomeCurrent})) + }) +} diff --git a/internal/git/apply_trigger.go b/internal/git/apply_trigger.go index 654cb9e2..912cbbe2 100644 --- a/internal/git/apply_trigger.go +++ b/internal/git/apply_trigger.go @@ -185,6 +185,9 @@ type applierState struct { // next attempt may go. failures int retryAt time.Time + // window is the applier's apply window: open from sending an attempt until Flux finished it + // (apply_window.go). Nil when no attempt was sent, or the last asked for no apply. + window *applyWindow } // inflightAttempt is what an attempt carries away from its applier's obligations. A failed attempt @@ -413,6 +416,7 @@ func (l *branchWorkerEventLoop) sendAttempt(state *applierState, tip string, ref id: l.nextApplyID, foreign: state.foreign, catchUp: state.catchUp, refusals: refusals, } state.foreign, state.catchUp, state.inflight = false, false, attempt + state.openApplyWindow(attempt.id, tip, time.Now()) for key := range refusals { delete(l.reapplyOwed, key) } @@ -446,6 +450,7 @@ func (l *branchWorkerEventLoop) settleApplyResults(now time.Time) { } attempt := state.inflight state.inflight = nil + state.settleApplyWindow(attempt.id, res.Outcome == ApplyDelivered, now) switch res.Outcome { case ApplyDelivered, ApplyCurrent: state.failures, state.retryAt = 0, time.Time{} diff --git a/internal/git/apply_window.go b/internal/git/apply_window.go new file mode 100644 index 00000000..963f0f2a --- /dev/null +++ b/internal/git/apply_window.go @@ -0,0 +1,157 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +import ( + "context" + "slices" + "time" + + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" + + "github.com/ConfigButler/gitops-reverser/internal/telemetry" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// The apply window is the time an apply GitOps Reverser asked for is running. An API edit that +// lands in it, to a field the applier applies, is reverted by that apply: Flux applies the commit +// it was asked for, which does not hold the edit yet. Mirroring that write would commit the revert +// as Flux's, and the edit would survive only in history. +// +// The repository's rule for an object both sides changed is that the API value wins. Applied to +// the window: a write by the applier's own field manager, during an attempt, to an object the API +// side edited after that attempt was sent, is the apply arriving late with no intent behind it. It +// is withheld. Git keeps the edit, and the CatchUp that edit's push owes applies it back. +// +// What the rule must never do is withhold an API edit, which would lose it outright. So it acts +// only on proof: the object's latest managedFields entry is the applier's manager alone. When the +// second-resolution times tie with another manager the write could be either, and it is mirrored +// as before; the collision is logged and counted as ambiguous so it is still seen. A DELETE carries +// no managedFields of its deleter and is never withheld. + +// FluxFieldManager is the field manager kustomize-controller applies with (operation Apply). It is +// fixed in Flux, not configurable (allFluxManagers in external-sources/flux/flux-operator +// cmd/cli/migrate_owner.go, and docs/facts/expansion-provenance-markers.md). +const FluxFieldManager = "kustomize-controller" + +// applyWindowGrace keeps a window open after Flux reported the apply finished. Flux's writes reach +// the worker through the watch, which can trail the status the dispatcher read. +const applyWindowGrace = 10 * time.Second + +// Apply-window collision outcomes, the outcome label of apply_window_collisions_total. +const ( + collisionWithheld = "withheld" + collisionAmbiguous = "ambiguous" +) + +// applyWindow is one applier's open apply window: from sending an attempt until Flux finished it, +// plus applyWindowGrace. +type applyWindow struct { + attempt uint64 + commit string + sentAt time.Time + // closesAt is zero while the attempt is out. + closesAt time.Time + // edited holds the objects the API side wrote since the attempt was sent, by objectKey. + edited map[string]struct{} +} + +// open reports whether the window is open at now. +func (a *applyWindow) open(now time.Time) bool { + return a != nil && (a.closesAt.IsZero() || now.Before(a.closesAt)) +} + +// objectKey is an event's object without its API version: a preferred-version bump during the +// window is still the same object. +func objectKey(id itypes.ResourceIdentifier) string { + return id.Group + "/" + id.Resource + "/" + id.Namespace + "/" + id.Name +} + +// openApplyWindow starts an applier's window as its attempt is sent. +func (state *applierState) openApplyWindow(attempt uint64, commit string, now time.Time) { + state.window = &applyWindow{attempt: attempt, commit: commit, sentAt: now, edited: map[string]struct{}{}} +} + +// settleApplyWindow ends an attempt's part in its applier's window: a delivered attempt's window +// stays open for applyWindowGrace, and any other attempt asked for no apply, so there is no window. +func (state *applierState) settleApplyWindow(attempt uint64, delivered bool, now time.Time) { + if state.window == nil || state.window.attempt != attempt { + return + } + if !delivered { + state.window = nil + return + } + state.window.closesAt = now.Add(applyWindowGrace) +} + +// covers reports whether the applier applies the GitTarget an event is for. +func (state *applierState) covers(event Event) bool { + return slices.ContainsFunc(state.targets, func(target branchTarget) bool { + return target.ref.Name == event.GitTargetName && target.ref.Namespace == event.GitTargetNamespace + }) +} + +// withholdLateApply reports whether a live event is an applier's apply arriving late over a newer +// API edit, and must not be mirrored. Every other event is recorded as an API-side edit in the +// windows of the appliers that cover it. It is called for each live event before it reaches the +// commit window. +func (l *branchWorkerEventLoop) withholdLateApply(event Event) bool { + if len(l.appliers) == 0 { + return false + } + now := time.Now() + key := objectKey(event.Identifier) + byApplier := slices.Contains(event.LastFieldManagers, FluxFieldManager) + for _, ref := range l.sortedAppliers() { + applier := l.appliers[ref] + window := applier.window + if !window.open(now) || !applier.covers(event) { + continue + } + _, editedSince := window.edited[key] + switch { + case !byApplier || event.Operation == string(itypes.OperationDelete): + window.edited[key] = struct{}{} + case !editedSince: + // The applier applying an object nobody edited since the attempt: its content is the + // commit's, which Git already holds. Mirrored as ever. + case len(event.LastFieldManagers) > 1: + l.recordApplyWindowCollision(applier, window, event, collisionAmbiguous) + // It may be the API edit itself, written in the same second: mirroring it is the + // choice that cannot lose an edit. It counts as one, too. + window.edited[key] = struct{}{} + default: + l.recordApplyWindowCollision(applier, window, event, collisionWithheld) + return true + } + } + return false +} + +// recordApplyWindowCollision logs and counts one write by an applier, inside its apply window, to +// an object the API side edited since the attempt was sent. +func (l *branchWorkerEventLoop) recordApplyWindowCollision( + applier *applierState, window *applyWindow, event Event, outcome string, +) { + w := l.w + log := w.Log.WithValues("branch", w.Branch, "applier", applier.ref.String(), + "gitTarget", event.GitTargetNamespace+"/"+event.GitTargetName, "object", event.Identifier.String(), + "requestedCommit", window.commit, "sentAgo", time.Since(window.sentAt).Round(time.Millisecond).String(), + "fieldManagers", event.LastFieldManagers) + if outcome == collisionWithheld { + log.Info("Withheld an apply that reverted a newer API edit: the applier applied the commit it was " + + "asked for over an edit made after the request, so Git keeps the API's value and the next " + + "apply puts it back on the cluster") + } else { + log.Info("Mirrored a write that may be an apply reverting a newer API edit: the applier and " + + "another field manager wrote the object in the same second, and only a proven late apply is " + + "withheld. If it was the apply, the revert is in Git as the applier's commit") + } + if telemetry.ApplyWindowCollisionsTotal == nil { + return + } + telemetry.ApplyWindowCollisionsTotal.Add(context.WithoutCancel(w.ctx), 1, + metric.WithAttributes(w.providerAttrs(attribute.String("outcome", outcome))...)) +} diff --git a/internal/git/apply_window_test.go b/internal/git/apply_window_test.go new file mode 100644 index 00000000..0b56228b --- /dev/null +++ b/internal/git/apply_window_test.go @@ -0,0 +1,154 @@ +// SPDX-License-Identifier: Apache-2.0 + +package git + +// The apply window: while an apply GitOps Reverser asked for runs, the applier's own write over an +// object the API side edited since the request was sent is withheld, so the API value wins. Only a +// proven applier write is withheld; one that could be the edit itself is mirrored and counted. See +// apply_window.go. + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + + "github.com/ConfigButler/gitops-reverser/internal/telemetry" + itypes "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// windowEvent is a live UPDATE to the Deployment "api" in target's folder, written last by managers. +func windowEvent(target, replicas string, managers ...string) Event { + obj := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "apps/v1", "kind": "Deployment", + "metadata": map[string]any{"name": "api", "namespace": "shop"}, + "spec": map[string]any{"replicas": replicas}, + }} + return Event{ + Object: obj, + Identifier: itypes.NewResourceIdentifier("apps", "v1", "deployments", "shop", "api"), + Operation: string(itypes.OperationUpdate), + LastFieldManagers: managers, + GitTargetName: target, + GitTargetNamespace: "team-a", + Path: "apps/" + target, + } +} + +// sentAttempt is a loop with alpha's applier sent an attempt that is still out. +func sentAttempt(t *testing.T) (*branchWorkerEventLoop, ApplyRequest) { + t.Helper() + loop, requester := manualLoop(t, applierTestTarget("alpha", true), applierTestTarget("bravo", false)) + owesForeign(loop.w) + loop.serviceAppliers() + out := requester.taken() + require.Len(t, out, 1) + return loop, out[0] +} + +func collisions(t *testing.T, reader *sdkmetric.ManualReader, outcome string) int64 { + t.Helper() + got, _ := telemetry.CollectInt64Sum(reader, "gitopsreverser_apply_window_collisions_total", + map[string]string{"outcome": outcome}) + return got +} + +// TestApplyWindow_TheAppliersLateWriteOverANewerEditIsWithheld is the race the window exists for: +// Flux applies the commit it was asked for after an API edit landed, and that write must not reach +// Git as Flux's revert of the edit. +func TestApplyWindow_TheAppliersLateWriteOverANewerEditIsWithheld(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + loop, _ := sentAttempt(t) + + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "3", "kubectl-edit")), "an API edit is mirrored") + assert.True(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager)), + "the applier's write over it is withheld") + assert.Equal(t, int64(1), collisions(t, reader, collisionWithheld)) +} + +// TestApplyWindow_OnlyAWithheldWriteIsKeptFromTheCommitWindow follows both events through the live +// path: the commit window ends up holding the API edit, not Flux's revert of it. +func TestApplyWindow_OnlyAWithheldWriteIsKeptFromTheCommitWindow(t *testing.T) { + loop, _ := sentAttempt(t) + + loop.handleLiveEvents(&WriteRequest{Events: []Event{ + windowEvent("alpha", "3", "kubectl-edit"), + windowEvent("alpha", "2", FluxFieldManager), + }, CommitMode: CommitModePerEvent}) + + require.NotNil(t, loop.openWindow) + require.Len(t, loop.openWindow.pathToEvent, 1) + for _, event := range loop.openWindow.pathToEvent { + replicas, _, _ := unstructured.NestedString(event.Object.Object, "spec", "replicas") + assert.Equal(t, "3", replicas, "Git keeps the API's value") + } +} + +// TestApplyWindow_WhatIsNeverWithheld: the applier's write is mirrored as ever when it could be +// the edit itself, when nobody edited the object since the attempt, when it is a delete, and for a +// folder another applier (or none) applies. +func TestApplyWindow_WhatIsNeverWithheld(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + loop, _ := sentAttempt(t) + + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager)), + "an object nobody edited since the attempt: the apply writes what Git holds") + + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "3", "kubectl-edit"))) + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager, "kubectl-edit")), + "a tie may be the API edit itself, and withholding that would lose it") + assert.Equal(t, int64(1), collisions(t, reader, collisionAmbiguous)) + + deleted := windowEvent("alpha", "", FluxFieldManager) + deleted.Operation = string(itypes.OperationDelete) + assert.False(t, loop.withholdLateApply(deleted), "a delete does not say who deleted") + + assert.False(t, loop.withholdLateApply(windowEvent("bravo", "3", "kubectl-edit"))) + assert.False(t, loop.withholdLateApply(windowEvent("bravo", "2", FluxFieldManager)), + "bravo's folder is not the applier's") + assert.Equal(t, int64(0), collisions(t, reader, collisionWithheld)) +} + +// TestApplyWindow_NoWindowBeforeAnAttempt: without an attempt out, the applier applies on its own +// schedule and its writes are mirrored as before. +func TestApplyWindow_NoWindowBeforeAnAttempt(t *testing.T) { + loop, _ := manualLoop(t, applierTestTarget("alpha", true)) + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "3", "kubectl-edit"))) + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager))) +} + +// TestApplyWindow_ClosesAGraceAfterFluxFinished: a delivered attempt keeps its window open a short +// grace, because Flux's writes reach the worker through the watch after Flux reported done; then it +// closes. +func TestApplyWindow_ClosesAGraceAfterFluxFinished(t *testing.T) { + loop, req := sentAttempt(t) + report(req, ApplyDelivered) + loop.serviceAppliers() + + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "3", "kubectl-edit"))) + assert.True(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager)), "within the grace") + + for _, state := range loop.appliers { + state.window.closesAt = time.Now().Add(-time.Millisecond) + } + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager)), "after it") +} + +// TestApplyWindow_AnAttemptThatAppliedNothingHasNoWindow: a stale, failed or dropped attempt +// stamped nothing, so nothing the applier writes afterwards is the apply it asked for. +func TestApplyWindow_AnAttemptThatAppliedNothingHasNoWindow(t *testing.T) { + for _, outcome := range []ApplyOutcome{ApplyStale, ApplyFailed, ApplyRejected, ApplyDropped, ApplyCurrent} { + t.Run(string(outcome), func(t *testing.T) { + loop, req := sentAttempt(t) + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "3", "kubectl-edit"))) + report(req, outcome) + loop.serviceAppliers() + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager))) + }) + } +} diff --git a/internal/git/branch_worker.go b/internal/git/branch_worker.go index 8a9f9513..90e1da33 100644 --- a/internal/git/branch_worker.go +++ b/internal/git/branch_worker.go @@ -1356,6 +1356,9 @@ func (l *branchWorkerEventLoop) handleLiveEvents(request *WriteRequest) { "gitTarget", event.GitTargetNamespace+"/"+event.GitTargetName) continue } + if l.withholdLateApply(event) { + continue + } if l.openWindow != nil && !l.openWindow.canAppend(event) { // Log the identity that broke the window so an unexpected split is // diagnosable: the common cause is an incoming event whose author is empty diff --git a/internal/git/types.go b/internal/git/types.go index 2511cef5..af77b9de 100644 --- a/internal/git/types.go +++ b/internal/git/types.go @@ -660,6 +660,14 @@ type Event struct { // Operation is the admission operation (CREATE, UPDATE, DELETE). Operation string + // LastFieldManagers names who wrote the object last: the managers of its managedFields + // entries for the object itself (not a subresource such as status) that carry the latest + // time. More than one when they tie, since those times resolve to the second. It is PROVENANCE + // like ResourceVersion, stamped where the live watch holds the unsanitized object (sanitize + // removes managedFields); empty for a DELETE and for producers that observed no object. The + // apply window reads it to tell an applier's write from an API edit (apply_window.go). + LastFieldManagers []string + // UserInfo contains user information for commit messages. UserInfo UserInfo diff --git a/internal/telemetry/apply_window.go b/internal/telemetry/apply_window.go new file mode 100644 index 00000000..89d6f429 --- /dev/null +++ b/internal/telemetry/apply_window.go @@ -0,0 +1,23 @@ +// SPDX-License-Identifier: Apache-2.0 + +package telemetry + +import "go.opentelemetry.io/otel/metric" + +// ApplyWindowCollisionsTotal counts writes by an applier, inside its apply window (from the attempt +// being sent until Flux finished it, plus a short grace), to an object the API side edited after the +// attempt was sent: the apply arriving late over a newer edit. Labelled by {provider_namespace, +// provider_name, branch, outcome}. outcome is a frozen enum: +// +// withheld — the write is proven to be the applier's (its field manager alone wrote last), so it +// was not mirrored. Git keeps the API's value, and the next apply puts it back on the +// cluster. The edit was reverted on the cluster until then. +// ambiguous — the applier and another field manager wrote in the same second, so the write may be +// the API edit itself. It was mirrored as ever, which cannot lose an edit; if it was the +// apply, the revert is in Git as the applier's commit. +// +// Any non-zero rate means edits are landing while an apply runs. A long quietPeriod makes the +// CatchUp windows rarer; the ForeignCommit and Refusal windows follow other people's pushes. +// +// Registered in registerCounters (exporter.go) with every other counter. +var ApplyWindowCollisionsTotal metric.Int64Counter diff --git a/internal/telemetry/exporter.go b/internal/telemetry/exporter.go index 36f8b59b..ca6f0f41 100644 --- a/internal/telemetry/exporter.go +++ b/internal/telemetry/exporter.go @@ -597,6 +597,7 @@ func registerCounters() error { {"gitopsreverser_watch_plan_passes_total", &WatchPlanPassesTotal}, {"gitopsreverser_watch_plan_triggers_total", &WatchPlanTriggersTotal}, {"gitopsreverser_apply_requests_total", &ApplyRequestsTotal}, + {"gitopsreverser_apply_window_collisions_total", &ApplyWindowCollisionsTotal}, {"gitopsreverser_receiver_deliveries_total", &ReceiverDeliveriesTotal}, } for _, s := range counters { diff --git a/internal/watch/last_field_managers_test.go b/internal/watch/last_field_managers_test.go new file mode 100644 index 00000000..3211f017 --- /dev/null +++ b/internal/watch/last_field_managers_test.go @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: Apache-2.0 + +package watch + +import ( + "testing" + "time" + + "github.com/stretchr/testify/assert" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + + "github.com/ConfigButler/gitops-reverser/internal/types" +) + +// TestLastFieldManagers pins who counts as the last writer: the latest entry, status left out so a +// controller's status write cannot hide the apply before it, scale kept because it writes replicas, +// and every manager of a same-second tie. +func TestLastFieldManagers(t *testing.T) { + at := func(seconds int) *metav1.Time { + return &metav1.Time{Time: time.Date(2026, 10, 17, 12, 0, seconds, 0, time.UTC)} + } + entry := func(manager, subresource string, time *metav1.Time) metav1.ManagedFieldsEntry { + return metav1.ManagedFieldsEntry{Manager: manager, Subresource: subresource, Time: time} + } + cases := []struct { + name string + entries []metav1.ManagedFieldsEntry + want []string + }{ + {name: "none", want: nil}, + {name: "the latest entry", entries: []metav1.ManagedFieldsEntry{ + entry("kubectl-edit", "", at(1)), entry("kustomize-controller", "", at(2)), + }, want: []string{"kustomize-controller"}}, + {name: "a later status write does not hide the apply", entries: []metav1.ManagedFieldsEntry{ + entry("kustomize-controller", "", at(2)), entry("kube-controller-manager", "status", at(3)), + }, want: []string{"kustomize-controller"}}, + {name: "a later scale write is a write", entries: []metav1.ManagedFieldsEntry{ + entry("kustomize-controller", "", at(2)), entry("horizontal-pod-autoscaler", "scale", at(3)), + }, want: []string{"horizontal-pod-autoscaler"}}, + {name: "a same-second tie names both", entries: []metav1.ManagedFieldsEntry{ + entry("kubectl-edit", "", at(2)), entry("kustomize-controller", "", at(2)), entry("old", "", at(1)), + }, want: []string{"kubectl-edit", "kustomize-controller"}}, + {name: "an entry without a time says nothing", entries: []metav1.ManagedFieldsEntry{ + entry("kubectl-edit", "", at(1)), entry("before-v1.14", "", nil), + }, want: []string{"kubectl-edit"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + u := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "apps/v1", "kind": "Deployment", + "metadata": map[string]any{"name": "api", "namespace": "shop"}, + }} + u.SetManagedFields(tc.entries) + assert.Equal(t, tc.want, lastFieldManagers(u)) + + event := targetWatchGitEvent( + schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"}, + u, + string(types.OperationUpdate), + ) + assert.Equal(t, tc.want, event.LastFieldManagers) + assert.Empty(t, event.Object.GetManagedFields(), "managedFields never reach Git") + }) + } +} diff --git a/internal/watch/target_watch.go b/internal/watch/target_watch.go index 11da8554..679cb4c1 100644 --- a/internal/watch/target_watch.go +++ b/internal/watch/target_watch.go @@ -6,6 +6,7 @@ import ( "context" "errors" "fmt" + "slices" "sort" "strconv" "strings" @@ -1328,11 +1329,34 @@ func targetWatchGitEvent(gvr schema.GroupVersionResource, u *unstructured.Unstru Generation: u.GetGeneration(), } if op != string(types.OperationDelete) { + event.LastFieldManagers = lastFieldManagers(u) event.Object = sanitize.Sanitize(u) } return event } +// lastFieldManagers returns the managers whose managedFields entries carry the latest time, in +// entry order: who wrote the object last, or several when the second-resolution times tie. Status +// entries are left out, because a controller writing status after an apply would otherwise hide who +// made the apply, and status is never mirrored. Other subresources stay: a write to scale is a +// write to spec.replicas. Entries without a time are left out too. +func lastFieldManagers(u *unstructured.Unstructured) []string { + var latest metav1.Time + var managers []string + for _, entry := range u.GetManagedFields() { + if entry.Subresource == "status" || entry.Time == nil { + continue + } + switch { + case entry.Time.After(latest.Time): + latest, managers = *entry.Time, []string{entry.Manager} + case entry.Time.Equal(&latest) && !slices.Contains(managers, entry.Manager): + managers = append(managers, entry.Manager) + } + } + return managers +} + func operationForWatchEvent(eventType watch.EventType) string { switch eventType { case watch.Added: From 8fbc01916c83e7b5c6d9000e909b23675b2724b1 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 06:47:16 +0000 Subject: [PATCH 28/36] docs(design): plan the Argo CD applier, with automated sync off as the standard GitOps Reverser takes full control of when Argo CD syncs: an Application a GitTarget names has automated sync off, and every sync is one GitOps Reverser asked for, pinned to a commit it checked. The plan records the upstream facts it rests on (argocd-controller is the field manager for client-side and server-side apply alike, the operation autosync builds, never replacing a running operation, sync windows apply to every operation), what to build, the tests, and three open questions: where prune comes from, the operation's initiator, and hooks on every foreign commit. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/apply-trigger.md | 3 +- docs/design/argocd-applier-plan.md | 142 +++++++++++++++++++++++++++++ 2 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 docs/design/argocd-applier-plan.md diff --git a/docs/design/apply-trigger.md b/docs/design/apply-trigger.md index 49796063..e775cbfb 100644 --- a/docs/design/apply-trigger.md +++ b/docs/design/apply-trigger.md @@ -213,7 +213,8 @@ webhook cutting a save's window, a `Kustomization` in the management cluster via `spec.applier.clusterProviderRef`), the hardened receiver, and the apply window. After it, in order, each one pull request: -1. **The Argo CD applier**, as described above. +1. **The Argo CD applier**, as described above, with automated sync off as the standard. The + implementation plan is [`argocd-applier-plan.md`](argocd-applier-plan.md). 2. **The webhook applier.** 3. **More than one replica.** Leader election is not switched on, and the chart runs one replica. Standbys need ownership coordination first (the HA plan in `docs/future/`), and deliveries then diff --git a/docs/design/argocd-applier-plan.md b/docs/design/argocd-applier-plan.md new file mode 100644 index 00000000..d593dade --- /dev/null +++ b/docs/design/argocd-applier-plan.md @@ -0,0 +1,142 @@ +# The Argo CD applier: implementation plan + +> **plan**: written 2026-10-17, to be built in a fresh session after (or stacked on) #419. Delete it +> once the PR lands; the lasting design goes into [`apply-trigger.md`](apply-trigger.md). +> Index: [`../INDEX.md`](../INDEX.md). + +**The standard: GitOps Reverser takes full control of when Argo CD syncs.** An `Application` named +by a `GitTarget` has automated sync switched off, and every sync it runs is one GitOps Reverser +asked for, pinned to a commit it checked. This is the supported setup for bi-directional editing +with Argo CD, not an option next to automated sync. + +Argo CD is the easier half of the apply trigger. Flux has no fetch-only mode: a new artifact applies +at once. With automated sync off, Argo CD's timed refresh and its Git webhook only **compare**, so +they can stay on and do no harm, and nothing syncs but us. There is no stale artifact to catch up, +so Argo CD gets no `CatchUp` (already decided; do not re-raise). + +## Start here (new session) + +- Read [`apply-trigger.md`](apply-trigger.md) first: the worker owns every obligation per applier, + the dispatcher makes one attempt and reports back, and quietness is per repository and branch. + None of that changes. +- The Flux applier is the template to copy: [`internal/applier/flux.go`](../../internal/applier/flux.go) + (`Validate`, `Request`, `Handled`), [`internal/applier/dispatcher.go`](../../internal/applier/dispatcher.go), + [`internal/git/apply_trigger.go`](../../internal/git/apply_trigger.go), + [`internal/git/apply_window.go`](../../internal/git/apply_window.go), + [`internal/controller/gittarget_applier.go`](../../internal/controller/gittarget_applier.go). +- The Argo CD upstream checkout is in the gitignored `external-sources/argo-cd` (3.5.0 at + `3cbae653d`). Read it as ground truth; never link to it from a tracked doc. +- The e2e corner that installs Argo CD is `task test-e2e-bi-directional` + ([`argocd_bi_directional_e2e_test.go`](../../test/e2e/argocd_bi_directional_e2e_test.go)). It + occasionally dies on an ECR rate limit pulling `argocd-redis`; rerun later, that is not the diff. + +## Facts verified in the upstream source + +| Fact | Where (external-sources/argo-cd) | Consequence | +| --- | --- | --- | +| Every apply writes as field manager `argocd-controller`, client-side and server-side alike | `gitops-engine/pkg/sync/sync_context.go` passes `serverSideApplyManager` to `ApplyResource`; `gitops-engine/pkg/utils/kube/resource_ops.go:532` sets it as `FieldManager` | The apply window works as for Flux, with `argocd-controller` in place of `kustomize-controller`. `kubectl-client-side-apply` is only the manager Argo CD migrates *from* | +| Autosync builds `Operation{Sync{Revision, Prune: automated.GetPrune(), SyncOptions, Source(s), Revisions}, InitiatedBy{Automated: true}, Retry: spec.syncPolicy.retry or {Limit: 5}}` | `controller/appcontroller.go:2628` | Build ours the same way, with `Revision` pinned to the checked commit | +| Setting an operation refuses while one is set (`ErrAnotherOperationInProgress`) | `util/argo/argo.go` `SetAppOperation` | Never replace a running operation; wait for it | +| Automated sync is on when `syncPolicy.automated` is present and `enabled` is unset or true | `pkg/apis/application/v1alpha1/types.go` `IsAutomatedSyncEnabled` | `AutomatedSyncEnabled` is exactly this rule; `automated: {enabled: false, prune: …}` is the off switch that keeps prune | +| Sync windows are checked for every operation, manual or not, by `InitiatedBy.Automated` | `controller/sync.go` `syncWindowPreventsSync` | A sync we set obeys the project's windows. Which rules apply depends on the `Automated` flag we set (open question 2) | + +To verify while building: the `Replace=true` and create paths (`UpdateResource`, `CreateResource`, +`ReplaceResource` in `sync_context.go`) may write under a different manager; hooks create objects +(`Job`s) during a sync. + +## What to build + +### API + +- **`spec.applier.applicationRef`** (`meta.NamespacedObjectReference`), the Argo CD `Application`. + Exactly one of `kustomizationRef` and `applicationRef` (CEL on `GitTargetApplier`); the namespace + defaults to the `GitTarget`'s, like `kustomizationRef` (the user decided: no namespace gate). +- **`clusterProviderRef`** works unchanged. It is how a hub Argo CD is reached: the `Application` + lives in the management cluster. +- **`quietPeriod`** means nothing without `CatchUp`: refuse it together with `applicationRef` (CEL), + rather than accept a knob that does nothing. +- **`onRefusal: Reapply`** works with `applicationRef`. That closes the known gap where Argo CD + folders lost the automatic refusal revert when `PushEmptyCommit` was removed. + +### The applier interface + +- `git.ApplierRef` gains a `Kind` (`Kustomization` | `Application`), part of the key. +- The worker skips `CatchUp` for an `Application` applier: `sendDue` never owes it one, and its quiet + period is not consulted. `ForeignCommit` and `Refusal` work as for Flux. +- The window's field manager comes from the applier's kind (`FluxFieldManager` today, add + `ArgoCDFieldManager = "argocd-controller"`). +- The dispatcher picks a client by kind behind one small interface (`Validate`, `Request`, + `Handled`), so `attempt` and `awaitHandled` stay as they are. Metrics already carry + `applier_kind`; add `application`. + +### The Argo CD client (`internal/applier/argocd.go`) + +- **Validate** (for `ApplierConfigured`): + - `ApplierNotFound`: no such `Application`, or the CRD is not served. + - `SourceMismatch`: `spec.source.repoURL` is not this repository (`repourl.Same`), or + `targetRevision` is not this branch (a branch name or `refs/heads/`; `HEAD` and tags + mismatch). A multi-source `Application` is refused here in v1, with a message that says so. + - **`AutomatedSyncEnabled`** (new, stalled): `spec.syncPolicy.automated` is present and not + `enabled: false`. The standard is full control, so this is a misconfiguration, not a warning. + - `Forbidden`: a SelfSubjectAccessReview for `patch` on `applications.argoproj.io` fails. +- **Request**: + 1. Read the `Application`. Check the source and the automated-sync rule again (a published + verdict can be older than the object). Either failing is `Rejected`. + 2. If an operation is set or `status.operationState.phase` is `Running`, wait for it to end, + bounded by the apply-window timeout, then read again. Never replace it. + 3. Ask `Current()`. False is `Stale`, as for Flux. + 4. Update the `Application` with `operation` set as autosync builds it (see the facts table), + `Revision` = the commit, `InitiatedBy.Username = "gitops-reverser"`. Use an update guarded by + `resourceVersion`, so a sync somebody else started in between makes it fail instead of + replacing theirs. +- **Handled**: `status.operationState` is for our operation (`initiatedBy.username` and the revision + match) and has `finishedAt`, whatever the phase. A failed sync still closes the window; Argo CD's + own retry belongs to the operation. +- There is no `OutcomeCurrent` for Argo CD: every `ForeignCommit` and `Refusal` syncs. + +### Controller, RBAC and docs + +- `ApplierConfigured` per kind; `applierReadiness` treats `AutomatedSyncEnabled` as stalled. +- RBAC: `config/rbac/argocd-applier-role.yaml` (`get` and `update` on `applications.argoproj.io`) + and Helm `applierAccess.argocd`, off by default, mirroring the Flux grant. +- Docs: a `docs/argocd.md` guide shaped like [`../flux.md`](../flux.md), with the standard spelled + out (automated sync off; refresh and the Argo CD webhook may stay). Update the Argo CD section of + [`../bi-directional.md`](../bi-directional.md), [`../configuration.md`](../configuration.md), + the support-boundary [`argocd-bi-directional.md`](support-boundary/argocd-bi-directional.md), + the metric rows in [`../interpreting-metrics.md`](../interpreting-metrics.md), and the "aim beyond + Flux" section of [`apply-trigger.md`](apply-trigger.md). + +## Tests + +- **Unit**: the operation equals what autosync would build except the pinned revision and the + initiator; a running operation is waited for and never replaced; a concurrent operation makes the + guarded update fail; every `Validate` reason; `Handled` for success, failure and somebody else's + operation; the worker owes an `Application` no `CatchUp`; the window withholds an + `argocd-controller` write over a newer edit. +- **e2e, bi-directional corner**: move the existing Argo CD specs to the standard (template + `Automated: false`, `spec.applier.applicationRef`). Prove: + - two API edits seconds apart keep the later value, with no further commit; + - a Git-side push is synced by a `ForeignCommit` request, pinned to that commit; + - a refused edit with `Reapply` is reverted by an Argo CD sync; + - an `Application` with automated sync on is `AutomatedSyncEnabled` and is never synced. + +## Open questions for the user + +1. **Where prune comes from.** Autosync reads `syncPolicy.automated.prune`, which does not exist + once `automated` is gone. A foreign commit that deletes a manifest needs prune to take effect. + Recommendation: read `automated.prune` when `automated` is present with `enabled: false` (Argo CD + 3.1 and later keeps the block for exactly this), and otherwise sync without prune, documented as + such. Alternative: a `spec.applier.prune` field of our own. +2. **Manual or automated initiator.** `InitiatedBy.Automated` decides which sync-window rules hold + us. Recommendation: `Automated: true` with `Username: "gitops-reverser"`, because we replace + autosync, so a window that blocks automated syncs should block ours too. +3. **Hooks on every foreign commit.** `ForeignCommit` asks every applier on the branch, whatever + folder the push touched. For Flux that is harmless; for Argo CD every sync runs the + `Application`'s hooks. Narrowing it needs the paths a push changed and each `Application`'s + `spec.source.path`. Recommendation: ship without, measure, narrow if it hurts. + +## Out of scope + +Multi-source `Application`s, `ApplicationSet`s (they generate `Application`s whose spec they own), +Argo CD's `syncPolicy.automated.selfHeal` (meaningless with automated sync off), and Argo CD +sharding or its own HA. From 3de49088de4824caaab547ca046ab6f6555947be Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 06:49:54 +0000 Subject: [PATCH 29/36] docs(design): record the Argo CD applier decisions Prune from automated.prune when the block is kept with enabled: false, the initiator Automated: true as gitops-reverser, and hooks on every foreign commit shipped as they are. Built on #419's branch. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/design/argocd-applier-plan.md | 34 ++++++++++++++---------------- 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/docs/design/argocd-applier-plan.md b/docs/design/argocd-applier-plan.md index d593dade..375325b3 100644 --- a/docs/design/argocd-applier-plan.md +++ b/docs/design/argocd-applier-plan.md @@ -1,6 +1,7 @@ # The Argo CD applier: implementation plan -> **plan**: written 2026-10-17, to be built in a fresh session after (or stacked on) #419. Delete it +> **plan**: written 2026-10-17, to be built in a fresh session on #419's branch +> (`docs-bi-directional-right`), before it merges. Delete it > once the PR lands; the lasting design goes into [`apply-trigger.md`](apply-trigger.md). > Index: [`../INDEX.md`](../INDEX.md). @@ -38,7 +39,7 @@ so Argo CD gets no `CatchUp` (already decided; do not re-raise). | Autosync builds `Operation{Sync{Revision, Prune: automated.GetPrune(), SyncOptions, Source(s), Revisions}, InitiatedBy{Automated: true}, Retry: spec.syncPolicy.retry or {Limit: 5}}` | `controller/appcontroller.go:2628` | Build ours the same way, with `Revision` pinned to the checked commit | | Setting an operation refuses while one is set (`ErrAnotherOperationInProgress`) | `util/argo/argo.go` `SetAppOperation` | Never replace a running operation; wait for it | | Automated sync is on when `syncPolicy.automated` is present and `enabled` is unset or true | `pkg/apis/application/v1alpha1/types.go` `IsAutomatedSyncEnabled` | `AutomatedSyncEnabled` is exactly this rule; `automated: {enabled: false, prune: …}` is the off switch that keeps prune | -| Sync windows are checked for every operation, manual or not, by `InitiatedBy.Automated` | `controller/sync.go` `syncWindowPreventsSync` | A sync we set obeys the project's windows. Which rules apply depends on the `Automated` flag we set (open question 2) | +| Sync windows are checked for every operation, manual or not, by `InitiatedBy.Automated` | `controller/sync.go` `syncWindowPreventsSync` | A sync we set obeys the project's windows; with `Automated: true` (decision 2) the automated rules hold | To verify while building: the `Replace=true` and create paths (`UpdateResource`, `CreateResource`, `ReplaceResource` in `sync_context.go`) may write under a different manager; hooks create objects @@ -77,7 +78,8 @@ To verify while building: the `Replace=true` and create paths (`UpdateResource`, `targetRevision` is not this branch (a branch name or `refs/heads/`; `HEAD` and tags mismatch). A multi-source `Application` is refused here in v1, with a message that says so. - **`AutomatedSyncEnabled`** (new, stalled): `spec.syncPolicy.automated` is present and not - `enabled: false`. The standard is full control, so this is a misconfiguration, not a warning. + `enabled: false` (`IsAutomatedSyncEnabled`). The standard is full control, so this is a + misconfiguration, not a warning. - `Forbidden`: a SelfSubjectAccessReview for `patch` on `applications.argoproj.io` fails. - **Request**: 1. Read the `Application`. Check the source and the automated-sync rule again (a published @@ -86,7 +88,8 @@ To verify while building: the `Replace=true` and create paths (`UpdateResource`, bounded by the apply-window timeout, then read again. Never replace it. 3. Ask `Current()`. False is `Stale`, as for Flux. 4. Update the `Application` with `operation` set as autosync builds it (see the facts table), - `Revision` = the commit, `InitiatedBy.Username = "gitops-reverser"`. Use an update guarded by + `Revision` = the commit, `InitiatedBy{Username: "gitops-reverser", Automated: true}`, + `Prune` per decision 1. Use an update guarded by `resourceVersion`, so a sync somebody else started in between makes it fail instead of replacing theirs. - **Handled**: `status.operationState` is for our operation (`initiatedBy.username` and the revision @@ -120,20 +123,15 @@ To verify while building: the `Replace=true` and create paths (`UpdateResource`, - a refused edit with `Reapply` is reverted by an Argo CD sync; - an `Application` with automated sync on is `AutomatedSyncEnabled` and is never synced. -## Open questions for the user - -1. **Where prune comes from.** Autosync reads `syncPolicy.automated.prune`, which does not exist - once `automated` is gone. A foreign commit that deletes a manifest needs prune to take effect. - Recommendation: read `automated.prune` when `automated` is present with `enabled: false` (Argo CD - 3.1 and later keeps the block for exactly this), and otherwise sync without prune, documented as - such. Alternative: a `spec.applier.prune` field of our own. -2. **Manual or automated initiator.** `InitiatedBy.Automated` decides which sync-window rules hold - us. Recommendation: `Automated: true` with `Username: "gitops-reverser"`, because we replace - autosync, so a window that blocks automated syncs should block ours too. -3. **Hooks on every foreign commit.** `ForeignCommit` asks every applier on the branch, whatever - folder the push touched. For Flux that is harmless; for Argo CD every sync runs the - `Application`'s hooks. Narrowing it needs the paths a push changed and each `Application`'s - `spec.source.path`. Recommendation: ship without, measure, narrow if it hurts. +## Decided (2026-10-17, do not re-raise) + +1. **Prune** comes from `syncPolicy.automated.prune` when `automated` is present with + `enabled: false` (the off switch that keeps the block). Without the block, the sync does not + prune, and the guide says so. No `spec.applier.prune` field. +2. **The initiator** is `InitiatedBy{Username: "gitops-reverser", Automated: true}`: we replace + autosync, so a sync window that blocks automated syncs blocks ours too. +3. **Hooks on every foreign commit** ship as they are. `ForeignCommit` keeps asking every applier on + the branch; narrowing by changed paths waits for a measured complaint. ## Out of scope From c5b1f0c4a8d24e6c0ea6662a5690096c783f92b0 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 07:15:54 +0000 Subject: [PATCH 30/36] feat(api): name an Argo CD Application as a GitTarget's applier spec.applier gains applicationRef beside kustomizationRef, exactly one of the two (CEL). quietPeriod times the CatchUp, and Argo CD is never asked to catch up, so it is rejected with an applicationRef; it loses its schema default for that reason, and an omitted one still means 30s. The branch worker keys an applier by its kind as well as its cluster, namespace and name. An Application is never owed a CatchUp: with automated sync off it applies nothing on its own, so nothing it holds goes stale. ForeignCommit and Refusal reach it as they reach a Kustomization. The apply window knows each kind's own field managers: kustomize-controller for Flux, argocd-controller and argocd-application-controller for Argo CD. Co-Authored-By: Claude Opus 5.5 (1M context) --- api/v1alpha3/gittarget_types.go | 96 +++++++++++++---- api/v1alpha3/zz_generated.deepcopy.go | 11 +- .../crd/bases/configbutler.ai_gittargets.yaml | 66 ++++++++---- internal/applier/dispatcher.go | 8 +- internal/applier/dispatcher_test.go | 19 ++-- internal/controller/gittarget_applier.go | 2 +- .../gittarget_applier_schema_test.go | 68 ++++++++++++ internal/controller/gittarget_applier_test.go | 2 +- internal/git/apply_request.go | 28 +++-- internal/git/apply_trigger.go | 12 ++- internal/git/apply_trigger_test.go | 100 ++++++++++++++++-- internal/git/apply_window.go | 57 ++++++++-- internal/git/apply_window_test.go | 27 +++++ internal/git/refusal_reapply_test.go | 2 +- 14 files changed, 411 insertions(+), 87 deletions(-) create mode 100644 internal/controller/gittarget_applier_schema_test.go diff --git a/api/v1alpha3/gittarget_types.go b/api/v1alpha3/gittarget_types.go index cbd7ed81..fb6e8c46 100644 --- a/api/v1alpha3/gittarget_types.go +++ b/api/v1alpha3/gittarget_types.go @@ -95,11 +95,12 @@ type GitTargetSpec struct { // Applier names the object that applies this folder back to its source cluster. With it set, // GitOps Reverser decides when that object applies: it asks for an apply only when the branch // holds no open commit window and no pending writes, after a commit it did not push, after a - // refused edit (spec.onRefusal: Reapply), and, once the branch has been quiet for quietPeriod, - // to keep the applier current with its own pushes. Switch off the applier's own triggers (short - // intervals, Git-host webhooks into it), or they go on applying without that check. The - // reference is looked up in the cluster spec.applier.clusterProviderRef names, by default the - // source cluster, with the identity that ClusterProvider uses. + // refused edit (spec.onRefusal: Reapply), and, for a Kustomization, once the branch has been + // quiet for quietPeriod, to keep it current with its own pushes. Switch off the applier's own + // triggers (short Flux intervals, Git-host webhooks into Flux, Argo CD automated sync), or they + // go on applying without that check. The reference is looked up in the cluster + // spec.applier.clusterProviderRef names, by default the source cluster, with the identity that + // ClusterProvider uses. // +optional Applier *GitTargetApplier `json:"applier,omitempty"` @@ -717,7 +718,15 @@ const ( RefusalActionReapply RefusalAction = "Reapply" ) +// Why quietPeriod is refused rather than ignored with an applicationRef: it times the CatchUp, and +// Argo CD is never asked to catch up (with automated sync off it applies nothing on its own, so it +// holds no stale state to correct). Accepting it would be a knob that does nothing. That is also +// why the field carries no schema default: a default would be present on every object and fail the +// rule for every Application. + // GitTargetApplier names the object that applies a GitTarget's folder to its source cluster. +// +kubebuilder:validation:XValidation:rule="has(self.kustomizationRef) != has(self.applicationRef)",message="spec.applier names exactly one of kustomizationRef and applicationRef" +// +kubebuilder:validation:XValidation:rule="!has(self.applicationRef) || !has(self.quietPeriod)",message="spec.applier.quietPeriod applies only to a kustomizationRef: Argo CD is never asked to catch up" type GitTargetApplier struct { // The same reference spec.clusterProviderRef is, for the same reason: a ClusterProvider is how // GitOps Reverser reaches any cluster, with its credential and its accessFrom gate. Flux run @@ -725,11 +734,12 @@ type GitTargetApplier struct { // cluster cannot be assumed to be the source cluster. Unlike the source, it may change: moving // an applier does not change what the folder means. - // ClusterProviderRef names the ClusterProvider whose cluster holds the Kustomization, such as - // a management cluster running Flux for the source cluster. Omitted, it is the GitTarget's - // own spec.clusterProviderRef. The provider must admit this GitTarget's namespace - // (spec.accessFrom), and its identity needs get on the Kustomization and the GitRepository and - // patch on both. A provider without spec.kubeConfig is the operator's own cluster. + // ClusterProviderRef names the ClusterProvider whose cluster holds the applier, such as a + // management cluster running Flux or Argo CD for the source cluster. Omitted, it is the + // GitTarget's own spec.clusterProviderRef. The provider must admit this GitTarget's namespace + // (spec.accessFrom). Its identity needs get and patch on a Kustomization and its GitRepository, + // or get and update on an Application. A provider without spec.kubeConfig is the operator's own + // cluster. // +optional // +kubebuilder:validation:XValidation:rule="self.name != ''",message="spec.applier.clusterProviderRef.name must not be empty" ClusterProviderRef *meta.LocalObjectReference `json:"clusterProviderRef,omitempty"` @@ -738,23 +748,73 @@ type GitTargetApplier struct { // target's repository and branch. Its namespace defaults to the GitTarget's namespace. GitOps // Reverser asks it to apply the way `flux reconcile kustomization --with-source` does: it // stamps reconcile.fluxcd.io/requestedAt on the GitRepository, and on the Kustomization too - // when the GitRepository's artifact already holds the commit. - // +required - KustomizationRef meta.NamespacedObjectReference `json:"kustomizationRef"` + // when the GitRepository's artifact already holds the commit. Exactly one of kustomizationRef + // and applicationRef is set. + // +optional + KustomizationRef *meta.NamespacedObjectReference `json:"kustomizationRef,omitempty"` + + // ApplicationRef is an Argo CD Application whose spec.source is this target's repository, with + // targetRevision this target's branch (the branch name, or refs/heads/ and the name). Its + // namespace defaults to the GitTarget's namespace. Automated sync must be off: the Application + // either has no spec.syncPolicy.automated, or sets automated.enabled: false, which keeps + // automated.prune for the syncs GitOps Reverser starts. One with automated sync on is + // ApplierConfigured=False (AutomatedSyncEnabled) and is never synced. GitOps Reverser starts + // each sync by setting the Application's operation, as Argo CD's automated sync does, pinned to + // the commit it checked, and never replaces an operation that is running. A multi-source + // Application is not supported. Exactly one of kustomizationRef and applicationRef is set. + // +optional + ApplicationRef *meta.NamespacedObjectReference `json:"applicationRef,omitempty"` // QuietPeriod is how long the branch must hold no open commit window and no pending writes, - // after GitOps Reverser's own pushes, before the applier is asked to catch up with them. Flux - // applies on its own timers and on a controller restart from whatever artifact it holds, so - // an artifact left behind those pushes would revert edits that are already in Git. Every new - // write restarts the wait. A Go duration of at most "1h". Defaults to "30s". + // after GitOps Reverser's own pushes, before a Kustomization is asked to catch up with them. + // Flux applies on its own timers and on a controller restart from whatever artifact it holds, + // so an artifact left behind those pushes would revert edits that are already in Git. Every new + // write restarts the wait. A Go duration of at most "1h". Omitted, it is "30s". Only for a + // kustomizationRef: it is rejected with an applicationRef. // +optional // +kubebuilder:validation:Type=string // +kubebuilder:validation:Pattern="^([0-9]+(\\.[0-9]+)?(ns|us|µs|μs|ms|s|m|h))+$" - // +kubebuilder:default="30s" // +kubebuilder:validation:XValidation:rule="duration(self) <= duration('1h')",message="spec.applier.quietPeriod must be a Go duration of at most 1h" QuietPeriod *metav1.Duration `json:"quietPeriod,omitempty"` } +// The kinds of applier spec.applier can name. +const ( + // ApplierKindKustomization is a Flux Kustomization (spec.applier.kustomizationRef). + ApplierKindKustomization = "Kustomization" + // ApplierKindApplication is an Argo CD Application (spec.applier.applicationRef). + ApplierKindApplication = "Application" +) + +// Kind is the kind of object the applier names: ApplierKindApplication for an applicationRef, +// otherwise ApplierKindKustomization. +func (a *GitTargetApplier) Kind() string { + if a.ApplicationRef != nil { + return ApplierKindApplication + } + return ApplierKindKustomization +} + +// Reference is the object the applier names, whichever kind it is. The zero reference when it +// names none, which the schema rejects. +func (a *GitTargetApplier) Reference() meta.NamespacedObjectReference { + switch { + case a.ApplicationRef != nil: + return *a.ApplicationRef + case a.KustomizationRef != nil: + return *a.KustomizationRef + default: + return meta.NamespacedObjectReference{} + } +} + +// CatchesUp reports whether the applier is asked to catch up with GitOps Reverser's own pushes. +// Only a Kustomization is: an Argo CD Application with automated sync off applies nothing on its +// own, so it never holds a stale state to correct. +func (a *GitTargetApplier) CatchesUp() bool { + return a.Kind() == ApplierKindKustomization +} + // DefaultApplierQuietPeriod is spec.applier.quietPeriod when it is omitted. const DefaultApplierQuietPeriod = 30 * time.Second diff --git a/api/v1alpha3/zz_generated.deepcopy.go b/api/v1alpha3/zz_generated.deepcopy.go index a43deb76..0b0c5113 100644 --- a/api/v1alpha3/zz_generated.deepcopy.go +++ b/api/v1alpha3/zz_generated.deepcopy.go @@ -803,7 +803,16 @@ func (in *GitTargetApplier) DeepCopyInto(out *GitTargetApplier) { *out = new(meta.LocalObjectReference) **out = **in } - out.KustomizationRef = in.KustomizationRef + if in.KustomizationRef != nil { + in, out := &in.KustomizationRef, &out.KustomizationRef + *out = new(meta.NamespacedObjectReference) + **out = **in + } + if in.ApplicationRef != nil { + in, out := &in.ApplicationRef, &out.ApplicationRef + *out = new(meta.NamespacedObjectReference) + **out = **in + } if in.QuietPeriod != nil { in, out := &in.QuietPeriod, &out.QuietPeriod *out = new(v1.Duration) diff --git a/config/crd/bases/configbutler.ai_gittargets.yaml b/config/crd/bases/configbutler.ai_gittargets.yaml index 6e4f27b2..6233c161 100644 --- a/config/crd/bases/configbutler.ai_gittargets.yaml +++ b/config/crd/bases/configbutler.ai_gittargets.yaml @@ -117,19 +117,43 @@ spec: Applier names the object that applies this folder back to its source cluster. With it set, GitOps Reverser decides when that object applies: it asks for an apply only when the branch holds no open commit window and no pending writes, after a commit it did not push, after a - refused edit (spec.onRefusal: Reapply), and, once the branch has been quiet for quietPeriod, - to keep the applier current with its own pushes. Switch off the applier's own triggers (short - intervals, Git-host webhooks into it), or they go on applying without that check. The - reference is looked up in the cluster spec.applier.clusterProviderRef names, by default the - source cluster, with the identity that ClusterProvider uses. + refused edit (spec.onRefusal: Reapply), and, for a Kustomization, once the branch has been + quiet for quietPeriod, to keep it current with its own pushes. Switch off the applier's own + triggers (short Flux intervals, Git-host webhooks into Flux, Argo CD automated sync), or they + go on applying without that check. The reference is looked up in the cluster + spec.applier.clusterProviderRef names, by default the source cluster, with the identity that + ClusterProvider uses. properties: + applicationRef: + description: |- + ApplicationRef is an Argo CD Application whose spec.source is this target's repository, with + targetRevision this target's branch (the branch name, or refs/heads/ and the name). Its + namespace defaults to the GitTarget's namespace. Automated sync must be off: the Application + either has no spec.syncPolicy.automated, or sets automated.enabled: false, which keeps + automated.prune for the syncs GitOps Reverser starts. One with automated sync on is + ApplierConfigured=False (AutomatedSyncEnabled) and is never synced. GitOps Reverser starts + each sync by setting the Application's operation, as Argo CD's automated sync does, pinned to + the commit it checked, and never replaces an operation that is running. A multi-source + Application is not supported. Exactly one of kustomizationRef and applicationRef is set. + properties: + name: + description: Name of the referent. + type: string + namespace: + description: Namespace of the referent, when not specified + it acts as LocalObjectReference. + type: string + required: + - name + type: object clusterProviderRef: description: |- - ClusterProviderRef names the ClusterProvider whose cluster holds the Kustomization, such as - a management cluster running Flux for the source cluster. Omitted, it is the GitTarget's - own spec.clusterProviderRef. The provider must admit this GitTarget's namespace - (spec.accessFrom), and its identity needs get on the Kustomization and the GitRepository and - patch on both. A provider without spec.kubeConfig is the operator's own cluster. + ClusterProviderRef names the ClusterProvider whose cluster holds the applier, such as a + management cluster running Flux or Argo CD for the source cluster. Omitted, it is the + GitTarget's own spec.clusterProviderRef. The provider must admit this GitTarget's namespace + (spec.accessFrom). Its identity needs get and patch on a Kustomization and its GitRepository, + or get and update on an Application. A provider without spec.kubeConfig is the operator's own + cluster. properties: name: description: Name of the referent. @@ -146,7 +170,8 @@ spec: target's repository and branch. Its namespace defaults to the GitTarget's namespace. GitOps Reverser asks it to apply the way `flux reconcile kustomization --with-source` does: it stamps reconcile.fluxcd.io/requestedAt on the GitRepository, and on the Kustomization too - when the GitRepository's artifact already holds the commit. + when the GitRepository's artifact already holds the commit. Exactly one of kustomizationRef + and applicationRef is set. properties: name: description: Name of the referent. @@ -159,22 +184,27 @@ spec: - name type: object quietPeriod: - default: 30s description: |- QuietPeriod is how long the branch must hold no open commit window and no pending writes, - after GitOps Reverser's own pushes, before the applier is asked to catch up with them. Flux - applies on its own timers and on a controller restart from whatever artifact it holds, so - an artifact left behind those pushes would revert edits that are already in Git. Every new - write restarts the wait. A Go duration of at most "1h". Defaults to "30s". + after GitOps Reverser's own pushes, before a Kustomization is asked to catch up with them. + Flux applies on its own timers and on a controller restart from whatever artifact it holds, + so an artifact left behind those pushes would revert edits that are already in Git. Every new + write restarts the wait. A Go duration of at most "1h". Omitted, it is "30s". Only for a + kustomizationRef: it is rejected with an applicationRef. pattern: ^([0-9]+(\.[0-9]+)?(ns|us|µs|μs|ms|s|m|h))+$ type: string x-kubernetes-validations: - message: spec.applier.quietPeriod must be a Go duration of at most 1h rule: duration(self) <= duration('1h') - required: - - kustomizationRef type: object + x-kubernetes-validations: + - message: spec.applier names exactly one of kustomizationRef and + applicationRef + rule: has(self.kustomizationRef) != has(self.applicationRef) + - message: 'spec.applier.quietPeriod applies only to a kustomizationRef: + Argo CD is never asked to catch up' + rule: '!has(self.applicationRef) || !has(self.quietPeriod)' branch: description: |- Branch to use for this target. diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index 6209cecd..a0d38abc 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -402,13 +402,7 @@ func namesApplier(target *v1alpha3.GitTarget, req git.ApplyRequest) bool { target.Spec.GitProviderRef.Name != req.Branch.RepoName || target.Spec.Branch != req.Branch.Branch { return false } - ref := target.Spec.Applier.KustomizationRef - namespace := ref.Namespace - if namespace == "" { - namespace = target.Namespace - } - return target.ApplierCluster() == req.Applier.Cluster && namespace == req.Applier.Namespace && - ref.Name == req.Applier.Name + return git.ApplierOf(target) == req.Applier } // delivered records a delivered request in the ledger and as an ApplyRequested Event on every diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index 6d889cc6..cbd77127 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -49,7 +49,7 @@ func gitTarget(name string, kustomizationRef *fluxmeta.NamespacedObjectReference }, } if kustomizationRef != nil { - target.Spec.Applier = &v1alpha3.GitTargetApplier{KustomizationRef: *kustomizationRef} + target.Spec.Applier = &v1alpha3.GitTargetApplier{KustomizationRef: kustomizationRef} } if cluster != "" { target.Spec.ClusterProviderRef = &fluxmeta.LocalObjectReference{Name: cluster} @@ -94,8 +94,13 @@ func ref(name string) itypes.ResourceReference { return itypes.NewResourceRefere // uid is the uid gitTarget gives the GitTarget of that name. func uid(name string) types.UID { return types.UID("uid-" + name) } +// kustomizationApplier is a Kustomization applier in a cluster. +func kustomizationApplier(cluster, namespace, name string) git.ApplierRef { + return git.ApplierRef{Cluster: cluster, Kind: v1alpha3.ApplierKindKustomization, Namespace: namespace, Name: name} +} + // checkout is the applier most tests ask: flux-system/checkout in the default cluster. -var checkout = git.ApplierRef{Cluster: v1alpha3.DefaultClusterProviderName, Namespace: fluxSystemNS, Name: "checkout"} +var checkout = kustomizationApplier(v1alpha3.DefaultClusterProviderName, fluxSystemNS, "checkout") // request is one attempt for an applier to apply testCommit, for the named targets, all asking for // reason. @@ -238,7 +243,7 @@ func TestDispatcher_AsksOnceForEveryTarget(t *testing.T) { // it names, under its own namespace and name; here a GitTarget whose kustomizationRef omits the // namespace and so names one in its own. func TestDispatcher_LooksTheApplierUpInItsCluster(t *testing.T) { - own := git.ApplierRef{Cluster: "prod-eu-1", Namespace: targetNS, Name: "own"} + own := kustomizationApplier("prod-eu-1", targetNS, "own") f := newDispatcherFixture(t, []client.Object{gitTarget("c", &fluxmeta.NamespacedObjectReference{Name: "own"}, "prod-eu-1")}, kustomizationObj(targetNS, "own", gitRepoSourceRef("own-source")), @@ -259,7 +264,7 @@ func appliedFrom(target *v1alpha3.GitTarget, cluster string) *v1alpha3.GitTarget // TestDispatcher_AsksAnApplierInItsOwnCluster: a GitTarget mirroring prod-eu-1 whose Kustomization // lives in a management cluster is asked through that cluster's client, never the source's. func TestDispatcher_AsksAnApplierInItsOwnCluster(t *testing.T) { - hub := git.ApplierRef{Cluster: "hub", Namespace: fluxSystemNS, Name: "checkout"} + hub := kustomizationApplier("hub", fluxSystemNS, "checkout") f := newDispatcherFixture( t, []client.Object{ @@ -274,7 +279,7 @@ func TestDispatcher_AsksAnApplierInItsOwnCluster(t *testing.T) { // The same Kustomization asked as if it lived in the source cluster is not this target's applier. f.source.clusters = nil - sourceSide := git.ApplierRef{Cluster: "prod-eu-1", Namespace: fluxSystemNS, Name: "checkout"} + sourceSide := kustomizationApplier("prod-eu-1", fluxSystemNS, "checkout") assert.Equal(t, git.ApplyDropped, f.attempt(request(sourceSide, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) assert.Empty(t, f.source.clusters) } @@ -283,7 +288,7 @@ func TestDispatcher_AsksAnApplierInItsOwnCluster(t *testing.T) { // may wield its credential exactly as a source's does, so a GitTarget its accessFrom does not admit // asks nothing, though its own source admits it. func TestDispatcher_AnApplierClusterThatDoesNotAdmitAsksNothing(t *testing.T) { - hub := git.ApplierRef{Cluster: "hub", Namespace: fluxSystemNS, Name: "checkout"} + hub := kustomizationApplier("hub", fluxSystemNS, "checkout") f := newDispatcherFixture(t, []client.Object{ appliedFrom(gitTarget("a", checkoutRef(), "prod-eu-1"), "hub"), @@ -340,7 +345,7 @@ func TestDispatcher_ASourceMismatchIsRejectedAndNeverStamped(t *testing.T) { // source client is shared by every GitTarget that declared the cluster, so a GitTarget its // ClusterProvider does not admit drove the applier on another GitTarget's admission. func TestDispatcher_ATargetItsClusterProviderDeniesAsksNothing(t *testing.T) { - restricted := git.ApplierRef{Cluster: "restricted", Namespace: fluxSystemNS, Name: "checkout"} + restricted := kustomizationApplier("restricted", fluxSystemNS, "checkout") f := newDispatcherFixture(t, []client.Object{ gitTarget("denied", checkoutRef(), "restricted"), diff --git a/internal/controller/gittarget_applier.go b/internal/controller/gittarget_applier.go index ce99bb8b..51b12d71 100644 --- a/internal/controller/gittarget_applier.go +++ b/internal/controller/gittarget_applier.go @@ -131,7 +131,7 @@ func keptOrUnknown(target *configbutleraiv1alpha3.GitTarget, message string) con // applierReference is spec.applier.kustomizationRef, its namespace defaulting to the GitTarget's. func applierReference(target *configbutleraiv1alpha3.GitTarget) k8stypes.NamespacedName { - ref := target.Spec.Applier.KustomizationRef + ref := target.Spec.Applier.Reference() namespace := ref.Namespace if namespace == "" { namespace = target.Namespace diff --git a/internal/controller/gittarget_applier_schema_test.go b/internal/controller/gittarget_applier_schema_test.go new file mode 100644 index 00000000..995bfbd5 --- /dev/null +++ b/internal/controller/gittarget_applier_schema_test.go @@ -0,0 +1,68 @@ +// SPDX-License-Identifier: Apache-2.0 + +package controller + +import ( + "context" + "time" + + meta "github.com/fluxcd/pkg/apis/meta" + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + + configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" +) + +// spec.applier names exactly one applier, of one kind, and quietPeriod (which times a CatchUp) +// belongs to a Kustomization alone: Argo CD is never asked to catch up. The API server enforces +// both, so a knob that would do nothing is never accepted. +var _ = Describe("GitTarget applier schema", func() { + kustomization := &meta.NamespacedObjectReference{Namespace: "flux-system", Name: "apps"} + application := &meta.NamespacedObjectReference{Namespace: "argocd", Name: "apps"} + quiet := &metav1.Duration{Duration: 5 * time.Second} + + create := func(name string, applier *configbutleraiv1alpha3.GitTargetApplier) error { + ctx := context.Background() + target := &configbutleraiv1alpha3.GitTarget{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "default"}, + Spec: configbutleraiv1alpha3.GitTargetSpec{ + GitProviderRef: meta.LocalObjectReference{Name: "prov"}, + Branch: "main", + Path: "apps/" + name, + Applier: applier, + }, + } + err := k8sClient.Create(ctx, target) + if err == nil { + DeferCleanup(func() { _ = k8sClient.Delete(ctx, target) }) + } + return err + } + + It("accepts a Kustomization, with or without a quiet period, and an Application without one", func() { + Expect(create("applier-kustomization", &configbutleraiv1alpha3.GitTargetApplier{ + KustomizationRef: kustomization, QuietPeriod: quiet, + })).To(Succeed()) + Expect(create("applier-kustomization-default", &configbutleraiv1alpha3.GitTargetApplier{ + KustomizationRef: kustomization, + })).To(Succeed()) + Expect(create("applier-application", &configbutleraiv1alpha3.GitTargetApplier{ + ApplicationRef: application, + })).To(Succeed()) + }) + + It("rejects both kinds at once, and neither", func() { + Expect(create("applier-both", &configbutleraiv1alpha3.GitTargetApplier{ + KustomizationRef: kustomization, ApplicationRef: application, + })).To(MatchError(ContainSubstring("exactly one of kustomizationRef and applicationRef"))) + Expect(create("applier-neither", &configbutleraiv1alpha3.GitTargetApplier{})). + To(MatchError(ContainSubstring("exactly one of kustomizationRef and applicationRef"))) + }) + + It("rejects a quiet period on an Application", func() { + Expect(create("applier-application-quiet", &configbutleraiv1alpha3.GitTargetApplier{ + ApplicationRef: application, QuietPeriod: quiet, + })).To(MatchError(ContainSubstring("quietPeriod applies only to a kustomizationRef"))) + }) +}) diff --git a/internal/controller/gittarget_applier_test.go b/internal/controller/gittarget_applier_test.go index b6a9aaa6..0e5149c2 100644 --- a/internal/controller/gittarget_applier_test.go +++ b/internal/controller/gittarget_applier_test.go @@ -77,7 +77,7 @@ func applierTarget(applierSet bool) *configbutleraiv1alpha3.GitTarget { } if applierSet { target.Spec.Applier = &configbutleraiv1alpha3.GitTargetApplier{ - KustomizationRef: meta.NamespacedObjectReference{Namespace: "flux-system", Name: "checkout"}, + KustomizationRef: &meta.NamespacedObjectReference{Namespace: "flux-system", Name: "checkout"}, } } return target diff --git a/internal/git/apply_request.go b/internal/git/apply_request.go index 4f3c11fa..fce8622e 100644 --- a/internal/git/apply_request.go +++ b/internal/git/apply_request.go @@ -12,26 +12,38 @@ import ( itypes "github.com/ConfigButler/gitops-reverser/internal/types" ) -// ApplierRef is one applier, as the GitTargets on a branch name it: a Flux Kustomization in a -// source cluster. Several GitTargets may name one applier, and it is asked once for all of them. +// ApplierRef is one applier, as the GitTargets on a branch name it: a Flux Kustomization or an +// Argo CD Application in a cluster. Several GitTargets may name one applier, and it is asked once +// for all of them. type ApplierRef struct { // Cluster is the ClusterProvider whose cluster the applier lives in (GitTarget.ApplierCluster()). Cluster string - // Namespace and Name are the Kustomization's, the namespace defaulted to the GitTarget's own. + // Kind is v1alpha3.ApplierKindKustomization or v1alpha3.ApplierKindApplication. + Kind string + // Namespace and Name are the object's, the namespace defaulted to the GitTarget's own. Namespace string Name string } -func (a ApplierRef) String() string { return a.Cluster + "/" + a.Namespace + "/" + a.Name } +func (a ApplierRef) String() string { + return a.Cluster + "/" + a.Kind + "/" + a.Namespace + "/" + a.Name +} + +// CatchesUp reports whether the applier is owed a CatchUp after GitOps Reverser's own pushes. An +// Argo CD Application is not: with automated sync off it applies nothing on its own, so it holds no +// stale state for a catch-up to correct. +func (a ApplierRef) CatchesUp() bool { return a.Kind != v1alpha3.ApplierKindApplication } -// applierOf is the applier a GitTarget names. Call it only for a target with spec.applier. -func applierOf(target *v1alpha3.GitTarget) ApplierRef { - ref := target.Spec.Applier.KustomizationRef +// ApplierOf is the applier a GitTarget names. Call it only for a target with spec.applier. +func ApplierOf(target *v1alpha3.GitTarget) ApplierRef { + ref := target.Spec.Applier.Reference() namespace := ref.Namespace if namespace == "" { namespace = target.Namespace } - return ApplierRef{Cluster: target.ApplierCluster(), Namespace: namespace, Name: ref.Name} + return ApplierRef{ + Cluster: target.ApplierCluster(), Kind: target.Spec.Applier.Kind(), Namespace: namespace, Name: ref.Name, + } } // ApplyRequest is one attempt, decided by a branch worker on a quiet branch, to have one applier diff --git a/internal/git/apply_trigger.go b/internal/git/apply_trigger.go index 912cbbe2..def003d0 100644 --- a/internal/git/apply_trigger.go +++ b/internal/git/apply_trigger.go @@ -31,7 +31,9 @@ import ( // - Refusal: a refused edit on a target with spec.onRefusal: Reapply (refusal_reapply.go). // - CatchUp: this worker's own pushes. Flux applies on its own timers and on a restart from // whatever artifact it holds, and an artifact behind these pushes would revert edits already in -// Git. Asked only once the branch has stayed quiet for the applier's quiet period. +// Git. Asked only once the branch has stayed quiet for the applier's quiet period, and never of +// an Argo CD Application: with automated sync off it applies nothing on its own, so it holds no +// stale state to correct (ApplierRef.CatchesUp). // // The three relations are kept apart on purpose. The BRANCH is the unit of quietness, because a // Kustomization applies the tip whichever GitTarget's folder an edit is in. The APPLIER is the unit @@ -161,7 +163,7 @@ func (w *BranchWorker) appliersOnBranch(ctx context.Context) ([]branchTarget, er listed = append(listed, branchTarget{ ref: itypes.NewResourceReference(target.Name, target.Namespace), uid: target.UID, - applier: applierOf(target), + applier: ApplierOf(target), quietPeriod: target.Spec.Applier.EffectiveQuietPeriod(), consents: ConsentsToReapply(target), }) @@ -185,8 +187,8 @@ type applierState struct { // next attempt may go. failures int retryAt time.Time - // window is the applier's apply window: open from sending an attempt until Flux finished it - // (apply_window.go). Nil when no attempt was sent, or the last asked for no apply. + // window is the applier's apply window: open from sending an attempt until the applier + // finished it (apply_window.go). Nil when no attempt was sent, or the last asked for no apply. window *applyWindow } @@ -266,7 +268,7 @@ func (l *branchWorkerEventLoop) serviceAppliers() { l.syncAppliers(targets) for _, state := range l.appliers { state.foreign = state.foreign || foreign - state.catchUp = state.catchUp || catchUp + state.catchUp = state.catchUp || (catchUp && state.ref.CatchesUp()) } l.armApplierTimer(now, l.sendDue(tip, now)) } diff --git a/internal/git/apply_trigger_test.go b/internal/git/apply_trigger_test.go index a53f8e86..e48fa710 100644 --- a/internal/git/apply_trigger_test.go +++ b/internal/git/apply_trigger_test.go @@ -78,7 +78,10 @@ func TestApplyDebt_RestorePutsBackWhatWasTaken(t *testing.T) { // applierRef is the applier testApplier(name) names for a GitTarget in team-a. func applierRef(name string) ApplierRef { - return ApplierRef{Cluster: configv1alpha3.DefaultClusterProviderName, Namespace: "team-a", Name: name} + return ApplierRef{ + Cluster: configv1alpha3.DefaultClusterProviderName, Kind: configv1alpha3.ApplierKindKustomization, + Namespace: "team-a", Name: name, + } } // applierTestTarget is a GitTarget on the trigger worker's provider and branch unless overridden. @@ -148,8 +151,7 @@ func TestServiceAppliers_ForeignCommitAsksEveryApplierOnTheBranch(t *testing.T) assert.Equal(t, "bbb", requests[i].Commit) assert.Equal(t, BranchKey{RepoNamespace: "team-a", RepoName: "test-provider", Branch: "main"}, requests[i].Branch) - assert.Equal(t, ApplierRef{Cluster: configv1alpha3.DefaultClusterProviderName, Namespace: "team-a", Name: name}, - requests[i].Applier) + assert.Equal(t, applierRef(name), requests[i].Applier) assert.Equal(t, []itypes.ResourceReference{itypes.NewResourceReference(name, "team-a")}, requests[i].TargetRefs(), "a suspended target stops writing, not deployments") } @@ -211,6 +213,74 @@ func TestServiceAppliers_CatchUpWaitsForTheQuietPeriod(t *testing.T) { assert.NotNil(t, loop.applierTimer) } +// argoApplier makes a GitTarget's applier an Argo CD Application of its own name. +func argoApplier(g *configv1alpha3.GitTarget) { + g.Spec.Applier = &configv1alpha3.GitTargetApplier{ + ApplicationRef: &meta.NamespacedObjectReference{Namespace: "argocd", Name: g.Name}, + } +} + +// TestServiceAppliers_AnApplicationIsNeverOwedACatchUp: with automated sync off Argo CD applies +// nothing on its own, so GitOps Reverser's own pushes (and the baseline after a start) owe an +// Application nothing, however long the branch stays quiet; a Kustomization beside it on the same +// branch is still caught up. +func TestServiceAppliers_AnApplicationIsNeverOwedACatchUp(t *testing.T) { + loop, requester := applyTriggerLoop(t, + applierTestTarget("argo", true, argoApplier), + applierTestTarget("flux", true), + ) + loop.w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByFetch}) // the baseline + loop.serviceAppliers() + loop.w.debt.observe(RemoteObservation{Commit: "bbb", By: ObservedByPush}) + loop.serviceAppliers() + loop.quietSince = time.Now().Add(-time.Hour) + loop.serviceAppliers() + + requests := requester.taken() + require.Len(t, requests, 1, "only the Kustomization catches up") + assert.Equal(t, applierRef("flux"), requests[0].Applier) + assert.Equal(t, configv1alpha3.ApplyReasonCatchUp, requests[0].Reason) + argo := ApplierRef{ + Cluster: configv1alpha3.DefaultClusterProviderName, Kind: configv1alpha3.ApplierKindApplication, + Namespace: "argocd", Name: "argo", + } + require.Contains(t, loop.appliers, argo) + assert.False(t, loop.appliers[argo].catchUp) +} + +// TestServiceAppliers_AnApplicationIsAskedOnAForeignCommitAndARefusal: the two reasons that are +// not catching up reach an Application as they reach a Kustomization. +func TestServiceAppliers_AnApplicationIsAskedOnAForeignCommitAndARefusal(t *testing.T) { + argo := ApplierRef{ + Cluster: configv1alpha3.DefaultClusterProviderName, Kind: configv1alpha3.ApplierKindApplication, + Namespace: "argocd", Name: "argo", + } + + t.Run("a foreign commit", func(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("argo", true, argoApplier)) + owesForeign(loop.w) + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1) + assert.Equal(t, argo, requests[0].Applier) + assert.Equal(t, configv1alpha3.ApplyReasonForeignCommit, requests[0].Reason) + assert.Equal(t, "bbb", requests[0].Commit) + }) + + t.Run("a refusal", func(t *testing.T) { + loop, requester := applyTriggerLoop(t, applierTestTarget("argo", true, argoApplier, + func(g *configv1alpha3.GitTarget) { g.Spec.OnRefusal = configv1alpha3.RefusalActionReapply })) + loop.w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByFetch}) + loop.oweReapply(refusalKeyFor(itypes.NewResourceReference("argo", "team-a"), "deployments"), + owedRefusal{detail: "refused", uid: "uid-argo"}, "obs") + loop.serviceAppliers() + requests := requester.taken() + require.Len(t, requests, 1, "a refusal is asked at once, with no quiet period") + assert.Equal(t, argo, requests[0].Applier) + assert.Equal(t, configv1alpha3.ApplyReasonRefusal, requests[0].Reason) + }) +} + func TestServiceAppliers_AWriteRestartsTheQuietPeriod(t *testing.T) { loop, requester := applyTriggerLoop(t, applierTestTarget("alpha", true)) loop.w.debt.observe(RemoteObservation{Commit: "aaa", By: ObservedByPush}) @@ -433,20 +503,32 @@ func TestApplyTrigger_ARejectedPushOwesTheReplayedTip(t *testing.T) { assert.Equal(t, "from-another-writer", gitOut(t, f.repoDir, "show", head+":OUTSIDE.md")) } -// TestApplierOf_KeysTheApplierByItsOwnCluster: one Kustomization name in two clusters is two -// appliers, and spec.applier.clusterProviderRef, not the source, decides which cluster. -func TestApplierOf_KeysTheApplierByItsOwnCluster(t *testing.T) { +// TestApplierOf_KeysTheApplierByItsOwnClusterAndKind: one name in two clusters is two appliers, and +// spec.applier.clusterProviderRef, not the source, decides which cluster. A Kustomization and an +// Application of one name are two appliers too. +func TestApplierOf_KeysTheApplierByItsOwnClusterAndKind(t *testing.T) { target := &configv1alpha3.GitTarget{ ObjectMeta: metav1.ObjectMeta{Namespace: "team-a", Name: "apps"}, Spec: configv1alpha3.GitTargetSpec{ ClusterProviderRef: &meta.LocalObjectReference{Name: "prod-eu-1"}, Applier: &configv1alpha3.GitTargetApplier{ - KustomizationRef: meta.NamespacedObjectReference{Name: "apps"}, + KustomizationRef: &meta.NamespacedObjectReference{Name: "apps"}, }, }, } - assert.Equal(t, ApplierRef{Cluster: "prod-eu-1", Namespace: "team-a", Name: "apps"}, applierOf(target)) + kustomization := configv1alpha3.ApplierKindKustomization + assert.Equal(t, ApplierRef{Cluster: "prod-eu-1", Kind: kustomization, Namespace: "team-a", Name: "apps"}, + ApplierOf(target)) target.Spec.Applier.ClusterProviderRef = &meta.LocalObjectReference{Name: "hub"} - assert.Equal(t, ApplierRef{Cluster: "hub", Namespace: "team-a", Name: "apps"}, applierOf(target)) + assert.Equal(t, ApplierRef{Cluster: "hub", Kind: kustomization, Namespace: "team-a", Name: "apps"}, + ApplierOf(target)) + + target.Spec.Applier.KustomizationRef = nil + target.Spec.Applier.ApplicationRef = &meta.NamespacedObjectReference{Namespace: "argocd", Name: "apps"} + app := ApplierOf(target) + assert.Equal(t, ApplierRef{ + Cluster: "hub", Kind: configv1alpha3.ApplierKindApplication, Namespace: "argocd", Name: "apps", + }, app) + assert.False(t, app.CatchesUp(), "an Application is never asked to catch up") } diff --git a/internal/git/apply_window.go b/internal/git/apply_window.go index 963f0f2a..f4e0afae 100644 --- a/internal/git/apply_window.go +++ b/internal/git/apply_window.go @@ -10,14 +10,15 @@ import ( "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/metric" + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" "github.com/ConfigButler/gitops-reverser/internal/telemetry" itypes "github.com/ConfigButler/gitops-reverser/internal/types" ) // The apply window is the time an apply GitOps Reverser asked for is running. An API edit that -// lands in it, to a field the applier applies, is reverted by that apply: Flux applies the commit -// it was asked for, which does not hold the edit yet. Mirroring that write would commit the revert -// as Flux's, and the edit would survive only in history. +// lands in it, to a field the applier applies, is reverted by that apply: Flux or Argo CD applies +// the commit it was asked for, which does not hold the edit yet. Mirroring that write would commit +// the revert as the applier's, and the edit would survive only in history. // // The repository's rule for an object both sides changed is that the API value wins. Applied to // the window: a write by the applier's own field manager, during an attempt, to an object the API @@ -25,8 +26,8 @@ import ( // is withheld. Git keeps the edit, and the CatchUp that edit's push owes applies it back. // // What the rule must never do is withhold an API edit, which would lose it outright. So it acts -// only on proof: the object's latest managedFields entry is the applier's manager alone. When the -// second-resolution times tie with another manager the write could be either, and it is mirrored +// only on proof: the object's latest managedFields entries are the applier's managers alone. When +// the second-resolution times tie with another manager the write could be either, and it is mirrored // as before; the collision is logged and counted as ambiguous so it is still seen. A DELETE carries // no managedFields of its deleter and is never withheld. @@ -35,8 +36,42 @@ import ( // cmd/cli/migrate_owner.go, and docs/facts/expansion-provenance-markers.md). const FluxFieldManager = "kustomize-controller" -// applyWindowGrace keeps a window open after Flux reported the apply finished. Flux's writes reach -// the worker through the watch, which can trail the status the dispatcher read. +// ArgoCDFieldManager is the field manager Argo CD's application controller applies with, for a +// client-side and a server-side apply alike (ArgoCDSSAManager in external-sources/argo-cd +// common/common.go, passed to every ApplyResource by gitops-engine's sync_context.go). +const ArgoCDFieldManager = "argocd-controller" + +// ArgoCDDefaultFieldManager is the manager of Argo CD's writes that name none: a sync with +// Replace=true creates and replaces through kubectl's create and replace options without a field +// manager (gitops-engine resource_ops.go), so the API server takes it from the user agent, which +// client-go defaults to the binary's name, /usr/local/bin/argocd-application-controller. +const ArgoCDDefaultFieldManager = "argocd-application-controller" + +// fieldManagers are the managers an applier of this kind writes with: a write by one of them alone +// is the applier's apply. +func (a ApplierRef) fieldManagers() []string { + if a.Kind == v1alpha3.ApplierKindApplication { + return []string{ArgoCDFieldManager, ArgoCDDefaultFieldManager} + } + return []string{FluxFieldManager} +} + +// writtenBy reports whether any of the managers is one of the applier's, and whether all are. +func (a ApplierRef) writtenBy(managers []string) (bool, bool) { + own := a.fieldManagers() + some, all := false, len(managers) > 0 + for _, manager := range managers { + if slices.Contains(own, manager) { + some = true + } else { + all = false + } + } + return some, all +} + +// applyWindowGrace keeps a window open after the applier reported the apply finished. Its writes +// reach the worker through the watch, which can trail the status the dispatcher read. const applyWindowGrace = 10 * time.Second // Apply-window collision outcomes, the outcome label of apply_window_collisions_total. @@ -45,8 +80,8 @@ const ( collisionAmbiguous = "ambiguous" ) -// applyWindow is one applier's open apply window: from sending an attempt until Flux finished it, -// plus applyWindowGrace. +// applyWindow is one applier's open apply window: from sending an attempt until the applier +// finished it, plus applyWindowGrace. type applyWindow struct { attempt uint64 commit string @@ -103,13 +138,13 @@ func (l *branchWorkerEventLoop) withholdLateApply(event Event) bool { } now := time.Now() key := objectKey(event.Identifier) - byApplier := slices.Contains(event.LastFieldManagers, FluxFieldManager) for _, ref := range l.sortedAppliers() { applier := l.appliers[ref] window := applier.window if !window.open(now) || !applier.covers(event) { continue } + byApplier, byApplierAlone := ref.writtenBy(event.LastFieldManagers) _, editedSince := window.edited[key] switch { case !byApplier || event.Operation == string(itypes.OperationDelete): @@ -117,7 +152,7 @@ func (l *branchWorkerEventLoop) withholdLateApply(event Event) bool { case !editedSince: // The applier applying an object nobody edited since the attempt: its content is the // commit's, which Git already holds. Mirrored as ever. - case len(event.LastFieldManagers) > 1: + case !byApplierAlone: l.recordApplyWindowCollision(applier, window, event, collisionAmbiguous) // It may be the API edit itself, written in the same second: mirroring it is the // choice that cannot lose an edit. It counts as one, too. diff --git a/internal/git/apply_window_test.go b/internal/git/apply_window_test.go index 0b56228b..83e1e103 100644 --- a/internal/git/apply_window_test.go +++ b/internal/git/apply_window_test.go @@ -152,3 +152,30 @@ func TestApplyWindow_AnAttemptThatAppliedNothingHasNoWindow(t *testing.T) { }) } } + +// TestApplyWindow_ArgoCDsLateWriteIsWithheldByItsOwnManagers: an Argo CD Application's window +// knows Argo CD's managers, its apply (argocd-controller) and its Replace=true create and replace +// (argocd-application-controller), and not Flux's. +func TestApplyWindow_ArgoCDsLateWriteIsWithheldByItsOwnManagers(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + loop, requester := manualLoop(t, applierTestTarget("alpha", true, argoApplier)) + owesForeign(loop.w) + loop.serviceAppliers() + require.Len(t, requester.taken(), 1) + + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "3", "kubectl-edit")), "an API edit is mirrored") + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "2", FluxFieldManager)), + "Flux is not this applier: its write is an edit like any other") + assert.True(t, loop.withholdLateApply(windowEvent("alpha", "3", ArgoCDFieldManager)), + "Argo CD's apply over the newer edit is withheld") + assert.True(t, loop.withholdLateApply(windowEvent("alpha", "3", ArgoCDDefaultFieldManager)), + "so is its replace") + assert.True(t, loop.withholdLateApply(windowEvent("alpha", "3", ArgoCDFieldManager, ArgoCDDefaultFieldManager)), + "both of its own managers in one second are still only Argo CD") + assert.Equal(t, int64(3), collisions(t, reader, collisionWithheld)) + + assert.False(t, loop.withholdLateApply(windowEvent("alpha", "3", ArgoCDFieldManager, "kubectl-edit")), + "a tie with another manager may be the edit itself") + assert.Equal(t, int64(1), collisions(t, reader, collisionAmbiguous)) +} diff --git a/internal/git/refusal_reapply_test.go b/internal/git/refusal_reapply_test.go index e4cc5f62..d8f0934b 100644 --- a/internal/git/refusal_reapply_test.go +++ b/internal/git/refusal_reapply_test.go @@ -471,7 +471,7 @@ func TestRefusalReapply_TheTimerTracksTheEarliestDeadline(t *testing.T) { // testApplier is a GitTarget applier naming a Kustomization of the given name. func testApplier(name string) *configv1alpha3.GitTargetApplier { - return &configv1alpha3.GitTargetApplier{KustomizationRef: meta.NamespacedObjectReference{Name: name}} + return &configv1alpha3.GitTargetApplier{KustomizationRef: &meta.NamespacedObjectReference{Name: name}} } // recordingRequester records every apply request a worker sends, and reports each one at once with From e2cb5661b40de64e56eac2c2d98e76e664620458 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 07:16:17 +0000 Subject: [PATCH 31/36] feat(applier): ask an Argo CD Application to sync a commit the worker checked The Argo CD client sets the Application's operation to the sync automated sync would start (source, prune from syncPolicy.automated, syncOptions and retry from syncPolicy), with the revision pinned to the commit and initiatedBy {username: gitops-reverser, automated: true}. It is an update of the whole Application, as Argo CD's SetAppOperation makes it, guarded by the resourceVersion read, and it clears status.operationState. An operation already running is waited out (up to the apply window) and never replaced; an update that lost a race reads again and decides again. Two places differ from what was planned, each read from Argo CD's source: a failed operation awaiting its retry is Running with a finishedAt, so a sync counts as finished only once its phase is Succeeded, Failed or Error; and a retry with refresh: true drops the pinned revision, so the copied retry omits it. The request's token travels in operation.info. The dispatcher picks Flux or Argo CD by the applier's kind behind one Client (Validate, Request, Handled), so an attempt and its wait for the applier are unchanged; apply_requests_total counts applier_kind="application". Co-Authored-By: Claude Opus 5.5 (1M context) --- internal/applier/applier.go | 291 +++++++++++++++ internal/applier/argocd.go | 396 ++++++++++++++++++++ internal/applier/argocd_test.go | 516 +++++++++++++++++++++++++++ internal/applier/dispatcher.go | 109 ++++-- internal/applier/dispatcher_test.go | 108 +++++- internal/applier/flux.go | 214 +---------- internal/applier/flux_test.go | 1 + internal/telemetry/apply_requests.go | 12 +- 8 files changed, 1391 insertions(+), 256 deletions(-) create mode 100644 internal/applier/applier.go create mode 100644 internal/applier/argocd.go create mode 100644 internal/applier/argocd_test.go diff --git a/internal/applier/applier.go b/internal/applier/applier.go new file mode 100644 index 00000000..942d7ef2 --- /dev/null +++ b/internal/applier/applier.go @@ -0,0 +1,291 @@ +// SPDX-License-Identifier: Apache-2.0 + +/* +Package applier delivers apply requests: what GitOps Reverser sends the object that applies a +GitTarget's folder back to its source cluster (its applier) when the branch worker has decided a +commit is safe to apply. The worker decides WHEN; this package decides HOW, off the worker's queue, +so a slow or unreachable applier never holds a branch. + +Two kinds are served, behind one Client: a Flux Kustomization whose sourceRef is a GitRepository +(spec.applier.kustomizationRef, flux.go), and an Argo CD Application with automated sync off +(spec.applier.applicationRef, argocd.go). See docs/design/apply-trigger.md. + +A request asks and never reports. Nothing here claims an apply happened; the applier's own status +(lastAppliedRevision on a Kustomization, status.operationState on an Application) is the record of +that. +*/ +package applier + +import ( + "context" + "errors" + "fmt" + "time" + + fluxmeta "github.com/fluxcd/pkg/apis/meta" + authorizationv1 "k8s.io/api/authorization/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + apimeta "k8s.io/apimachinery/pkg/api/meta" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" +) + +// Client asks one kind of applier, in one cluster, to apply. The dispatcher and the +// ApplierConfigured check pick one by the applier's kind (NewClient), and use nothing else of it. +type Client interface { + // Validate answers ApplierConfigured: the applier exists, applies want, and may be asked. + Validate(ctx context.Context, ref types.NamespacedName, want Binding) Verdict + // Request asks the applier to apply commit, after checking it still applies want and asking + // current just before it writes. A permanent error is a configuration no retry can fix; + // errStale is a decision that no longer held. + Request( + ctx context.Context, + ref types.NamespacedName, + want Binding, + commit string, + reason v1alpha3.ApplyReason, + current func() bool, + ) (Stamped, error) + // Handled reports whether the applier has finished the apply a Request asked for. + Handled(ctx context.Context, ref types.NamespacedName, s Stamped) (bool, error) +} + +var ( + _ Client = (*Flux)(nil) + _ Client = (*ArgoCD)(nil) +) + +// NewClient returns the Client for an applier of kind (v1alpha3.ApplierKind*) over the dynamic +// client of the cluster it lives in. +func NewClient(kind string, client dynamic.Interface) Client { + if kind == v1alpha3.ApplierKindApplication { + return NewArgoCD(client) + } + return NewFlux(client) +} + +// selfSubjectAccessReviewGVR is the access check Validate creates. +func selfSubjectAccessReviewGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{ + Group: "authorization.k8s.io", Version: "v1", Resource: "selfsubjectaccessreviews", + } +} + +// Outcome is what one request did. +type Outcome string + +// OutcomeCurrent asked for nothing: the applier needed no request (see each client's Request). +const OutcomeCurrent Outcome = "Current" + +// ApplierConfigured reasons (docs/configuration.md, "Who applies this folder"). Succeeded and +// Failed are the generic reasons, aliased from Flux's meta package as the conditions guide asks. +const ( + // ReasonSucceeded is ApplierConfigured=True: the applier exists, applies this repository and + // branch, and may be asked to apply. + ReasonSucceeded = fluxmeta.SucceededReason + // ReasonApplierNotFound is no such object in the source cluster, or its kind is not served there. + ReasonApplierNotFound = "ApplierNotFound" + // ReasonSourceMismatch is an applier that applies a different repository or branch. + ReasonSourceMismatch = "SourceMismatch" + // ReasonAutomatedSyncEnabled is an Argo CD Application with automated sync on, which syncs on + // its own and so is never asked to. + ReasonAutomatedSyncEnabled = "AutomatedSyncEnabled" + // ReasonForbidden is a source-cluster identity that may not read or write the applier. + ReasonForbidden = "Forbidden" + // ReasonCheckFailed is a check that could not complete: the source cluster did not answer. + // Verdict.Err carries the cause; it says nothing about the applier itself. + ReasonCheckFailed = fluxmeta.FailedReason +) + +// Verdict is the answer Validate gives for the ApplierConfigured condition. +type Verdict struct { + // OK is ApplierConfigured=True. + OK bool + // Reason is the condition reason. + Reason string + // Message names the object and what is wrong with it. + Message string + // Err is set only with ReasonCheckFailed: the read that failed. A caller may prefer to keep the + // verdict it already published over replacing a known answer with "could not tell". + Err error +} + +// Stamped is what one Request did, with what the applier reported before it, so Handled can tell +// when it has finished the apply it was asked for. +type Stamped struct { + // Outcome is what was asked, if anything. + Outcome Outcome + // Token identifies the request: the reconcile.fluxcd.io/requestedAt value stamped on a Flux + // object, or the request entry in an Argo CD operation's info. Empty when nothing was asked. + Token string + // Source is the GitRepository the Kustomization applies from (Flux only). + Source types.NamespacedName + // PriorAttemptedRevision is the Kustomization's status.lastAttemptedRevision before the stamp + // (Flux only). + PriorAttemptedRevision string + // PriorArtifactRevision is the GitRepository's status.artifact.revision before the stamp + // (Flux only). + PriorArtifactRevision string +} + +// Binding is the repository and branch an applier must apply from: the GitProvider's spec.url and +// the GitTarget's spec.branch. Request checks it before it writes anything, not only Validate, +// because a published ApplierConfigured verdict can be older than the applier it describes. +type Binding struct { + RepositoryURL string + Branch string +} + +// AccessCheck reports whether the source-cluster identity may perform attrs. +type AccessCheck func(ctx context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) + +// permanentError marks a request no retry can make succeed: the applier is configured in a way a +// request cannot work with, or applies another repository or branch. The dispatcher hands it back +// to the branch worker at once rather than retrying it, and the worker asks again later, because +// the configuration may have been fixed in between. +type permanentError struct{ err error } + +func (e permanentError) Error() string { return e.err.Error() } +func (e permanentError) Unwrap() error { return e.err } + +// errStale reports a request whose decision no longer held when the write was due. +var errStale = errors.New("the decision no longer holds: the branch moved on or became busy") + +// isPermanent reports whether err is a permanentError. +func isPermanent(err error) bool { + var p permanentError + return errors.As(err, &p) +} + +// statusString reads a string under an object's status. +func statusString(obj *unstructured.Unstructured, fields ...string) string { + value, _, _ := unstructured.NestedString(obj.Object, append([]string{"status"}, fields...)...) + return value +} + +// requestToken is the token a request carries: the time it was made, to the nanosecond, as +// `flux reconcile` stamps it. +func requestToken(now time.Time) string { + return now.Format(time.RFC3339Nano) +} + +// readFailure classifies a failed GET for Validate. product names what installs the kind, for the +// message ("Flux", "Argo CD"). +// +// A kind that is not served is ApplierNotFound like a missing object, because the fix is the same +// (install the product, or name the right object) and the condition is a gate either way. The two +// are told apart only for the message: a client with a RESTMapper says so with a NoKindMatch, and +// the API server answers the dynamic client's GET of an unserved resource with a 404 whose details +// name no object, where a missing object's 404 names it. +func readFailure( + product, kind string, gvr schema.GroupVersionResource, ref types.NamespacedName, err error, +) Verdict { + switch { + case apimeta.IsNoMatchError(err) || (apierrors.IsNotFound(err) && !namesObject(err, ref.Name)): + return Verdict{ + Reason: ReasonApplierNotFound, + Message: fmt.Sprintf("the source cluster does not serve %s (%s); is %s installed there?", + kind, gvr.GroupVersion(), product), + } + case apierrors.IsNotFound(err): + return Verdict{ + Reason: ReasonApplierNotFound, + Message: fmt.Sprintf("%s %s not found in the source cluster", kind, ref), + } + case apierrors.IsForbidden(err): + return Verdict{ + Reason: ReasonForbidden, + Message: fmt.Sprintf("the source-cluster identity may not get %s %s: %v", kind, ref, err), + } + default: + return Verdict{ + Reason: ReasonCheckFailed, + Message: fmt.Sprintf("could not read %s %s: %v", kind, ref, err), + Err: err, + } + } +} + +// namesObject reports whether a NotFound error is about the named object, rather than the resource. +func namesObject(err error, name string) bool { + var status apierrors.APIStatus + if !errors.As(err, &status) { + return false + } + details := status.Status().Details + return details != nil && details.Name == name +} + +// accessVerdict runs one access check for Validate and returns the verdict it forces, or nil when +// the identity may perform it. +func accessVerdict( + ctx context.Context, canI AccessCheck, kind string, gvr schema.GroupVersionResource, + ref types.NamespacedName, verb string, +) *Verdict { + allowed, err := canI(ctx, authorizationv1.ResourceAttributes{ + Namespace: ref.Namespace, + Verb: verb, + Group: gvr.Group, + Version: gvr.Version, + Resource: gvr.Resource, + Name: ref.Name, + }) + if err != nil { + return &Verdict{ + Reason: ReasonCheckFailed, + Message: fmt.Sprintf("could not check whether %s %s may be %s: %v", kind, ref, pastTense(verb), err), + Err: err, + } + } + if !allowed { + return &Verdict{ + Reason: ReasonForbidden, + Message: fmt.Sprintf("the source-cluster identity may not %s %s %s (%s)", + verb, kind, ref, gvr.GroupResource()), + } + } + return nil +} + +// pastTense is a verb as a message names a check of it. +func pastTense(verb string) string { + if verb == "update" { + return "updated" + } + return verb + "ed" +} + +// SelfSubjectAccessCheck answers an AccessCheck with a SelfSubjectAccessReview created through +// client, so it is answered for the identity client authenticates as. It goes through the dynamic +// client rather than a typed clientset because that client is the only one a source cluster has. +func SelfSubjectAccessCheck(client dynamic.Interface) AccessCheck { + return func(ctx context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) { + review := &authorizationv1.SelfSubjectAccessReview{ + TypeMeta: metav1.TypeMeta{ + APIVersion: authorizationv1.SchemeGroupVersion.String(), + Kind: "SelfSubjectAccessReview", + }, + Spec: authorizationv1.SelfSubjectAccessReviewSpec{ResourceAttributes: &attrs}, + } + content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(review) + if err != nil { + return false, err + } + created, err := client.Resource(selfSubjectAccessReviewGVR()). + Create(ctx, &unstructured.Unstructured{Object: content}, metav1.CreateOptions{}) + if err != nil { + return false, fmt.Errorf("create SelfSubjectAccessReview: %w", err) + } + allowed, _, err := unstructured.NestedBool(created.Object, "status", "allowed") + if err != nil { + return false, fmt.Errorf("read SelfSubjectAccessReview status: %w", err) + } + return allowed, nil + } +} diff --git a/internal/applier/argocd.go b/internal/applier/argocd.go new file mode 100644 index 00000000..92eaa3bc --- /dev/null +++ b/internal/applier/argocd.go @@ -0,0 +1,396 @@ +// SPDX-License-Identifier: Apache-2.0 + +package applier + +import ( + "context" + "fmt" + "strings" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" + "k8s.io/client-go/dynamic" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" + "github.com/ConfigButler/gitops-reverser/internal/repourl" +) + +// The Argo CD half of the applier. Its standard is full control: the Application has automated +// sync off, so its refresh only compares, and every sync it runs is one GitOps Reverser started, +// pinned to a commit the branch worker checked. Argo CD is read through the dynamic client, as +// Flux is, so a cluster without it installed costs one failed GET and this module does not import +// Argo CD's API. +// +// Every rule below is Argo CD's own, read from external-sources/argo-cd (3.5.0): +// +// - Automated sync is on when spec.syncPolicy.automated is present and its enabled is unset or +// true (SyncPolicy.IsAutomatedSyncEnabled in pkg/apis/application/v1alpha1/types.go). +// - A sync is started by setting the Application's operation, and never while one is set +// (SetAppOperation in util/argo/argo.go). It is an update of the whole object, which carries +// its status (the CRD has no status subresource), and clears status.operationState, so the +// controller starts the new operation rather than resume the last one. +// - The controller clears operation when the operation completes (Succeeded, Failed or Error), +// and leaves the result in status.operationState (setOperationState in +// controller/appcontroller.go). A failed operation that will be retried is Running again, with +// a finishedAt: it has not completed. +// - Automated sync builds its operation from the Application (autoSync, ibid.): source, prune +// from syncPolicy.automated, syncOptions and retry from syncPolicy, retry {limit: 5} when none +// is set, initiatedBy {automated: true}. + +// applicationGVR is Argo CD's Application. +func applicationGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: "argoproj.io", Version: "v1alpha1", Resource: "applications"} +} + +const kindApplication = "Application" + +// OutcomeSyncRequested set the Application's operation to a sync of the commit. +const OutcomeSyncRequested Outcome = "SyncRequested" + +// ArgoCDInitiator is the username every sync GitOps Reverser starts carries in +// operation.initiatedBy, next to automated: true. It is what Argo CD's UI and history show as the +// sync's initiator. +const ArgoCDInitiator = "gitops-reverser" + +// Info entries a sync carries in operation.info, which Argo CD shows with the operation. +const ( + // argoInfoRequest is the request's token, which tells this request's operation state from any + // other, including an earlier sync of the same commit. + argoInfoRequest = "gitops-reverser request" + // argoInfoReason is why GitOps Reverser asked. + argoInfoReason = "gitops-reverser reason" +) + +// defaultArgoRetryLimit is the retry an automated sync gets when the Application sets none. +const defaultArgoRetryLimit = 5 + +// maxArgoUpdateConflicts bounds how often one request re-reads the Application after its update +// lost a race. The controller writes an Application's status on every refresh, and with no status +// subresource each of those writes conflicts with an update made from an older read. +const maxArgoUpdateConflicts = 5 + +// Operation phases, from gitops-engine's pkg/sync/common. +const ( + argoPhaseRunning = "Running" + argoPhaseTerminating = "Terminating" +) + +// ArgoCD asks an Argo CD Application to sync, in one cluster. +type ArgoCD struct { + client dynamic.Interface + + // Now is the clock a request's token is taken from. + Now func() time.Time + // CanI is the access check Validate runs for update. NewArgoCD sets it to a + // SelfSubjectAccessReview through the same client. + CanI AccessCheck + // OperationWait bounds how long one request waits for an operation that is running to end + // before it sets its own; zero does not wait, and the request fails at once. + OperationWait time.Duration + // PollInterval is how often that wait reads the Application. + PollInterval time.Duration +} + +// NewArgoCD returns an Argo CD client over a cluster's dynamic client. +func NewArgoCD(client dynamic.Interface) *ArgoCD { + return &ArgoCD{ + client: client, + Now: time.Now, + CanI: SelfSubjectAccessCheck(client), + PollInterval: DefaultHandledPollInterval, + } +} + +// Request asks the Application ref to sync commit, the way Argo CD's automated sync would, with +// the revision pinned to commit and initiatedBy {username: gitops-reverser, automated: true}. +// +// It first checks that the Application applies want and has automated sync off, and refuses with a +// permanent error when not: a sync of another branch would deploy that branch, and an Application +// that syncs on its own is not under GitOps Reverser's control. A sync somebody else started is +// never replaced: Request waits for it to end (OperationWait), and its update is guarded by the +// resourceVersion it read, so an operation set in between makes the update fail rather than be +// overwritten; it then reads again and waits for that one too. +// +// current is asked just before the update. When it reports false the decision no longer holds and +// nothing is set: errStale. +// +// An empty commit (the remote does not carry the branch) has nothing to pin, and asks nothing +// (OutcomeCurrent): Argo CD reports the missing branch itself on its next refresh, and the branch +// coming back is an observation that owes a new request. +func (a *ArgoCD) Request( + ctx context.Context, + ref types.NamespacedName, + want Binding, + commit string, + reason v1alpha3.ApplyReason, + current func() bool, +) (Stamped, error) { + var waitUntil time.Time + for conflicts := 0; ; { + app, err := a.readControlled(ctx, ref, want) + if err != nil { + return Stamped{}, err + } + if commit == "" { + return Stamped{Outcome: OutcomeCurrent}, nil + } + if operationInProgress(app) { + if waitUntil.IsZero() { + waitUntil = a.Now().Add(a.OperationWait) + } + if err := a.awaitOperation(ctx, ref, waitUntil); err != nil { + return Stamped{}, err + } + continue + } + if current != nil && !current() { + return Stamped{}, errStale + } + stamped, err := a.setOperation(ctx, ref, app, commit, reason) + if apierrors.IsConflict(err) && conflicts < maxArgoUpdateConflicts { + // Somebody wrote the Application since it was read: the controller's status, or an + // operation of its own. Read it again, and decide again from what it is now. + conflicts++ + continue + } + return stamped, err + } +} + +// readControlled reads the Application, and refuses with a permanent error one that does not apply +// want, or that has automated sync on. +func (a *ArgoCD) readControlled( + ctx context.Context, ref types.NamespacedName, want Binding, +) (*unstructured.Unstructured, error) { + app, err := a.client.Resource(applicationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + if err != nil { + return nil, fmt.Errorf("get %s %s: %w", kindApplication, ref, err) + } + if msg := applicationMismatch(app, ref, want); msg != "" { + return nil, permanentError{fmt.Errorf("%s %s: %s", kindApplication, ref, msg)} + } + if automatedSyncEnabled(app) { + return nil, permanentError{fmt.Errorf("%s %s: %s", kindApplication, ref, automatedSyncMessage)} + } + return app, nil +} + +// setOperation updates app, as read, with the sync operation and no operation state. The update +// carries the resourceVersion read, so it fails with a conflict when anything wrote the +// Application since. +func (a *ArgoCD) setOperation( + ctx context.Context, ref types.NamespacedName, app *unstructured.Unstructured, commit string, + reason v1alpha3.ApplyReason, +) (Stamped, error) { + token := requestToken(a.Now()) + update := app.DeepCopy() + if err := unstructured.SetNestedField(update.Object, syncOperation(app, commit, reason, token), + "operation"); err != nil { + return Stamped{}, fmt.Errorf("build the sync operation for %s %s: %w", kindApplication, ref, err) + } + unstructured.RemoveNestedField(update.Object, "status", "operationState") + _, err := a.client.Resource(applicationGVR()).Namespace(ref.Namespace). + Update(ctx, update, metav1.UpdateOptions{}) + if err != nil { + return Stamped{}, fmt.Errorf("set the sync operation on %s %s: %w", kindApplication, ref, err) + } + return Stamped{Outcome: OutcomeSyncRequested, Token: token}, nil +} + +// awaitOperation waits until the Application has no operation in progress, or fails once +// waitUntil passes. It decides nothing: Request reads the Application again after it. +func (a *ArgoCD) awaitOperation(ctx context.Context, ref types.NamespacedName, waitUntil time.Time) error { + for { + if !a.Now().Before(waitUntil) { + return fmt.Errorf("%s %s has an operation in progress; asking again once the branch worker decides to", + kindApplication, ref) + } + timer := time.NewTimer(min(a.PollInterval, waitUntil.Sub(a.Now()))) + select { + case <-ctx.Done(): + timer.Stop() + return fmt.Errorf("wait for the operation in progress on %s %s: %w", kindApplication, ref, ctx.Err()) + case <-timer.C: + } + app, err := a.client.Resource(applicationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + if err != nil { + return fmt.Errorf("get %s %s: %w", kindApplication, ref, err) + } + if !operationInProgress(app) { + return nil + } + } +} + +// Handled reports whether Argo CD has finished the sync s asked for, which is when the apply +// window closes. The operation state is this request's when its info carries the request's token; +// it has finished once its phase is Succeeded, Failed or Error. A failed sync closes the window +// too: Argo CD's own retries of it are part of the operation, which stays Running until they end. +// An operation that is gone before it ever ran (somebody cleared it) runs nothing, and is handled. +func (a *ArgoCD) Handled(ctx context.Context, ref types.NamespacedName, s Stamped) (bool, error) { + if s.Outcome != OutcomeSyncRequested { + return true, nil + } + app, err := a.client.Resource(applicationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + if err != nil { + return false, fmt.Errorf("get %s %s: %w", kindApplication, ref, err) + } + if state, found, _ := unstructured.NestedMap(app.Object, "status", "operationState"); found && + operationToken(state["operation"]) == s.Token { + phase, _, _ := unstructured.NestedString(state, "phase") + return phase != "" && phase != argoPhaseRunning && phase != argoPhaseTerminating, nil + } + operation, found, _ := unstructured.NestedMap(app.Object, "operation") + return !found || operationToken(operation) != s.Token, nil +} + +// Validate answers ApplierConfigured for an Application: that it exists in the cluster and its +// kind is served there, that it applies want from a single source, that automated sync is off, +// and that this identity may update it, since a request is an update. +func (a *ArgoCD) Validate(ctx context.Context, ref types.NamespacedName, want Binding) Verdict { + app, err := a.client.Resource(applicationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) + if err != nil { + return readFailure("Argo CD", kindApplication, applicationGVR(), ref, err) + } + if msg := applicationMismatch(app, ref, want); msg != "" { + return Verdict{Reason: ReasonSourceMismatch, Message: msg} + } + if automatedSyncEnabled(app) { + return Verdict{ + Reason: ReasonAutomatedSyncEnabled, + Message: fmt.Sprintf("%s %s %s", kindApplication, ref, automatedSyncMessage), + } + } + if verdict := accessVerdict(ctx, a.CanI, kindApplication, applicationGVR(), ref, "update"); verdict != nil { + return *verdict + } + return Verdict{ + OK: true, + Reason: ReasonSucceeded, + Message: fmt.Sprintf("%s %s syncs branch %s, with automated sync off", + kindApplication, ref, want.Branch), + } +} + +// automatedSyncMessage is why an Application with automated sync on is neither valid nor synced. +const automatedSyncMessage = "has automated sync on, so it syncs every new commit on its own, " + + "without the check that nothing live is missing from it; set spec.syncPolicy.automated.enabled " + + "to false (or remove spec.syncPolicy.automated) and GitOps Reverser starts every sync" + +// automatedSyncEnabled is Argo CD's IsAutomatedSyncEnabled: spec.syncPolicy.automated is present, +// and its enabled is unset or true. +func automatedSyncEnabled(app *unstructured.Unstructured) bool { + automated, found, err := unstructured.NestedMap(app.Object, "spec", "syncPolicy", "automated") + if err != nil || !found { + return false + } + enabled, found, err := unstructured.NestedBool(automated, "enabled") + return err != nil || !found || enabled +} + +// operationInProgress reports whether the Application has an operation set, or one still running: +// Running (a retry pending included) or Terminating. Either one is not to be replaced. +func operationInProgress(app *unstructured.Unstructured) bool { + if operation, found, _ := unstructured.NestedFieldNoCopy(app.Object, "operation"); found && operation != nil { + return true + } + phase := statusString(app, "operationState", "phase") + return phase == argoPhaseRunning || phase == argoPhaseTerminating +} + +// operationToken is the request token an operation's info carries, or "". +func operationToken(operation any) string { + op, ok := operation.(map[string]any) + if !ok { + return "" + } + info, _, _ := unstructured.NestedSlice(op, "info") + for _, item := range info { + entry, ok := item.(map[string]any) + if ok && entry["name"] == argoInfoRequest { + value, _ := entry["value"].(string) + return value + } + } + return "" +} + +// syncOperation is the operation Argo CD's automated sync would set, with the revision pinned to +// commit and GitOps Reverser as the initiator. It carries the request's token and reason in info. +// +// One difference is deliberate: a retry never drops the pin. An Application whose +// syncPolicy.retry sets refresh: true has each retry sync the newest revision instead of the one +// asked for, which would apply a commit the branch worker never checked; the retry is copied +// without it. +func syncOperation( + app *unstructured.Unstructured, commit string, reason v1alpha3.ApplyReason, token string, +) map[string]any { + sync := map[string]any{ + "revision": commit, + "prune": automatedPrune(app), + } + if source, found, _ := unstructured.NestedMap(app.Object, "spec", "source"); found { + sync["source"] = source + } + if options, found, _ := unstructured.NestedSlice(app.Object, "spec", "syncPolicy", "syncOptions"); found && + len(options) > 0 { + sync["syncOptions"] = options + } + + retry := map[string]any{"limit": int64(defaultArgoRetryLimit)} + if configured, found, _ := unstructured.NestedMap(app.Object, "spec", "syncPolicy", "retry"); found { + retry = configured + delete(retry, "refresh") + } + + return map[string]any{ + "sync": sync, + "initiatedBy": map[string]any{"username": ArgoCDInitiator, "automated": true}, + "info": []any{ + map[string]any{"name": argoInfoRequest, "value": token}, + map[string]any{"name": argoInfoReason, "value": string(reason)}, + }, + "retry": retry, + } +} + +// automatedPrune is spec.syncPolicy.automated.prune: the off switch automated.enabled: false keeps +// it, and without the block a sync does not prune, as automated sync's GetPrune defaults it. +func automatedPrune(app *unstructured.Unstructured) bool { + prune, _, _ := unstructured.NestedBool(app.Object, "spec", "syncPolicy", "automated", "prune") + return prune +} + +// applicationMismatch returns why an Application does not sync branch of repositoryURL from a +// single source, or "" when it does. targetRevision must name the branch, as its name or as +// refs/heads/: HEAD, a tag or a commit would never follow the branch GitOps Reverser writes. +func applicationMismatch(app *unstructured.Unstructured, ref types.NamespacedName, want Binding) string { + if sources, _, _ := unstructured.NestedSlice(app.Object, "spec", "sources"); len(sources) > 0 { + return fmt.Sprintf("%s %s has spec.sources; GitOps Reverser syncs only a single-source Application", + kindApplication, ref) + } + if _, found, _ := unstructured.NestedFieldNoCopy(app.Object, "spec", "sourceHydrator"); found { + return fmt.Sprintf("%s %s has spec.sourceHydrator; GitOps Reverser syncs only an Application "+ + "whose spec.source it writes", kindApplication, ref) + } + source, found, _ := unstructured.NestedMap(app.Object, "spec", "source") + if !found { + return fmt.Sprintf("%s %s names no spec.source", kindApplication, ref) + } + url, _, _ := unstructured.NestedString(source, "repoURL") + if !repourl.Same(url, want.RepositoryURL) { + return fmt.Sprintf("%s %s syncs %s, not %s", kindApplication, ref, url, want.RepositoryURL) + } + revision, _, _ := unstructured.NestedString(source, "targetRevision") + if strings.TrimPrefix(revision, "refs/heads/") != want.Branch { + if revision == "" { + revision = "HEAD" + } + return fmt.Sprintf("%s %s targets revision %q, not branch %s", kindApplication, ref, revision, want.Branch) + } + return "" +} diff --git a/internal/applier/argocd_test.go b/internal/applier/argocd_test.go new file mode 100644 index 00000000..ca806346 --- /dev/null +++ b/internal/applier/argocd_test.go @@ -0,0 +1,516 @@ +// SPDX-License-Identifier: Apache-2.0 + +package applier + +import ( + "context" + "errors" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + authorizationv1 "k8s.io/api/authorization/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/types" + dynamicfake "k8s.io/client-go/dynamic/fake" + k8stesting "k8s.io/client-go/testing" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" +) + +const argoNS = "argocd" + +var platformApp = types.NamespacedName{Namespace: argoNS, Name: "platform"} + +// applicationObj builds an Argo CD Application on a single source. A nil syncPolicy omits it. +func applicationObj(repoURL, targetRevision string, syncPolicy map[string]any) *unstructured.Unstructured { + obj := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "argoproj.io/v1alpha1", + "kind": kindApplication, + "metadata": map[string]any{"namespace": argoNS, "name": "platform", "resourceVersion": "1"}, + "spec": map[string]any{ + "project": "default", + "source": map[string]any{ + "repoURL": repoURL, "targetRevision": targetRevision, "path": "apps", + "directory": map[string]any{"recurse": true}, + }, + "destination": map[string]any{"server": "https://kubernetes.default.svc", "namespace": "apps"}, + }, + }} + if syncPolicy != nil { + _ = unstructured.SetNestedMap(obj.Object, syncPolicy, "spec", "syncPolicy") + } + return obj +} + +// manualApp is the standard: an Application on testRepoURL and testBranch with no automated sync. +func manualApp() *unstructured.Unstructured { return applicationObj(testRepoURL, testBranch, nil) } + +func newArgoSource(objs ...runtime.Object) *dynamicfake.FakeDynamicClient { + return newSourceClient(objs...) +} + +func pinnedArgo(client *dynamicfake.FakeDynamicClient) *ArgoCD { + argo := NewArgoCD(client) + argo.Now = func() time.Time { return stampTime } + argo.CanI = allowAll + return argo +} + +func readApp(t *testing.T, client *dynamicfake.FakeDynamicClient) *unstructured.Unstructured { + t.Helper() + app, err := client.Resource(applicationGVR()).Namespace(argoNS).Get(context.Background(), "platform", + metav1.GetOptions{}) + require.NoError(t, err) + return app +} + +func writeApp(t *testing.T, client *dynamicfake.FakeDynamicClient, app *unstructured.Unstructured) { + t.Helper() + _, err := client.Resource(applicationGVR()).Namespace(argoNS).Update(context.Background(), app, + metav1.UpdateOptions{}) + require.NoError(t, err) +} + +// theirOperation is a sync somebody else started. +func theirOperation() map[string]any { + return map[string]any{ + "sync": map[string]any{"revision": olderCommit}, + "initiatedBy": map[string]any{"username": "alice"}, + } +} + +func updates(client *dynamicfake.FakeDynamicClient) int { + n := 0 + for _, action := range client.Actions() { + if action.GetVerb() == "update" { + n++ + } + } + return n +} + +// TestArgoCDRequest_TheOperationIsWhatAutomatedSyncWouldBuild: the operation Argo CD's autoSync +// builds (controller/appcontroller.go) from the Application, but for the pinned revision, the +// initiator, the info carrying the request, and a retry that keeps the pin. +func TestArgoCDRequest_TheOperationIsWhatAutomatedSyncWouldBuild(t *testing.T) { + source := map[string]any{ + "repoURL": testRepoURL, "targetRevision": testBranch, "path": "apps", + "directory": map[string]any{"recurse": true}, + } + tests := []struct { + name string + syncPolicy map[string]any + wantSync map[string]any + wantRetry map[string]any + }{ + { + name: "no syncPolicy: no prune, no options, the default retry", + wantSync: map[string]any{"revision": testCommit, "prune": false, "source": source}, + wantRetry: map[string]any{"limit": int64(defaultArgoRetryLimit)}, + }, + { + name: "automated switched off keeps its prune, and syncOptions and retry come along", + syncPolicy: map[string]any{ + "automated": map[string]any{"enabled": false, "prune": true, "selfHeal": true}, + "syncOptions": []any{"ServerSideApply=true", "CreateNamespace=true"}, + "retry": map[string]any{ + "limit": int64(2), "backoff": map[string]any{"duration": "5s", "factor": int64(2)}, + }, + }, + wantSync: map[string]any{ + "revision": testCommit, "prune": true, "source": source, + "syncOptions": []any{"ServerSideApply=true", "CreateNamespace=true"}, + }, + wantRetry: map[string]any{ + "limit": int64(2), "backoff": map[string]any{"duration": "5s", "factor": int64(2)}, + }, + }, + { + name: "a retry that would refresh to the newest revision keeps the pin", + syncPolicy: map[string]any{ + "retry": map[string]any{"limit": int64(3), "refresh": true}, + }, + wantSync: map[string]any{"revision": testCommit, "prune": false, "source": source}, + wantRetry: map[string]any{"limit": int64(3)}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := newArgoSource(applicationObj(testRepoURL, testBranch, tt.syncPolicy)) + + stamped, err := pinnedArgo(client).Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonForeignCommit, nil) + + require.NoError(t, err) + assert.Equal(t, Stamped{Outcome: OutcomeSyncRequested, Token: requestToken(stampTime)}, stamped) + operation, found, _ := unstructured.NestedMap(readApp(t, client).Object, "operation") + require.True(t, found) + assert.Equal(t, map[string]any{ + "sync": tt.wantSync, + "initiatedBy": map[string]any{"username": ArgoCDInitiator, "automated": true}, + "info": []any{ + map[string]any{"name": argoInfoRequest, "value": requestToken(stampTime)}, + map[string]any{"name": argoInfoReason, "value": "ForeignCommit"}, + }, + "retry": tt.wantRetry, + }, operation) + }) + } +} + +// TestArgoCDRequest_ClearsTheLastOperationState: as SetAppOperation does, so the controller starts +// the new operation instead of resuming the last. +func TestArgoCDRequest_ClearsTheLastOperationState(t *testing.T) { + app := manualApp() + _ = unstructured.SetNestedMap(app.Object, map[string]any{ + "phase": "Succeeded", "operation": theirOperation(), + }, "status", "operationState") + _ = unstructured.SetNestedField(app.Object, "Synced", "status", "sync", "status") + client := newArgoSource(app) + + _, err := pinnedArgo(client).Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonRefusal, nil) + + require.NoError(t, err) + written := readApp(t, client) + _, found, _ := unstructured.NestedFieldNoCopy(written.Object, "status", "operationState") + assert.False(t, found) + assert.Equal(t, "Synced", statusString(written, "sync", "status"), "the rest of the status is kept") +} + +// TestArgoCDRequest_ARunningOperationIsNeverReplaced: a sync somebody else started is waited out, +// and only then is GitOps Reverser's own set; without time to wait, the attempt fails and theirs +// stays. +func TestArgoCDRequest_ARunningOperationIsNeverReplaced(t *testing.T) { + running := func() *unstructured.Unstructured { + app := manualApp() + _ = unstructured.SetNestedMap(app.Object, theirOperation(), "operation") + _ = unstructured.SetNestedField(app.Object, "Running", "status", "operationState", "phase") + return app + } + + t.Run("with no time to wait, the attempt fails and writes nothing", func(t *testing.T) { + client := newArgoSource(running()) + + _, err := pinnedArgo(client).Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonForeignCommit, nil) + + require.ErrorContains(t, err, "has an operation in progress") + assert.False(t, isPermanent(err), "the operation ends; asking again later can succeed") + assert.Zero(t, updates(client)) + }) + + t.Run("an operation that ends in time is waited for, then ours is set", func(t *testing.T) { + client := newArgoSource(running()) + var gets atomic.Int32 + client.PrependReactor("get", "applications", func(k8stesting.Action) (bool, runtime.Object, error) { + if gets.Add(1) == 3 { + // Argo CD finishes their sync: operation cleared, the result left in operationState. + app := manualApp() + _ = unstructured.SetNestedField(app.Object, "Succeeded", "status", "operationState", "phase") + require.NoError(t, client.Tracker().Update(applicationGVR(), app, argoNS)) + } + return false, nil, nil + }) + argo := NewArgoCD(client) + argo.OperationWait, argo.PollInterval = 5*time.Second, time.Millisecond + + stamped, err := argo.Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonForeignCommit, nil) + + require.NoError(t, err) + assert.Equal(t, OutcomeSyncRequested, stamped.Outcome) + assert.Equal(t, stamped.Token, operationToken(readApp(t, client).Object["operation"])) + }) + + t.Run("an operation set between the read and the update makes the update fail, not replace it", + func(t *testing.T) { + client := newArgoSource(manualApp()) + client.PrependReactor("update", "applications", func(k8stesting.Action) (bool, runtime.Object, error) { + // Somebody's sync lands after the read: the guarded update conflicts with it. + app := manualApp() + app.SetResourceVersion("2") + _ = unstructured.SetNestedMap(app.Object, theirOperation(), "operation") + require.NoError(t, client.Tracker().Update(applicationGVR(), app, argoNS)) + return true, nil, apierrors.NewConflict(applicationGVR().GroupResource(), "platform", + errors.New("the object has been modified")) + }) + + _, err := pinnedArgo(client).Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonForeignCommit, nil) + + require.ErrorContains(t, err, "has an operation in progress") + assert.Equal(t, theirOperation(), readApp(t, client).Object["operation"], "theirs is untouched") + }) + + t.Run("a retry pending on a failed operation still counts as running", func(t *testing.T) { + app := manualApp() + _ = unstructured.SetNestedMap(app.Object, map[string]any{ + "phase": "Running", "finishedAt": "2026-10-20T10:14:00Z", "operation": theirOperation(), + }, "status", "operationState") + client := newArgoSource(app) + + _, err := pinnedArgo(client).Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonForeignCommit, nil) + + require.ErrorContains(t, err, "has an operation in progress") + assert.Zero(t, updates(client)) + }) +} + +// TestArgoCDRequest_AConflictWithAStatusWriteIsReadAgain: the controller writes the Application's +// status on every refresh, and with no status subresource each write moves its resourceVersion. A +// conflict that brought no operation is decided again from a fresh read. +func TestArgoCDRequest_AConflictWithAStatusWriteIsReadAgain(t *testing.T) { + client := newArgoSource(manualApp()) + var conflicted atomic.Bool + client.PrependReactor("update", "applications", func(k8stesting.Action) (bool, runtime.Object, error) { + if conflicted.CompareAndSwap(false, true) { + return true, nil, apierrors.NewConflict(applicationGVR().GroupResource(), "platform", + errors.New("the object has been modified")) + } + return false, nil, nil + }) + + stamped, err := pinnedArgo(client).Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonForeignCommit, nil) + + require.NoError(t, err) + assert.Equal(t, OutcomeSyncRequested, stamped.Outcome) + assert.Equal(t, 2, updates(client)) +} + +// TestArgoCDRequest_RefusesWhatIsNotUnderControl: a published verdict can be older than the +// Application, so a request checks the source and automated sync again, and writes nothing on +// either. A stale decision writes nothing either, and an empty commit has nothing to pin. +func TestArgoCDRequest_RefusesWhatIsNotUnderControl(t *testing.T) { + tests := []struct { + name string + app *unstructured.Unstructured + commit string + current func() bool + wantErr string + wantPerm bool + wantStale bool + wantOutcome Outcome + }{ + {name: "another branch", app: applicationObj(testRepoURL, "release", nil), commit: testCommit, + wantErr: `targets revision "release", not branch main`, wantPerm: true}, + {name: "automated sync on", commit: testCommit, + app: applicationObj(testRepoURL, testBranch, map[string]any{"automated": map[string]any{}}), + wantErr: "has automated sync on", wantPerm: true}, + {name: "the decision went stale", app: manualApp(), commit: testCommit, + current: func() bool { return false }, wantStale: true}, + {name: "no commit to pin", app: manualApp(), wantOutcome: OutcomeCurrent}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + client := newArgoSource(tt.app) + + stamped, err := pinnedArgo(client).Request(context.Background(), platformApp, testBinding, tt.commit, + v1alpha3.ApplyReasonForeignCommit, tt.current) + + switch { + case tt.wantErr != "": + require.ErrorContains(t, err, tt.wantErr) + assert.Equal(t, tt.wantPerm, isPermanent(err)) + case tt.wantStale: + require.ErrorIs(t, err, errStale) + default: + require.NoError(t, err) + assert.Equal(t, tt.wantOutcome, stamped.Outcome) + } + assert.Zero(t, updates(client), "nothing is written") + }) + } +} + +// TestArgoCDValidate_Reasons covers every ApplierConfigured answer for an Application. +func TestArgoCDValidate_Reasons(t *testing.T) { + sources := map[string]any{"sources": []any{map[string]any{"repoURL": testRepoURL}}} + tests := []struct { + name string + objs []runtime.Object + mutate func(*unstructured.Unstructured) + reactor k8stesting.ReactionFunc + canI AccessCheck + wantReason string + wantMessage string + wantErr bool + }{ + {name: "configured", objs: []runtime.Object{manualApp()}, wantReason: ReasonSucceeded, + wantMessage: "Application argocd/platform syncs branch main, with automated sync off"}, + {name: "configured, as refs/heads/ names the branch", + objs: []runtime.Object{applicationObj(testRepoURL, "refs/heads/main", nil)}, wantReason: ReasonSucceeded}, + {name: "configured, with the off switch that keeps prune", + objs: []runtime.Object{applicationObj(testRepoURL, testBranch, map[string]any{ + "automated": map[string]any{"enabled": false, "prune": true}, + })}, wantReason: ReasonSucceeded}, + {name: "configured, over an equivalent URL", + objs: []runtime.Object{applicationObj(testRepoURL+".git", testBranch, nil)}, wantReason: ReasonSucceeded}, + {name: "missing", wantReason: ReasonApplierNotFound, + wantMessage: "Application argocd/platform not found"}, + { + name: "kind not served", + reactor: func(k8stesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewNotFound(applicationGVR().GroupResource(), "") + }, + wantReason: ReasonApplierNotFound, + wantMessage: "does not serve Application (argoproj.io/v1alpha1); is Argo CD installed there?", + }, + {name: "another repository", + objs: []runtime.Object{applicationObj("https://github.com/acme/other", testBranch, nil)}, + wantReason: ReasonSourceMismatch, wantMessage: "syncs https://github.com/acme/other, not"}, + {name: "another branch", objs: []runtime.Object{applicationObj(testRepoURL, "release", nil)}, + wantReason: ReasonSourceMismatch, wantMessage: `targets revision "release", not branch main`}, + {name: "HEAD", objs: []runtime.Object{applicationObj(testRepoURL, "HEAD", nil)}, + wantReason: ReasonSourceMismatch, wantMessage: `targets revision "HEAD"`}, + {name: "no targetRevision, which is HEAD", objs: []runtime.Object{applicationObj(testRepoURL, "", nil)}, + wantReason: ReasonSourceMismatch, wantMessage: `targets revision "HEAD"`}, + {name: "a tag", objs: []runtime.Object{applicationObj(testRepoURL, "v1.0.0", nil)}, + wantReason: ReasonSourceMismatch, wantMessage: `targets revision "v1.0.0"`}, + {name: "multi-source", objs: []runtime.Object{manualApp()}, + mutate: func(app *unstructured.Unstructured) { + unstructured.RemoveNestedField(app.Object, "spec", "source") + _ = unstructured.SetNestedField(app.Object, sources["sources"], "spec", "sources") + }, + wantReason: ReasonSourceMismatch, wantMessage: "syncs only a single-source Application"}, + {name: "a source hydrator", objs: []runtime.Object{manualApp()}, + mutate: func(app *unstructured.Unstructured) { + _ = unstructured.SetNestedMap(app.Object, map[string]any{}, "spec", "sourceHydrator") + }, + wantReason: ReasonSourceMismatch, wantMessage: "has spec.sourceHydrator"}, + { + name: "no source", + objs: []runtime.Object{manualApp()}, + mutate: func(app *unstructured.Unstructured) { unstructured.RemoveNestedField(app.Object, "spec", "source") }, + wantReason: ReasonSourceMismatch, + wantMessage: "names no spec.source", + }, + {name: "automated sync on, as an empty block turns it on", + objs: []runtime.Object{applicationObj(testRepoURL, testBranch, map[string]any{ + "automated": map[string]any{"prune": true, "selfHeal": false}, + })}, + wantReason: ReasonAutomatedSyncEnabled, wantMessage: "has automated sync on"}, + {name: "automated sync on, enabled explicitly", + objs: []runtime.Object{applicationObj(testRepoURL, testBranch, map[string]any{ + "automated": map[string]any{"enabled": true}, + })}, + wantReason: ReasonAutomatedSyncEnabled}, + {name: "may not update it", objs: []runtime.Object{manualApp()}, canI: denyOn("applications"), + wantReason: ReasonForbidden, + wantMessage: "may not update Application argocd/platform (applications.argoproj.io)"}, + { + name: "the access review fails", + objs: []runtime.Object{manualApp()}, + canI: func(context.Context, authorizationv1.ResourceAttributes) (bool, error) { + return false, errors.New("connection refused") + }, + wantReason: ReasonCheckFailed, + wantMessage: "could not check whether Application argocd/platform may be updated", + wantErr: true, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + for _, obj := range tt.objs { + if tt.mutate != nil { + tt.mutate(obj.(*unstructured.Unstructured)) + } + } + client := newArgoSource(tt.objs...) + if tt.reactor != nil { + client.PrependReactor("get", "applications", tt.reactor) + } + argo := pinnedArgo(client) + if tt.canI != nil { + argo.CanI = tt.canI + } + + verdict := argo.Validate(context.Background(), platformApp, testBinding) + + assert.Equal(t, tt.wantReason, verdict.Reason, verdict.Message) + assert.Equal(t, tt.wantReason == ReasonSucceeded, verdict.OK) + assert.Contains(t, verdict.Message, tt.wantMessage) + assert.Equal(t, tt.wantErr, verdict.Err != nil) + }) + } +} + +// TestArgoCDValidate_AsksForUpdate: a request is an update of the Application, so that is the +// access checked, on that object. +func TestArgoCDValidate_AsksForUpdate(t *testing.T) { + var asked []authorizationv1.ResourceAttributes + argo := pinnedArgo(newArgoSource(manualApp())) + argo.CanI = func(_ context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) { + asked = append(asked, attrs) + return true, nil + } + + require.True(t, argo.Validate(context.Background(), platformApp, testBinding).OK) + assert.Equal(t, []authorizationv1.ResourceAttributes{{ + Namespace: argoNS, Verb: "update", Group: "argoproj.io", Version: "v1alpha1", + Resource: "applications", Name: "platform", + }}, asked) +} + +// TestArgoCD_Handled pins when an Argo CD apply window closes: when this request's operation has +// completed, whatever its phase, and not while it runs or retries, nor on another operation's state. +func TestArgoCD_Handled(t *testing.T) { + client := newArgoSource(manualApp()) + argo := pinnedArgo(client) + stamped, err := argo.Request(context.Background(), platformApp, testBinding, testCommit, + v1alpha3.ApplyReasonForeignCommit, nil) + require.NoError(t, err) + ours, _, _ := unstructured.NestedMap(readApp(t, client).Object, "operation") + + setState := func(t *testing.T, operation map[string]any, phase string, keepOperation bool) { + t.Helper() + app := readApp(t, client) + _ = unstructured.SetNestedMap(app.Object, map[string]any{ + "operation": operation, "phase": phase, "finishedAt": "2026-10-20T10:15:00Z", + }, "status", "operationState") + if !keepOperation { + unstructured.RemoveNestedField(app.Object, "operation") + } + writeApp(t, client, app) + } + handled := func(t *testing.T) bool { + t.Helper() + ok, err := argo.Handled(context.Background(), platformApp, stamped) + require.NoError(t, err) + return ok + } + + assert.False(t, handled(t), "set, not started") + setState(t, theirOperation(), "Succeeded", true) + assert.False(t, handled(t), "another operation's state is not ours, and ours is still set") + setState(t, ours, "Running", true) + assert.False(t, handled(t), "running, or a failed sync waiting for its retry (Running with a finishedAt)") + setState(t, ours, "Failed", false) + assert.True(t, handled(t), "a failed sync closes the window") + setState(t, ours, "Succeeded", false) + assert.True(t, handled(t)) + setState(t, theirOperation(), "Succeeded", false) + assert.True(t, handled(t), "ours was cleared before it ran: nothing of it will apply") + + assert.True(t, func() bool { + ok, err := argo.Handled(context.Background(), platformApp, Stamped{Outcome: OutcomeCurrent}) + require.NoError(t, err) + return ok + }(), "nothing asked is nothing to wait for") +} + +// TestNewClient_PicksTheClientByKind: the dispatcher and the ApplierConfigured check use the kind +// spec.applier names. +func TestNewClient_PicksTheClientByKind(t *testing.T) { + client := newArgoSource() + assert.IsType(t, &ArgoCD{}, NewClient(v1alpha3.ApplierKindApplication, client)) + assert.IsType(t, &Flux{}, NewClient(v1alpha3.ApplierKindKustomization, client)) +} diff --git a/internal/applier/dispatcher.go b/internal/applier/dispatcher.go index a0d38abc..cac5f641 100644 --- a/internal/applier/dispatcher.go +++ b/internal/applier/dispatcher.go @@ -40,6 +40,7 @@ const ( // apply_requests_total label values. const ( applierKindKustomization = "kustomization" + applierKindApplication = "application" outcomeSent = "sent" outcomeFailed = "failed" @@ -61,13 +62,14 @@ type SourceClients func(ctx context.Context, clusterID string) (dynamic.Interfac // branch worker decides when to try again. const DefaultAttemptTimeout = 30 * time.Second -// DefaultApplyWindowTimeout bounds how long a delivered attempt stays out waiting for Flux to finish -// the apply. An apply that waits on health checks can take minutes; one Flux never finishes (the -// controller is down, the fetch failed) must not hold the applier forever, so the window then -// closes by time. +// DefaultApplyWindowTimeout bounds how long a delivered attempt stays out waiting for the applier to +// finish the apply. An apply that waits on health checks can take minutes; one the applier never +// finishes (the controller is down, the fetch failed) must not hold the applier forever, so the +// window then closes by time. It also bounds how long an Argo CD attempt waits for an operation +// already running on the Application before it sets its own. const DefaultApplyWindowTimeout = 5 * time.Minute -// DefaultHandledPollInterval is how often a delivered attempt reads whether Flux finished. +// DefaultHandledPollInterval is how often a delivered attempt reads whether the applier finished. const DefaultHandledPollInterval = 2 * time.Second // Dispatcher carries out the apply requests the branch workers decide. It implements @@ -84,22 +86,24 @@ const DefaultHandledPollInterval = 2 * time.Second // GitTarget again and asks only for those that are still the incarnation the worker decided for, // still name this applier on this branch, are still admitted by their ClusterProvider, and, for a // Refusal, still consent. When no target still holds the request's reason it asks nothing at all, -// rather than ask for something weaker: a weaker request is the worker's to decide. Then the Flux -// client checks the source binding before it stamps. +// rather than ask for something weaker: a weaker request is the worker's to decide. Then the +// applier's client (Flux or Argo CD, by the applier's kind) checks the source binding before it +// writes. type Dispatcher struct { reader client.Reader sourceClients SourceClients recorder record.EventRecorder log logr.Logger - // AttemptTimeout bounds one attempt up to its stamp. Set before Start. + // AttemptTimeout bounds one attempt up to its request. An Argo CD attempt may wait out an + // operation already running for up to ApplyWindowTimeout more. Set before Start. AttemptTimeout time.Duration - // ApplyWindowTimeout bounds the wait, after a stamp, for Flux to finish the apply it asked for; - // zero does not wait. Set before Start. + // ApplyWindowTimeout bounds the wait, after a request, for the applier to finish the apply it + // asked for; zero does not wait. Set before Start. ApplyWindowTimeout time.Duration - // HandledPollInterval is how often that wait reads Flux's status. Set before Start. + // HandledPollInterval is how often that wait reads the applier's status. Set before Start. HandledPollInterval time.Duration - // Now is the clock requestedAt stamps and the ledger read. + // Now is the clock request tokens and the ledger read. Now func() time.Time runMu sync.RWMutex @@ -209,12 +213,16 @@ type eligibleTarget struct { reason v1alpha3.ApplyReason } -// attempt carries out one request and returns its report. A delivered one is reported once Flux has -// finished the apply (awaitHandled), so the branch worker's attempt stays out for exactly the apply -// window: the seconds in which an edit can be reverted by it. +// attempt carries out one request and returns its report. A delivered one is reported once the +// applier has finished the apply (awaitHandled), so the branch worker's attempt stays out for +// exactly the apply window: the seconds in which an edit can be reverted by it. func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.ApplyResult { run := ctx - ctx, cancel := context.WithTimeout(ctx, d.AttemptTimeout) + timeout := d.AttemptTimeout + if req.Applier.Kind == v1alpha3.ApplierKindApplication { + timeout += d.ApplyWindowTimeout + } + ctx, cancel := context.WithTimeout(ctx, timeout) defer cancel() log := d.log.WithValues("applier", req.Applier.String(), "branch", req.Branch.String(), "reason", req.Reason, "commit", req.Commit) @@ -227,7 +235,7 @@ func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.Appl return d.failure(ctx, req, nil, report(git.ApplyFailed, err), log) } if !holdsReason(eligible, req.Reason) { - d.count(ctx, req.Reason, outcomeDropped) + d.count(ctx, req, outcomeDropped) log.Info("no GitTarget may still ask this applier for this; asking nothing") return report(git.ApplyDropped, nil) } @@ -237,12 +245,11 @@ func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.Appl return d.failure(ctx, req, eligible, report(git.ApplyFailed, fmt.Errorf("source cluster %q: %w", req.Applier.Cluster, err)), log) } - flux := NewFlux(sourceClient) - flux.Now = d.Now - stamped, err := flux.Request(ctx, applierName(req.Applier), want, req.Commit, req.Reason, req.Current) + applier := d.client(req.Applier.Kind, sourceClient) + stamped, err := applier.Request(ctx, applierName(req.Applier), want, req.Commit, req.Reason, req.Current) switch { case errors.Is(err, errStale): - d.count(ctx, req.Reason, outcomeStale) + d.count(ctx, req, outcomeStale) log.Info("the decision no longer holds; asking nothing, the branch worker decides again") return report(git.ApplyStale, nil) case isPermanent(err): @@ -251,22 +258,37 @@ func (d *Dispatcher) attempt(ctx context.Context, req git.ApplyRequest) git.Appl return d.failure(ctx, req, eligible, report(git.ApplyFailed, err), log) } - d.count(ctx, req.Reason, outcomeSent) + d.count(ctx, req, outcomeSent) if stamped.Outcome == OutcomeCurrent { - log.V(1).Info("applier already holds the commit; nothing to request") + log.V(1).Info("the applier needs no request for this commit; asking nothing") return report(git.ApplyCurrent, nil) } log.Info("apply requested", "outcome", stamped.Outcome) d.delivered(req, eligible) - d.awaitHandled(run, flux, applierName(req.Applier), stamped, log) + d.awaitHandled(run, applier, applierName(req.Applier), stamped, log) return report(git.ApplyDelivered, nil) } -// awaitHandled waits, up to ApplyWindowTimeout, for Flux to finish the apply stamped asked for. It -// decides nothing: the request was delivered either way, and a wait that ends by time or by -// shutdown only closes the window sooner than Flux did. +// client is the Client for an applier of kind, over its cluster's client and this dispatcher's +// clock and timings. +func (d *Dispatcher) client(kind string, sourceClient dynamic.Interface) Client { + if kind == v1alpha3.ApplierKindApplication { + argo := NewArgoCD(sourceClient) + argo.Now = d.Now + argo.OperationWait = d.ApplyWindowTimeout + argo.PollInterval = d.HandledPollInterval + return argo + } + flux := NewFlux(sourceClient) + flux.Now = d.Now + return flux +} + +// awaitHandled waits, up to ApplyWindowTimeout, for the applier to finish the apply stamped asked +// for. It decides nothing: the request was delivered either way, and a wait that ends by time or by +// shutdown only closes the window sooner than the applier did. func (d *Dispatcher) awaitHandled( - ctx context.Context, flux *Flux, ref types.NamespacedName, stamped Stamped, log logr.Logger, + ctx context.Context, applier Client, ref types.NamespacedName, stamped Stamped, log logr.Logger, ) { if d.ApplyWindowTimeout <= 0 { return @@ -277,18 +299,18 @@ func (d *Dispatcher) awaitHandled( ticker := time.NewTicker(d.HandledPollInterval) defer ticker.Stop() for { - handled, err := flux.Handled(ctx, ref, stamped) + handled, err := applier.Handled(ctx, ref, stamped) switch { case handled: - log.V(1).Info("Flux finished the requested apply", "after", d.Now().Sub(started).String()) + log.V(1).Info("the applier finished the requested apply", "after", d.Now().Sub(started).String()) return case err != nil && ctx.Err() == nil: - log.V(1).Info("could not read whether Flux finished the requested apply", "error", err.Error()) + log.V(1).Info("could not read whether the applier finished the requested apply", "error", err.Error()) } select { case <-ctx.Done(): if errors.Is(ctx.Err(), context.DeadlineExceeded) { - log.Info("Flux did not report the requested apply finished in time; the apply window closes now", + log.Info("the applier did not report the requested apply finished in time; the apply window closes now", "waited", d.Now().Sub(started).String(), "timeout", d.ApplyWindowTimeout.String()) } return @@ -297,7 +319,7 @@ func (d *Dispatcher) awaitHandled( } } -// applierName is the Kustomization an applier reference names. +// applierName is the object an applier reference names. func applierName(ref git.ApplierRef) types.NamespacedName { return types.NamespacedName{Namespace: ref.Namespace, Name: ref.Name} } @@ -317,14 +339,14 @@ func holdsReason(eligible []eligibleTarget, reason v1alpha3.ApplyReason) bool { func (d *Dispatcher) failure( ctx context.Context, req git.ApplyRequest, eligible []eligibleTarget, res git.ApplyResult, log logr.Logger, ) git.ApplyResult { - d.count(ctx, req.Reason, outcomeFailed) + d.count(ctx, req, outcomeFailed) log.Error(res.Err, "apply attempt failed; the branch worker decides when to try again", "outcome", res.Outcome, "priorFailures", req.PriorFailures) if req.PriorFailures == 0 && d.recorder != nil { for _, e := range eligible { d.recorder.Eventf(e.target, corev1.EventTypeWarning, EventReasonApplyRequestFailed, "could not ask %s to apply %s (%s), and will ask again once the branch is quiet: %v", - applierName(req.Applier), describeCommit(req.Commit), e.reason, res.Err) + describeApplier(req.Applier), describeCommit(req.Commit), e.reason, res.Err) } } return res @@ -427,10 +449,15 @@ func (d *Dispatcher) delivered(req git.ApplyRequest, eligible []eligibleTarget) } for _, e := range eligible { d.recorder.Eventf(e.target, corev1.EventTypeNormal, EventReasonApplyRequested, - "asked %s to apply %s (%s)", applierName(req.Applier), describeCommit(req.Commit), e.reason) + "asked %s to apply %s (%s)", describeApplier(req.Applier), describeCommit(req.Commit), e.reason) } } +// describeApplier names an applier in an Event: its kind and namespaced name. +func describeApplier(ref git.ApplierRef) string { + return ref.Kind + " " + applierName(ref).String() +} + // describeCommit names a commit in an Event: its short hash, or what an empty one means. func describeCommit(commit string) string { if commit == "" { @@ -443,13 +470,17 @@ func describeCommit(commit string) string { } // count records one apply_requests_total attempt. -func (d *Dispatcher) count(ctx context.Context, reason v1alpha3.ApplyReason, outcome string) { +func (d *Dispatcher) count(ctx context.Context, req git.ApplyRequest, outcome string) { if telemetry.ApplyRequestsTotal == nil { return } + kind := applierKindKustomization + if req.Applier.Kind == v1alpha3.ApplierKindApplication { + kind = applierKindApplication + } telemetry.ApplyRequestsTotal.Add(context.WithoutCancel(ctx), 1, metric.WithAttributes( - attribute.String("applier_kind", applierKindKustomization), - attribute.String("reason", string(reason)), + attribute.String("applier_kind", kind), + attribute.String("reason", string(req.Reason)), attribute.String("outcome", outcome), )) } diff --git a/internal/applier/dispatcher_test.go b/internal/applier/dispatcher_test.go index cbd77127..1a386f76 100644 --- a/internal/applier/dispatcher_test.go +++ b/internal/applier/dispatcher_test.go @@ -16,6 +16,7 @@ import ( "github.com/stretchr/testify/require" sdkmetric "go.opentelemetry.io/otel/sdk/metric" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/dynamic" @@ -164,8 +165,12 @@ func (f *dispatcherFixture) attempt(req git.ApplyRequest) git.ApplyResult { } func (f *dispatcherFixture) count(reason v1alpha3.ApplyReason, outcome string) int64 { + return f.countKind(applierKindKustomization, reason, outcome) +} + +func (f *dispatcherFixture) countKind(kind string, reason v1alpha3.ApplyReason, outcome string) int64 { value, _ := telemetry.CollectInt64Sum(f.reader, applyRequestsMetric, map[string]string{ - "applier_kind": applierKindKustomization, "reason": string(reason), "outcome": outcome, + "applier_kind": kind, "reason": string(reason), "outcome": outcome, }) return value } @@ -233,8 +238,8 @@ func TestDispatcher_AsksOnceForEveryTarget(t *testing.T) { assert.True(t, got.RequestedAt.Equal(&metav1.Time{Time: stampTime}), name) } assert.ElementsMatch(t, []string{ - "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (Refusal)", - "Normal ApplyRequested asked flux-system/checkout to apply 4f2c1ab9 (CatchUp)", + "Normal ApplyRequested asked Kustomization flux-system/checkout to apply 4f2c1ab9 (Refusal)", + "Normal ApplyRequested asked Kustomization flux-system/checkout to apply 4f2c1ab9 (CatchUp)", }, f.events()) assert.Equal(t, int64(1), f.count(v1alpha3.ApplyReasonRefusal, outcomeSent)) } @@ -314,8 +319,10 @@ func TestDispatcher_AFailureIsReportedAndNeverRetried(t *testing.T) { assert.Len(t, f.source.clusters, 1, "one attempt, no retry") events := f.events() require.Len(t, events, 1) - assert.True(t, strings.HasPrefix(events[0], - "Warning ApplyRequestFailed could not ask flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)"), events[0]) + assert.True(t, strings.HasPrefix( + events[0], + "Warning ApplyRequestFailed could not ask Kustomization flux-system/checkout to apply 4f2c1ab9 (ForeignCommit)", + ), events[0]) retry := request(checkout, v1alpha3.ApplyReasonForeignCommit, "a") retry.PriorFailures = 1 @@ -556,3 +563,94 @@ func TestDispatcher_AnApplyFluxNeverFinishesClosesByTime(t *testing.T) { assert.Equal(t, git.ApplyDelivered, f.attempt(request(checkout, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) } + +// argoTarget is the GitTarget "a", whose applier is the Application argocd/platform. +func argoTarget() *v1alpha3.GitTarget { + target := gitTarget("a", nil, "") + target.Spec.Applier = &v1alpha3.GitTargetApplier{ + ApplicationRef: &fluxmeta.NamespacedObjectReference{Namespace: argoNS, Name: "platform"}, + } + return target +} + +// argoPlatform is the applier argoTarget names. +var argoPlatform = git.ApplierRef{ + Cluster: v1alpha3.DefaultClusterProviderName, Kind: v1alpha3.ApplierKindApplication, + Namespace: argoNS, Name: "platform", +} + +func (f *dispatcherFixture) updates() []string { + var out []string + for _, action := range f.source.client.Actions() { + if action.GetVerb() == "update" { + out = append(out, action.GetResource().Resource+" "+action.GetNamespace()) + } + } + return out +} + +// TestDispatcher_AsksAnApplicationToSync: an Application applier is asked through the Argo CD +// client, with the commit pinned, counted and recorded under its own kind. +func TestDispatcher_AsksAnApplicationToSync(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{reapplying(argoTarget())}, manualApp()) + + res := f.attempt(request(argoPlatform, v1alpha3.ApplyReasonRefusal, "a")) + + assert.Equal(t, git.ApplyResult{ID: 1, Outcome: git.ApplyDelivered}, res) + assert.Equal(t, []string{"applications " + argoNS}, f.updates()) + assert.Empty(t, f.patches(), "no Flux object is stamped") + app, err := f.source.client.Resource(applicationGVR()).Namespace(argoNS). + Get(context.Background(), "platform", metav1.GetOptions{}) + require.NoError(t, err) + revision, _, _ := unstructured.NestedString(app.Object, "operation", "sync", "revision") + assert.Equal(t, testCommit, revision) + assert.Equal(t, []string{ + "Normal ApplyRequested asked Application argocd/platform to apply 4f2c1ab9 (Refusal)", + }, f.events()) + assert.Equal(t, int64(1), f.countKind(applierKindApplication, v1alpha3.ApplyReasonRefusal, outcomeSent)) + assert.Zero(t, f.count(v1alpha3.ApplyReasonRefusal, outcomeSent)) +} + +// TestDispatcher_AnApplicationWithAutomatedSyncIsRejectedAndNeverSynced: the standard is full +// control, so an Application that syncs on its own is never asked to, whatever its verdict says. +func TestDispatcher_AnApplicationWithAutomatedSyncIsRejectedAndNeverSynced(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{argoTarget()}, + applicationObj(testRepoURL, testBranch, map[string]any{"automated": map[string]any{"prune": true}})) + + res := f.attempt(request(argoPlatform, v1alpha3.ApplyReasonForeignCommit, "a")) + + assert.Equal(t, git.ApplyRejected, res.Outcome) + require.ErrorContains(t, res.Err, "has automated sync on") + assert.Empty(t, f.updates()) + assert.Equal(t, int64(1), f.countKind(applierKindApplication, v1alpha3.ApplyReasonForeignCommit, outcomeFailed)) +} + +// TestDispatcher_AnApplicationIsNotTheKustomizationOfItsName: the kind is part of which applier a +// GitTarget names, so a request for a Kustomization of the same name asks nothing for it. +func TestDispatcher_AnApplicationIsNotTheKustomizationOfItsName(t *testing.T) { + f := newDispatcherFixture(t, []client.Object{argoTarget()}, manualApp()) + sameName := kustomizationApplier(v1alpha3.DefaultClusterProviderName, argoNS, "platform") + + assert.Equal(t, git.ApplyDropped, f.attempt(request(sameName, v1alpha3.ApplyReasonForeignCommit, "a")).Outcome) + assert.Empty(t, f.updates()) +} + +// TestDispatcher_AnApplicationAttemptWaitsWithinItsWindow: an Argo CD attempt may wait out an +// operation already running, so it is bounded by the apply window on top of the attempt timeout. +func TestDispatcher_AnApplicationAttemptWaitsWithinItsWindow(t *testing.T) { + running := manualApp() + _ = unstructured.SetNestedMap(running.Object, theirOperation(), "operation") + f := newDispatcherFixture(t, []client.Object{argoTarget()}, running) + f.dispatcher.AttemptTimeout = 30 * time.Millisecond + f.dispatcher.ApplyWindowTimeout = 100 * time.Millisecond + f.dispatcher.HandledPollInterval = time.Millisecond + f.dispatcher.Now = time.Now + + res := f.attempt(request(argoPlatform, v1alpha3.ApplyReasonForeignCommit, "a")) + + assert.Equal(t, git.ApplyFailed, res.Outcome) + require.ErrorContains(t, res.Err, "has an operation in progress") + require.NotErrorIs(t, res.Err, context.DeadlineExceeded, + "the wait ends by the operation wait, not by the shorter attempt timeout") + assert.Empty(t, f.updates()) +} diff --git a/internal/applier/flux.go b/internal/applier/flux.go index 2ca43c57..6f6da725 100644 --- a/internal/applier/flux.go +++ b/internal/applier/flux.go @@ -1,34 +1,17 @@ // SPDX-License-Identifier: Apache-2.0 -/* -Package applier delivers apply requests: what GitOps Reverser sends the object that applies a -GitTarget's folder back to its source cluster (its applier) when the branch worker has decided a -commit is safe to apply. The worker decides WHEN; this package decides HOW, off the worker's queue, -so a slow or unreachable applier never holds a branch. - -Only Flux is served: spec.applier.kustomizationRef, a Kustomization whose sourceRef is a -GitRepository. See docs/design/apply-trigger.md. - -A request asks and never reports. Nothing here claims an apply happened; the applier's own status -(lastAppliedRevision on a Kustomization) is the record of that. -*/ package applier import ( "context" "encoding/json" - "errors" "fmt" "strings" "time" fluxmeta "github.com/fluxcd/pkg/apis/meta" - authorizationv1 "k8s.io/api/authorization/v1" - apierrors "k8s.io/apimachinery/pkg/api/errors" - apimeta "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" - "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/dynamic" @@ -52,21 +35,13 @@ func gitRepositoryGVR() schema.GroupVersionResource { return schema.GroupVersionResource{Group: "source.toolkit.fluxcd.io", Version: "v1", Resource: "gitrepositories"} } -// selfSubjectAccessReviewGVR is the access check Validate creates. -func selfSubjectAccessReviewGVR() schema.GroupVersionResource { - return schema.GroupVersionResource{ - Group: "authorization.k8s.io", Version: "v1", Resource: "selfsubjectaccessreviews", - } -} - const ( kindKustomization = "Kustomization" kindGitRepository = "GitRepository" ) -// Outcome is what one Flux request did. -type Outcome string - +// What a Flux request stamped. OutcomeCurrent stamps nothing: a CatchUp or ForeignCommit found the +// artifact already at the commit, so the Kustomization was already woken by it. const ( // OutcomeStampedSource stamped the GitRepository: its artifact did not hold the commit, and the // new artifact it produces wakes the Kustomization by itself. @@ -74,67 +49,8 @@ const ( // OutcomeStampedKustomization stamped the Kustomization: the artifact already held the commit, // so a source stamp would produce no new artifact and wake nothing. OutcomeStampedKustomization Outcome = "StampedKustomization" - // OutcomeCurrent stamped nothing: a CatchUp or ForeignCommit found the artifact already at the - // commit, so the Kustomization was already woken by it. - OutcomeCurrent Outcome = "Current" ) -// ApplierConfigured reasons (docs/configuration.md, "Who applies this folder"). Succeeded and -// Failed are the generic reasons, aliased from Flux's meta package as the conditions guide asks. -const ( - // ReasonSucceeded is ApplierConfigured=True: the applier exists, applies this repository and - // branch, and may be asked to apply. - ReasonSucceeded = fluxmeta.SucceededReason - // ReasonApplierNotFound is no such object in the source cluster, or its kind is not served there. - ReasonApplierNotFound = "ApplierNotFound" - // ReasonSourceMismatch is an applier that applies a different repository or branch. - ReasonSourceMismatch = "SourceMismatch" - // ReasonForbidden is a source-cluster identity that may not read or patch the applier. - ReasonForbidden = "Forbidden" - // ReasonCheckFailed is a check that could not complete: the source cluster did not answer. - // Verdict.Err carries the cause; it says nothing about the applier itself. - ReasonCheckFailed = fluxmeta.FailedReason -) - -// Verdict is the answer Validate gives for the ApplierConfigured condition. -type Verdict struct { - // OK is ApplierConfigured=True. - OK bool - // Reason is the condition reason. - Reason string - // Message names the object and what is wrong with it. - Message string - // Err is set only with ReasonCheckFailed: the read that failed. A caller may prefer to keep the - // verdict it already published over replacing a known answer with "could not tell". - Err error -} - -// Stamped is what one Request did, with what Flux reported before it, so Handled can tell when -// Flux has finished the apply it asked for. -type Stamped struct { - // Outcome is what was stamped, if anything. - Outcome Outcome - // Token is the reconcile.fluxcd.io/requestedAt value stamped; empty when nothing was. - Token string - // Source is the GitRepository the Kustomization applies from. - Source types.NamespacedName - // PriorAttemptedRevision is the Kustomization's status.lastAttemptedRevision before the stamp. - PriorAttemptedRevision string - // PriorArtifactRevision is the GitRepository's status.artifact.revision before the stamp. - PriorArtifactRevision string -} - -// Binding is the repository and branch an applier must apply from: the GitProvider's spec.url and -// the GitTarget's spec.branch. Request checks it before it writes anything, not only Validate, -// because a published ApplierConfigured verdict can be older than the applier it describes. -type Binding struct { - RepositoryURL string - Branch string -} - -// AccessCheck reports whether the source-cluster identity may perform attrs. -type AccessCheck func(ctx context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) - // Flux asks a Flux Kustomization to apply, in one source cluster. type Flux struct { client dynamic.Interface @@ -152,24 +68,6 @@ func NewFlux(client dynamic.Interface) *Flux { return &Flux{client: client, Now: time.Now, CanI: SelfSubjectAccessCheck(client)} } -// permanentError marks a request no retry can make succeed: the Kustomization is configured in a -// way a request cannot work with, or applies another repository or branch. The dispatcher hands it -// back to the branch worker at once rather than retrying it, and the worker asks again later, -// because the configuration may have been fixed in between. -type permanentError struct{ err error } - -func (e permanentError) Error() string { return e.err.Error() } -func (e permanentError) Unwrap() error { return e.err } - -// errStale reports a request whose decision no longer held when the stamp was due. -var errStale = errors.New("the decision no longer holds: the branch moved on or became busy") - -// isPermanent reports whether err is a permanentError. -func isPermanent(err error) bool { - var p permanentError - return errors.As(err, &p) -} - // Request asks the Kustomization ref to apply commit, the way `flux reconcile kustomization // --with-source` does, minus the waiting: Request only asks, and Handled tells when Flux is done. // @@ -278,12 +176,6 @@ func (f *Flux) Handled(ctx context.Context, ref types.NamespacedName, s Stamped) statusString(source, "artifact", "revision") == s.PriorArtifactRevision, nil } -// statusString reads a string under an object's status. -func statusString(obj *unstructured.Unstructured, fields ...string) string { - value, _, _ := unstructured.NestedString(obj.Object, append([]string{"status"}, fields...)...) - return value -} - // stamp sets reconcile.fluxcd.io/requestedAt to now, which is the whole of what `flux reconcile` // does to ask: requestReconciliation in external-sources/flux/flux2/cmd/flux/reconcile.go // (@1ed703a3) merge-patches that annotation, and only that, with time.Now().Format(time.RFC3339Nano). @@ -292,7 +184,7 @@ func statusString(obj *unstructured.Unstructured, fields ...string) string { // // It returns the value stamped, which Flux copies to status.lastHandledReconcileAt once handled. func (f *Flux) stamp(ctx context.Context, gvr schema.GroupVersionResource, ref types.NamespacedName) (string, error) { - token := f.Now().Format(time.RFC3339Nano) + token := requestToken(f.Now()) patch, err := json.Marshal(map[string]any{ "metadata": map[string]any{ "annotations": map[string]string{fluxmeta.ReconcileRequestAnnotation: token}, @@ -374,7 +266,7 @@ func splitRevision(revision string) (string, string) { func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, want Binding) Verdict { ks, err := f.client.Resource(kustomizationGVR()).Namespace(ref.Namespace).Get(ctx, ref.Name, metav1.GetOptions{}) if err != nil { - return readFailure(kindKustomization, kustomizationGVR(), ref, err) + return readFailure("Flux", kindKustomization, kustomizationGVR(), ref, err) } sourceRef, err := gitRepositoryRef(ks) if err != nil { @@ -383,7 +275,7 @@ func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, want Bind source, err := f.client.Resource(gitRepositoryGVR()).Namespace(sourceRef.Namespace). Get(ctx, sourceRef.Name, metav1.GetOptions{}) if err != nil { - verdict := readFailure(kindGitRepository, gitRepositoryGVR(), sourceRef, err) + verdict := readFailure("Flux", kindGitRepository, gitRepositoryGVR(), sourceRef, err) verdict.Message = fmt.Sprintf("%s (the sourceRef of %s %s)", verdict.Message, kindKustomization, ref) return verdict } @@ -399,27 +291,8 @@ func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, want Bind {kindGitRepository, gitRepositoryGVR(), sourceRef}, {kindKustomization, kustomizationGVR(), ref}, } { - allowed, err := f.CanI(ctx, authorizationv1.ResourceAttributes{ - Namespace: obj.ref.Namespace, - Verb: "patch", - Group: obj.gvr.Group, - Version: obj.gvr.Version, - Resource: obj.gvr.Resource, - Name: obj.ref.Name, - }) - if err != nil { - return Verdict{ - Reason: ReasonCheckFailed, - Message: fmt.Sprintf("could not check whether %s %s may be patched: %v", obj.kind, obj.ref, err), - Err: err, - } - } - if !allowed { - return Verdict{ - Reason: ReasonForbidden, - Message: fmt.Sprintf("the source-cluster identity may not patch %s %s (%s)", - obj.kind, obj.ref, obj.gvr.GroupResource()), - } + if verdict := accessVerdict(ctx, f.CanI, obj.kind, obj.gvr, obj.ref, "patch"); verdict != nil { + return *verdict } } @@ -431,50 +304,6 @@ func (f *Flux) Validate(ctx context.Context, ref types.NamespacedName, want Bind } } -// readFailure classifies a failed GET for Validate. -// -// A kind that is not served is ApplierNotFound like a missing object, because the fix is the same -// (install Flux, or name the right object) and the condition is a gate either way. The two are -// told apart only for the message: a client with a RESTMapper says so with a NoKindMatch, and the -// API server answers the dynamic client's GET of an unserved resource with a 404 whose details name -// no object, where a missing object's 404 names it. -func readFailure(kind string, gvr schema.GroupVersionResource, ref types.NamespacedName, err error) Verdict { - switch { - case apimeta.IsNoMatchError(err) || (apierrors.IsNotFound(err) && !namesObject(err, ref.Name)): - return Verdict{ - Reason: ReasonApplierNotFound, - Message: fmt.Sprintf("the source cluster does not serve %s (%s); is Flux installed there?", - kind, gvr.GroupVersion()), - } - case apierrors.IsNotFound(err): - return Verdict{ - Reason: ReasonApplierNotFound, - Message: fmt.Sprintf("%s %s not found in the source cluster", kind, ref), - } - case apierrors.IsForbidden(err): - return Verdict{ - Reason: ReasonForbidden, - Message: fmt.Sprintf("the source-cluster identity may not get %s %s: %v", kind, ref, err), - } - default: - return Verdict{ - Reason: ReasonCheckFailed, - Message: fmt.Sprintf("could not read %s %s: %v", kind, ref, err), - Err: err, - } - } -} - -// namesObject reports whether a NotFound error is about the named object, rather than the resource. -func namesObject(err error, name string) bool { - var status apierrors.APIStatus - if !errors.As(err, &status) { - return false - } - details := status.Status().Details - return details != nil && details.Name == name -} - // sourceMismatch returns why a GitRepository does not fetch branch of repositoryURL, or "" when it // does. A ref that pins anything stronger than a branch is a mismatch even when it also names the // branch: Flux resolves commit, then name, then semver, then tag, before branch, so the pin wins and @@ -500,32 +329,3 @@ func sourceMismatch(source *unstructured.Unstructured, ref types.NamespacedName, } return "" } - -// SelfSubjectAccessCheck answers an AccessCheck with a SelfSubjectAccessReview created through -// client, so it is answered for the identity client authenticates as. It goes through the dynamic -// client rather than a typed clientset because that client is the only one a source cluster has. -func SelfSubjectAccessCheck(client dynamic.Interface) AccessCheck { - return func(ctx context.Context, attrs authorizationv1.ResourceAttributes) (bool, error) { - review := &authorizationv1.SelfSubjectAccessReview{ - TypeMeta: metav1.TypeMeta{ - APIVersion: authorizationv1.SchemeGroupVersion.String(), - Kind: "SelfSubjectAccessReview", - }, - Spec: authorizationv1.SelfSubjectAccessReviewSpec{ResourceAttributes: &attrs}, - } - content, err := runtime.DefaultUnstructuredConverter.ToUnstructured(review) - if err != nil { - return false, err - } - created, err := client.Resource(selfSubjectAccessReviewGVR()). - Create(ctx, &unstructured.Unstructured{Object: content}, metav1.CreateOptions{}) - if err != nil { - return false, fmt.Errorf("create SelfSubjectAccessReview: %w", err) - } - allowed, _, err := unstructured.NestedBool(created.Object, "status", "allowed") - if err != nil { - return false, fmt.Errorf("read SelfSubjectAccessReview status: %w", err) - } - return allowed, nil - } -} diff --git a/internal/applier/flux_test.go b/internal/applier/flux_test.go index ee6bdb2b..d0cbd718 100644 --- a/internal/applier/flux_test.go +++ b/internal/applier/flux_test.go @@ -43,6 +43,7 @@ func newSourceClient(objs ...runtime.Object) *dynamicfake.FakeDynamicClient { map[schema.GroupVersionResource]string{ kustomizationGVR(): "KustomizationList", gitRepositoryGVR(): "GitRepositoryList", + applicationGVR(): "ApplicationList", }, objs...) } diff --git a/internal/telemetry/apply_requests.go b/internal/telemetry/apply_requests.go index de17fe86..0eaf2fa0 100644 --- a/internal/telemetry/apply_requests.go +++ b/internal/telemetry/apply_requests.go @@ -7,11 +7,13 @@ import "go.opentelemetry.io/otel/metric" // ApplyRequestsTotal counts the apply attempts GitOps Reverser made, one per attempt to one applier // object, labelled by {applier_kind, reason, outcome}. // -// applier_kind names the applier: "kustomization" (a Flux Kustomization). reason is the attempt's -// ApplyReason (ForeignCommit, Refusal, CatchUp). outcome is a frozen enum: +// applier_kind names the applier: "kustomization" (a Flux Kustomization) or "application" (an Argo +// CD Application). reason is the attempt's ApplyReason (ForeignCommit, Refusal, CatchUp; never +// CatchUp for an application). outcome is a frozen enum: // -// sent — the applier was asked and the ask landed: a requestedAt stamp was patched, or, for a -// CatchUp, the applier's source already held the commit and nothing was owed. +// sent — the applier was asked and the ask landed: a requestedAt stamp was patched or a sync +// operation set, or, for a CatchUp, the applier's source already held the commit and +// nothing was owed. // failed — the attempt could not ask, including an applier that applies another repository or // branch. The branch worker owns the obligation and decides a retry on a quiet branch, // with a backoff of its own, so a failure that persists keeps counting. An @@ -19,7 +21,7 @@ import "go.opentelemetry.io/otel/metric" // dropped — the attempt asked nothing, because no GitTarget may still ask for its reason: each // was deleted or recreated, moved to another branch or applier, is no longer admitted by // its ClusterProvider, or withdrew its consent to a Refusal. -// stale — the attempt asked nothing, because the decision no longer held when the patch was due: +// stale — the attempt asked nothing, because the decision no longer held when the write was due: // the branch had moved on or become busy. The worker decides again at the next quiet // moment. Frequent stale attempts mean edits keep arriving within an attempt's lifetime. // From 16bc0986757b3f0b6dbb232dad4a7d9052413d62 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 07:17:47 +0000 Subject: [PATCH 32/36] feat(controller): check an Argo CD applier, grant its RBAC, and document it ApplierConfigured checks the applier by its kind: an Application must exist, sync this repository and branch from a single source (targetRevision the branch name or refs/heads/), have automated sync off, and be updatable. Automated sync on is the new AutomatedSyncEnabled reason, which stalls the GitTarget like the other configuration errors. RBAC: config/rbac/argocd-applier-role.yaml and the chart's applierAccess.argocd grant get and update on argoproj.io applications, off by default in the chart, beside the Flux grant. docs/argocd.md is the Argo CD guide, with the standard spelled out (automated sync off; refresh and Argo CD's webhook may stay) and its complete example held to the code by argocd_guide_test.go. The configuration reference, the bi-directional guide, the support boundary, the metrics guide and UPGRADING describe the Application applier, and apply-trigger.md takes what lasts of the implementation plan, which is deleted. Co-Authored-By: Claude Opus 5.5 (1M context) --- .docs-lint-scope | 1 + charts/gitops-reverser/README.md | 3 +- .../templates/applier-rbac.yaml | 43 +++ charts/gitops-reverser/values.schema.json | 6 +- charts/gitops-reverser/values.yaml | 4 + cmd/main_chart_receiver_test.go | 35 ++ config/rbac/argocd-applier-role.yaml | 30 ++ config/rbac/argocd-applier-rolebinding.yaml | 15 + config/rbac/kustomization.yaml | 3 + docs/README.md | 1 + docs/UPGRADING.md | 17 +- docs/argocd.md | 344 ++++++++++++++++++ docs/bi-directional.md | 113 +++--- docs/configuration.md | 107 +++--- docs/design/apply-trigger.md | 130 ++++--- docs/design/argocd-applier-plan.md | 140 ------- .../support-boundary/argocd-bi-directional.md | 48 ++- docs/interpreting-metrics.md | 11 +- hack/crdfields/fields.yaml | 9 +- internal/applier/argocd_guide_test.go | 76 ++++ internal/applier/flux_guide_test.go | 10 +- internal/controller/gittarget_applier.go | 48 ++- internal/controller/gittarget_applier_test.go | 83 ++++- internal/controller/gittarget_controller.go | 13 +- 24 files changed, 944 insertions(+), 346 deletions(-) create mode 100644 config/rbac/argocd-applier-role.yaml create mode 100644 config/rbac/argocd-applier-rolebinding.yaml create mode 100644 docs/argocd.md delete mode 100644 docs/design/argocd-applier-plan.md create mode 100644 internal/applier/argocd_guide_test.go diff --git a/.docs-lint-scope b/.docs-lint-scope index 0c1fd9ba..5e1a3ec3 100644 --- a/.docs-lint-scope +++ b/.docs-lint-scope @@ -26,6 +26,7 @@ README.md docs/architecture.md docs/configuration.md docs/flux.md +docs/argocd.md docs/spec/attribution.md docs/components.md docs/quickstart.md diff --git a/charts/gitops-reverser/README.md b/charts/gitops-reverser/README.md index 14f4e57e..73231299 100644 --- a/charts/gitops-reverser/README.md +++ b/charts/gitops-reverser/README.md @@ -181,7 +181,8 @@ nodeSelector: | `controllerManager.commitRequestTTL` | How long a `CommitRequest` is kept once it finishes. Written onto the request as the `configbutler.ai/delete-after` annotation; remove or edit that to keep one. `"0"` writes none | `48h` | | `git.refreshInterval` | How often an idle `GitTarget` re-proves where its branch is on the Git remote. `"0"` disables it | `10m` | | `receiver.bindAddress` | `host:port` the push receiver listens on, over plain HTTP (`--receiver-bind-address`), for Git hosts' push webhooks to `GitProvider.status.webhookPath`. Empty or `"0"` serves no receiver. When set, the chart adds a `-receiver` Service; expose it with an Ingress or Gateway route of your own that terminates TLS. See [push receiver settings](../../docs/configuration.md#push-receiver-settings) | `""` | -| `applierAccess.flux` | Grant the operator `get`, `list`, `watch`, `patch` on Flux `Kustomizations` and `GitRepositories`, and `create` on `SelfSubjectAccessReviews`, so it can ask Flux to apply a commit somebody else pushed | `false` | +| `applierAccess.flux` | Grant the operator `get` and `patch` on Flux `Kustomizations` and `GitRepositories`, and `create` on `SelfSubjectAccessReviews`, so it can ask Flux to apply a commit somebody else pushed | `false` | +| `applierAccess.argocd` | Grant the operator `get` and `update` on Argo CD `Applications`, and `create` on `SelfSubjectAccessReviews`, so it can start the syncs of an `Application` named by `spec.applier.applicationRef` | `false` | | `auditService.type` | Service type for the dedicated audit Service | `NodePort` | | `auditService.nodePort` | Fixed NodePort for the audit Service when `auditService.type=NodePort` | `30444` | | `auditService.clusterIP` | Optional fixed ClusterIP for the dedicated audit Service | `""` | diff --git a/charts/gitops-reverser/templates/applier-rbac.yaml b/charts/gitops-reverser/templates/applier-rbac.yaml index f5a5fe94..683b1704 100644 --- a/charts/gitops-reverser/templates/applier-rbac.yaml +++ b/charts/gitops-reverser/templates/applier-rbac.yaml @@ -48,3 +48,46 @@ subjects: name: {{ include "gitops-reverser.serviceAccountName" . }} namespace: {{ .Release.Namespace }} {{- end }} +{{- if and .Values.rbac.create .Values.applierAccess.argocd }} +--- +# What GitOps Reverser needs to ask an Argo CD Application to sync a commit it checked: read the +# Application and update it with a sync operation (how Argo CD's own automated sync starts one), and +# create SelfSubjectAccessReviews so the operator can check it holds that access before relying on +# it. Separate from the manager role because it is optional and names another project's API: an +# install without Argo CD grants nothing on it. See docs/argocd.md. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: {{ include "gitops-reverser.fullname" . }}-applier-argocd + labels: + {{- include "gitops-reverser.labels" . | nindent 4 }} +rules: +- apiGroups: + - argoproj.io + resources: + - applications + verbs: + - get + - update +- apiGroups: + - authorization.k8s.io + resources: + - selfsubjectaccessreviews + verbs: + - create +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: {{ include "gitops-reverser.fullname" . }}-applier-argocd + labels: + {{- include "gitops-reverser.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: {{ include "gitops-reverser.fullname" . }}-applier-argocd +subjects: +- kind: ServiceAccount + name: {{ include "gitops-reverser.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} +{{- end }} diff --git a/charts/gitops-reverser/values.schema.json b/charts/gitops-reverser/values.schema.json index a59a353b..3916f782 100644 --- a/charts/gitops-reverser/values.schema.json +++ b/charts/gitops-reverser/values.schema.json @@ -152,7 +152,11 @@ "properties": { "flux": { "type": "boolean", - "description": "Grant get/list/watch/patch on Flux Kustomizations and GitRepositories, plus create on SelfSubjectAccessReviews." + "description": "Grant get/patch on Flux Kustomizations and GitRepositories, plus create on SelfSubjectAccessReviews." + }, + "argocd": { + "type": "boolean", + "description": "Grant get/update on Argo CD Applications, plus create on SelfSubjectAccessReviews." } } }, diff --git a/charts/gitops-reverser/values.yaml b/charts/gitops-reverser/values.yaml index 24ec452c..38e3568a 100644 --- a/charts/gitops-reverser/values.yaml +++ b/charts/gitops-reverser/values.yaml @@ -134,6 +134,10 @@ applierAccess: # on it). Cluster-wide, because a GitTarget's applier can live in any # namespace. flux: false + # Grant get/update on argoproj.io Applications (a sync is started by updating the Application's + # operation, as Argo CD's own automated sync does), plus create on SelfSubjectAccessReviews. + # Cluster-wide, like the Flux grant. + argocd: false # cert-manager issuer shared by every certificate the chart mints. One self-signed CA # backs them all, so the issuer is genuinely cross-cutting and lives here; each server diff --git a/cmd/main_chart_receiver_test.go b/cmd/main_chart_receiver_test.go index e83589cc..9fc4d251 100644 --- a/cmd/main_chart_receiver_test.go +++ b/cmd/main_chart_receiver_test.go @@ -196,3 +196,38 @@ func TestChartApplierAccess_Flux(t *testing.T) { assert.Equal(t, "ServiceAccount", binding.Subjects[0].Kind) assert.Equal(t, "gitops-reverser", binding.Subjects[0].Name) } + +// applierAccess.argocd is off by default, and on it grants exactly what starting an Argo CD sync +// needs (get, and update to set the Application's operation), apart from the Flux grant. +func TestChartApplierAccess_ArgoCD(t *testing.T) { + assert.Empty(t, helmTemplateDocs(t, "templates/applier-rbac.yaml", + "applierAccess.argocd=true", "rbac.create=false"), "rbac.create=false renders no RBAC at all") + + docs := helmTemplateDocs(t, "templates/applier-rbac.yaml", "applierAccess.argocd=true") + require.Len(t, docs, 2, "the Flux grant stays off") + + var role rbacv1.ClusterRole + require.NoError(t, yaml.Unmarshal([]byte(docs[0]), &role)) + assert.Equal(t, "gitops-reverser-applier-argocd", role.Name) + assert.Equal(t, []rbacv1.PolicyRule{ + { + APIGroups: []string{"argoproj.io"}, + Resources: []string{"applications"}, + Verbs: []string{"get", "update"}, + }, + { + APIGroups: []string{"authorization.k8s.io"}, + Resources: []string{"selfsubjectaccessreviews"}, + Verbs: []string{"create"}, + }, + }, role.Rules) + + var binding rbacv1.ClusterRoleBinding + require.NoError(t, yaml.Unmarshal([]byte(docs[1]), &binding)) + assert.Equal(t, role.Name, binding.RoleRef.Name) + require.Len(t, binding.Subjects, 1) + assert.Equal(t, "gitops-reverser", binding.Subjects[0].Name) + + assert.Len(t, helmTemplateDocs(t, "templates/applier-rbac.yaml", + "applierAccess.argocd=true", "applierAccess.flux=true"), 4, "both grants together") +} diff --git a/config/rbac/argocd-applier-role.yaml b/config/rbac/argocd-applier-role.yaml new file mode 100644 index 00000000..e8d1512c --- /dev/null +++ b/config/rbac/argocd-applier-role.yaml @@ -0,0 +1,30 @@ +# What GitOps Reverser needs to drive an Argo CD Application named in GitTarget.spec.applier: read +# the Application, set its operation to a sync (an update of the whole object, as Argo CD's own +# SetAppOperation makes it), and check that it may. +# +# It is kept OUT of role.yaml because no GitTarget needs it until one names an Application, and an +# install that does not use Argo CD should not hold update rights on Applications. Drop this pair +# when no GitTarget sets spec.applier.applicationRef (Helm: `applierAccess.argocd: false`, the +# default there). For an Application in another cluster, grant the same to the identity of the +# ClusterProvider's kubeconfig there. +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + labels: + app.kubernetes.io/managed-by: kustomize + app.kubernetes.io/name: gitops-reverser + name: gitops-reverser-argocd-applier +rules: + - apiGroups: + - argoproj.io + resources: + - applications + verbs: + - get + - update + - apiGroups: + - authorization.k8s.io + resources: + - selfsubjectaccessreviews + verbs: + - create diff --git a/config/rbac/argocd-applier-rolebinding.yaml b/config/rbac/argocd-applier-rolebinding.yaml new file mode 100644 index 00000000..8b9f6e37 --- /dev/null +++ b/config/rbac/argocd-applier-rolebinding.yaml @@ -0,0 +1,15 @@ +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + labels: + app.kubernetes.io/managed-by: kustomize + app.kubernetes.io/name: gitops-reverser + name: gitops-reverser-argocd-applier +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: gitops-reverser-argocd-applier +subjects: +- kind: ServiceAccount + name: gitops-reverser + namespace: sut diff --git a/config/rbac/kustomization.yaml b/config/rbac/kustomization.yaml index 10cdf148..53a056e9 100644 --- a/config/rbac/kustomization.yaml +++ b/config/rbac/kustomization.yaml @@ -10,6 +10,9 @@ resources: # Patch rights on Flux Kustomizations and GitRepositories, for GitTarget.spec.applier. See the file. - flux-applier-role.yaml - flux-applier-rolebinding.yaml + # Update rights on Argo CD Applications, for GitTarget.spec.applier.applicationRef. See the file. + - argocd-applier-role.yaml + - argocd-applier-rolebinding.yaml - cluster-role-jane-aggregated.yaml - cluster-rolebinding.yaml - cluster-rolebinding-jane.yaml # E2E-only helper: allows impersonated user jane@acme.com writes in tests. diff --git a/docs/README.md b/docs/README.md index de359711..9f7c0ffa 100644 --- a/docs/README.md +++ b/docs/README.md @@ -30,6 +30,7 @@ If you only want the supported product docs, start with the files below. - [`rbac.md`](rbac.md): the two ClusterRoles, and how to stop the reverser enumerating Secrets - [`bi-directional.md`](bi-directional.md): safe shared-path and handoff patterns - [`flux.md`](flux.md): the complete Flux setup when GitOps Reverser decides when Flux applies +- [`argocd.md`](argocd.md): the complete Argo CD setup when GitOps Reverser starts every sync - [`alternatives.md`](alternatives.md): nearby tools and when another approach fits better - [`UPGRADING.md`](UPGRADING.md): breaking changes and migration steps, newest first - [`../CONTRIBUTING.md`](../CONTRIBUTING.md): contributor workflow and validation commands diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index a49549b7..a7aa3c46 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -12,11 +12,12 @@ We are pre-1.0, so breaking changes bump the **minor** version (release-please i **Breaking.** `GitTarget.spec.onRefusal` accepts `Ignore` and `Reapply`. `PushEmptyCommit` is gone: GitOps Reverser no longer moves the branch with an empty commit when an edit is refused. With `Reapply` it asks the target's applier, the Flux `Kustomization` named in the new -`spec.applier.kustomizationRef`, to apply the current tip once the branch holds no open commit window -and no pending writes. Only that applier is asked, rather than every applier watching the branch, -though a `GitRepository` that has to fetch the tip still wakes every `Kustomization` that reads it. -An allowed edit still in a commit window is published before the apply instead of being reverted -with the refused one. The API rejects `Reapply` without `spec.applier`. +`spec.applier.kustomizationRef` or the Argo CD `Application` named in `spec.applier.applicationRef`, +to apply the current tip once the branch holds no open commit window and no pending writes. Only +that applier is asked, rather than every applier watching the branch, though a `GitRepository` that +has to fetch the tip still wakes every `Kustomization` that reads it. An allowed edit still in a +commit window is published before the apply instead of being reverted with the refused one. The API +rejects `Reapply` without `spec.applier`. A stored `GitTarget` that still says `PushEmptyCommit` reads as `Ignore`, and any update to it is rejected until the field changes. The old CRD rejects `Reapply`, so move in this order: @@ -37,6 +38,12 @@ schedule as well. The operator needs `get` and `patch` on `kustomizations` and ` and `create` on `selfsubjectaccessreviews`, in the source cluster: Helm `applierAccess.flux: true` grants them for the operator's own cluster. +For an Argo CD `Application`, GitOps Reverser starts every sync, so switch its automated sync off +(`spec.syncPolicy.automated.enabled: false`, keeping `prune`) before naming it; one with automated +sync on reports `ApplierConfigured=False` (`AutomatedSyncEnabled`) and is never synced. The operator +needs `get` and `update` on `applications`: Helm `applierAccess.argocd: true`. See +[Using GitOps Reverser with Argo CD](argocd.md). + ## `git_queue_drops_total` is now `git_queue_refusals_total` `gitopsreverser_git_queue_drops_total` is renamed `gitopsreverser_git_queue_refusals_total`, with the diff --git a/docs/argocd.md b/docs/argocd.md new file mode 100644 index 00000000..674c9aa8 --- /dev/null +++ b/docs/argocd.md @@ -0,0 +1,344 @@ +# Using GitOps Reverser with Argo CD + +When GitOps Reverser publishes a folder that an Argo CD `Application` syncs, GitOps Reverser takes +full control of when that `Application` syncs. The `Application` has automated sync switched off, +and every sync it runs is one GitOps Reverser started, pinned to a commit it checked. Argo CD still +renders, applies, prunes, runs hooks and reports health; it no longer decides when. This page is the +complete Argo CD setup: what to change on the `Application`, what to declare on the `GitTarget`, +where the Git host's webhook goes, how to check it works, and which tests prove each part. + +The [bi-directional guide](bi-directional.md) explains the model behind this for any applier, and +the [configuration reference](configuration.md#who-applies-this-folder-specapplier) has every field +and condition reason. [Using GitOps Reverser with Flux](flux.md) is the same setup for Flux. + +## What changes + +| Before | With GitOps Reverser | +| --- | --- | +| `spec.syncPolicy.automated` syncs every new commit | Automated sync is off: `automated.enabled: false`, or no `automated` block | +| A sync starts when Argo CD's refresh finds a new commit | A sync starts when GitOps Reverser sets the `Application`'s `operation`, pinned to a commit | +| `selfHeal` reverts live drift | Nothing reverts live drift by itself; GitOps Reverser publishes it to Git | +| The Git host's push webhook goes to Argo CD | It goes to GitOps Reverser's receiver too; Argo CD's may stay | +| Nothing ties a folder to its `Application` | The `GitTarget` names it in `spec.applier.applicationRef` | + +Argo CD's timed refresh and its `/api/webhook` can stay as they are. With automated sync off a +refresh only **compares**: it resolves the branch tip and reports `Synced` or `OutOfSync`, and +applies nothing. + +## Why automated sync must be off + +**Automated sync applies every new commit it finds, whenever its refresh finds it.** Each push +GitOps Reverser makes is a new commit, so two to three minutes after each push (Argo CD's +`timeout.reconciliation`, 120s plus jitter), or seconds after it when Argo CD's webhook is wired, +the whole `Application` is synced over whatever is pending at that moment. `selfHeal: false` only +skips a commit already synced; it does not stop the next one. That brings back the two races GitOps +Reverser exists to close: + +- **Its own pushes.** Two API edits a few seconds apart produce two commits. A sync of the first + commit after the second edit reverts that edit, and GitOps Reverser then publishes the revert as + a third commit. +- **Somebody else's push.** While an API edit is still in an open commit window, a Git-side push + arrives. Synced straight away, the push's tree reverts the edit, because the edit is not in Git + yet. GitOps Reverser instead closes the window, replays the edit onto the push, pushes the + result, and only then asks Argo CD to sync it. + +Both apply something correct, at the wrong moment. Only GitOps Reverser can see the edits that are +live and not yet pushed, so only it can tell when a sync is safe. It starts one only on a **quiet +branch**: no open commit window and no pending writes. + +An `Application` with automated sync on is reported `ApplierConfigured=False` with reason +`AutomatedSyncEnabled`, and GitOps Reverser never syncs it. It is a misconfiguration to fix, and +stalls the `GitTarget`: an `Application` that syncs on its own is outside GitOps Reverser's control. + +**There is no catch-up.** Flux needs one, because it applies on its own timers and on a restart +from whatever artifact it holds, and a stale artifact would revert edits already in Git. With +automated sync off, Argo CD applies nothing on its own, so nothing it holds can go stale. GitOps +Reverser's own commits are already live (they were read from the cluster), and after its next +refresh the `Application` reports them `Synced`. So `spec.applier.quietPeriod` is rejected with an +`applicationRef`. + +## A complete example + +One folder, `apps/checkout` on branch `main` of `github.com/acme/platform`, published by GitOps +Reverser and synced by the `Application` `argocd/checkout`. Every object below is checked by a unit +test against the code that validates and drives the applier (see +[what the tests prove](#what-the-tests-prove)), so this example cannot drift from what is accepted. + +The Argo CD side. The `Application` follows exactly the branch the `GitTarget` writes, and its +automated sync is switched off while keeping `prune` for the syncs GitOps Reverser starts: + +```yaml +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: checkout + namespace: argocd +spec: + project: default + source: + repoURL: https://github.com/acme/platform + targetRevision: main + path: apps/checkout + directory: + recurse: true + destination: + server: https://kubernetes.default.svc + namespace: checkout + syncPolicy: + automated: + enabled: false + prune: true + syncOptions: + - ServerSideApply=true +``` + +The GitOps Reverser side. The `GitProvider` points at the same repository (any URL form of it) and +serves a push receiver; the `GitTarget` names the `Application` as its applier: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: platform-receiver + namespace: team-a +stringData: + token: replace-with-a-long-random-string +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitProvider +metadata: + name: platform + namespace: team-a +spec: + url: git@github.com:acme/platform.git + allowedBranches: ["main"] + secretRef: + name: platform-git + receiver: + type: github + secretRef: + name: platform-receiver +--- +apiVersion: configbutler.ai/v1alpha3 +kind: GitTarget +metadata: + name: checkout + namespace: team-a +spec: + gitProviderRef: + name: platform + branch: main + path: apps/checkout + applier: + applicationRef: + namespace: argocd + name: checkout + onRefusal: Reapply +``` + +`directory.recurse: true` matters: GitOps Reverser writes `////.yaml`, +so a non-recursive directory source sees nothing. + +## The `Application` + +`ApplierConfigured` is `True` only when the `Application` syncs the branch the `GitTarget` writes, +from a single source, with automated sync off. Each rule is checked when the target reconciles and +again before every sync, so an `Application` changed after the check is still never synced against +the rules. + +| Rule | Reported when broken | +| --- | --- | +| `spec.source.repoURL` is the `GitProvider`'s repository. The forms compare as one repository: `https://` and `ssh://`, `git@host:org/repo` (SCP style), a trailing `.git` or `/`, letter case, and the default port | `SourceMismatch`: `syncs , not ` | +| `spec.source.targetRevision` is the `GitTarget`'s `spec.branch`, as the name or as `refs/heads/`. `HEAD` (and an empty `targetRevision`, which means `HEAD`), a tag and a commit never follow the branch | `SourceMismatch`: `targets revision "HEAD", not branch main` | +| A single `spec.source`: no `spec.sources`, no `spec.sourceHydrator` | `SourceMismatch`: `syncs only a single-source Application` | +| Automated sync is off: `spec.syncPolicy.automated` is absent, or has `enabled: false`. This is Argo CD's own rule; an `automated` block with no `enabled` is on | `AutomatedSyncEnabled` | + +**Prune.** A sync GitOps Reverser starts prunes when `spec.syncPolicy.automated.prune` is `true`, +read from the `automated` block that `enabled: false` switches off, exactly as automated sync would +read it. Without an `automated` block, its syncs do not prune: a file removed from Git leaves its +object running. Keep the block with `enabled: false` and `prune: true` when a deletion in Git should +delete the live object. + +`selfHeal` means nothing with automated sync off, and is ignored. `syncOptions` and `retry` apply to +GitOps Reverser's syncs as they would to an automated one. + +## The `GitTarget` + +```yaml +spec: + applier: + applicationRef: + namespace: argocd # default: this GitTarget's namespace + name: checkout + clusterProviderRef: # default: this GitTarget's spec.clusterProviderRef + name: management +``` + +`clusterProviderRef` is for a hub Argo CD, which keeps its `Application`s in a management cluster and +deploys to another. Name the `ClusterProvider` of the cluster the `Application` lives in; the +`GitTarget`'s own `spec.clusterProviderRef` stays the cluster it mirrors. Leave it out when Argo CD +runs in the source cluster. + +`spec.applier` names exactly one of `kustomizationRef` and `applicationRef`, and `quietPeriod` is +rejected with an `applicationRef`. + +The target reports `ApplierConfigured`, and `Ready` includes it: + +| Reason | Means | Readiness | +| --- | --- | --- | +| `Succeeded` | The `Application` exists, syncs this repository and branch, has automated sync off, and may be updated | ready | +| `ApplierNotFound` | No such `Application` in the applier's cluster, or Argo CD is not installed there | progressing: `Application`s are often created after the `GitTarget` | +| `SourceMismatch` | One of the [`Application` rules](#the-application) about its source is broken | stalled: needs a human | +| `AutomatedSyncEnabled` | Automated sync is on | stalled: needs a human | +| `Forbidden` | The applier cluster's identity may not update the `Application` | stalled: needs a human | +| `NamespaceNotAuthorized`, `ClusterProviderNotFound` | `spec.applier.clusterProviderRef` names a provider that does not admit this namespace, or does not exist | stalled: needs a human | + +## Permissions + +GitOps Reverser looks the `Application` up in the cluster of `spec.applier.clusterProviderRef`, by +default the **source cluster**, with the identity of that `ClusterProvider`. That identity needs: + +| API group | Resource | Verbs | +| --- | --- | --- | +| `argoproj.io` | `applications` | `get`, `update` | +| `authorization.k8s.io` | `selfsubjectaccessreviews` | `create`, to check the one above | + +A sync is started by an `update` of the `Application`, the way Argo CD's own automated sync starts +one. For the operator's own cluster, install the chart with `applierAccess.argocd: true`, or apply +[`config/rbac/argocd-applier-role.yaml`](../config/rbac/argocd-applier-role.yaml) and its binding. +For a remote cluster, grant the same to the identity in the `ClusterProvider`'s kubeconfig. A `Role` +in the `Application`'s namespace is enough when you would rather not grant cluster-wide. + +Argo CD's own RBAC does not apply: GitOps Reverser writes the `Application` through the Kubernetes +API, as `kubectl` would, not through `argocd-server`. + +## The Git host's webhook + +Point the Git host's push webhook at GitOps Reverser's receiver, so a Git-side push is synced in +seconds, after GitOps Reverser has replayed any pending edits onto it. The receiver is the same as +for Flux: [the Flux guide](flux.md#the-git-hosts-webhook) has the steps. + +Argo CD's own webhook into `/api/webhook` may stay. It only makes Argo CD refresh sooner, and with +automated sync off a refresh only compares. Without GitOps Reverser's receiver, a foreign push is +still found, later: by the next rejected push or by the periodic refresh (`git.refreshInterval`, +`10m`). + +## What GitOps Reverser asks, and when + +| Reason | When | Asked | +| --- | --- | --- | +| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Pending edits are replayed onto it and pushed first | Every applier declared on the branch | +| `Refusal` | An edit was refused and the target has `onRefusal: Reapply` | The refusing target's `Application` | + +There is no `CatchUp` for an `Application` ([why](#why-automated-sync-must-be-off)), and every +request starts a sync: there is no "already current" for Argo CD. + +**A request sets the `Application`'s `operation` to the sync automated sync would start**, with +these differences: + +- `sync.revision` is **pinned to the commit GitOps Reverser checked**, so the sync applies exactly + that commit, not whatever the branch holds by then. +- `initiatedBy` is `{username: gitops-reverser, automated: true}`. Argo CD's UI and history show + the sync as GitOps Reverser's, and a [sync window](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/) + that blocks automated syncs blocks it too. +- `info` carries the request's reason, and a token that tells this sync's result from any other. +- A `retry` with `refresh: true` is copied without it. Argo CD's refresh-on-retry drops the pinned + revision and syncs the newest one, a commit GitOps Reverser never checked. + +`source`, `prune`, `syncOptions` and `retry` (`limit: 5` when the `Application` sets none) are taken +from the `Application`, as automated sync takes them. + +**A sync already running is never replaced.** Argo CD refuses a new operation while one is set, and +so does GitOps Reverser: it waits for it to end (at most 5 minutes), then reads the `Application` +again and decides from what it is then. The update carries the `resourceVersion` it read, so a sync +somebody starts in between makes it fail instead of being overwritten. A failed sync waiting for +Argo CD's retry still counts as running. + +Before it writes, GitOps Reverser checks that its decision still holds: the commit is still the +branch tip, and no edit is waiting to be published. When either changed, it writes nothing and +decides again at the next quiet moment. + +**Hooks run on every sync**, as they would on an automated one: every `ForeignCommit` asks every +applier on the branch, whether or not the push touched its folder. + +**An edit made while Argo CD syncs keeps its value.** Until the sync's operation completes +(`Succeeded`, `Failed` or `Error`, at most 5 minutes), GitOps Reverser treats it as running. An API +edit made in those seconds, to a field the `Application` sets, is reverted on the cluster by the +sync, because the commit it applies does not hold the edit yet. GitOps Reverser recognizes that +write by Argo CD's field manager (`argocd-controller`, for a client-side and a server-side apply +alike, or `argocd-application-controller` for a `Replace=true` sync) and does not mirror it: the +API's value stays in Git. Each such write is logged and counted in +`apply_window_collisions_total{outcome="withheld"}`. Unlike Flux, nothing applies the edit back by +itself, since there is no catch-up: the cluster runs the reverted value until the next sync. + +Only a write proven to be Argo CD's is withheld. When Argo CD and another manager wrote the object +in the same second, the write could be the edit itself, so it is mirrored as before and counted as +`ambiguous`. A delete is never withheld. + +The request runs off the branch worker, so a slow Argo CD never holds a publication. A request that +fails is decided again from scratch, after that `Application`'s own backoff, exactly as for +[Flux](flux.md#what-gitops-reverser-asks-and-when). + +## Reverting refused edits + +With `onRefusal: Reapply`, an edit GitOps Reverser cannot write (one that has no home in the folder, +such as a field a kustomize base owns) is reverted by asking the target's `Application` to sync the +tip, once the branch is quiet. Only that `Application` is asked. +[Reverting a refused edit](configuration.md#reverting-a-refused-edit-speconrefusal) has the +eligibility rules. + +## Checking that it works + +```bash +# The Application is found, syncs this repository and branch, has automated sync off, and may be updated. +kubectl get gittarget checkout -n team-a \ + -o jsonpath='{.status.conditions[?(@.type=="ApplierConfigured")]}{"\n"}' + +# What GitOps Reverser last asked for, and why. +kubectl get gittarget checkout -n team-a -o jsonpath='{.status.applier}{"\n"}' + +# The last sync: who started it, which commit, and how it ended. A request asks, it never reports. +kubectl get application checkout -n argocd \ + -o jsonpath='{.status.operationState.operation.initiatedBy}{" "}{.status.operationState.syncResult.revision}{" "}{.status.operationState.phase}{"\n"}' +# {"automated":true,"username":"gitops-reverser"} 4f2c1ab9... Succeeded +``` + +`gitopsreverser_apply_requests_total{applier_kind="application",reason,outcome}` counts the +requests; [interpreting metrics](interpreting-metrics.md#apply-requests-and-the-push-receiver) reads +it. + +## Troubleshooting + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `ApplierConfigured=False`, `AutomatedSyncEnabled` | `spec.syncPolicy.automated` is present without `enabled: false` | Set `automated.enabled: false` (keep `prune`), or remove the block | +| `ApplierConfigured=False`, `ApplierNotFound` | The `Application` does not exist in the applier's cluster under that namespace and name, or Argo CD is not installed there | Create it, or fix `applicationRef`; an omitted namespace is the `GitTarget`'s | +| `ApplierConfigured=False`, `SourceMismatch` | The message names the broken [`Application` rule](#the-application) | Fix the `Application`; a request it was holding is delivered afterward | +| `ApplierConfigured=False`, `Forbidden` | The identity lacks one of the [permissions](#permissions) | Grant it, or set `applierAccess.argocd: true` | +| A Warning `ApplyRequestFailed` Event | A request could not be delivered; the message says why, such as a sync somebody else started that did not end within 5 minutes | GitOps Reverser keeps asking on a quiet branch, at most every five minutes | +| A file deleted in Git leaves its object running | The `Application` has no `automated.prune: true` | Add `automated: {enabled: false, prune: true}` | +| A sync is skipped | A sync window denies automated syncs now | Wait for the window, or allow automated syncs in it | + +## While GitOps Reverser is down + +GitOps Reverser becomes the only path to a deployment. While it is down, a Git-side push reaches +the cluster through a manual sync (`argocd app sync`, or the UI), which is safe then because nothing +is publishing. When it starts again, its first look at the remote owes no `ForeignCommit`: a sync on +startup could revert edits made while it was down, before its watches have caught them. + +## What the tests prove + +| Covered | Where | +| --- | --- | +| The [complete example](#a-complete-example) decodes strictly, reports `ApplierConfigured=True`, and its request sets a sync pinned to the commit | [`argocd_guide_test.go`](../internal/applier/argocd_guide_test.go) | +| The operation is the one automated sync builds, but for the pinned revision, the initiator and a pinned retry; a running sync is waited for and never replaced; a sync started between the read and the update makes it fail; every `ApplierConfigured` reason; when a sync counts as finished | [`argocd_test.go`](../internal/applier/argocd_test.go) | +| An `Application` is never owed a `CatchUp`, and is asked on a `ForeignCommit` and a `Refusal`; Argo CD's late write over a newer edit is withheld, by its own field managers | [`apply_trigger_test.go`](../internal/git/apply_trigger_test.go), [`apply_window_test.go`](../internal/git/apply_window_test.go) | +| Two API edits seconds apart keep the later value, with no further commit | [Bi-directional (Argo CD)](../test/e2e/argocd_bi_directional_e2e_test.go) | +| A Git-side push is synced by a `ForeignCommit` request, pinned to that commit | Bi-directional (Argo CD) | +| A refused edit with `onRefusal: Reapply` is reverted by an Argo CD sync | Bi-directional (Argo CD) | +| An `Application` with automated sync on is `AutomatedSyncEnabled` and is never synced | Bi-directional (Argo CD) | +| Argo CD's bookkeeping never reaches Git, asserted as zero commits during a sync | Bi-directional (Argo CD) | + +`task test-e2e-bi-directional` runs the Argo CD specs. Not covered end to end yet: a hub Argo CD in +another cluster, `Forbidden` against cluster RBAC, and a `Replace=true` sync's field manager, which +is read from Argo CD's source rather than measured. diff --git a/docs/bi-directional.md b/docs/bi-directional.md index 60b04e9f..77133850 100644 --- a/docs/bi-directional.md +++ b/docs/bi-directional.md @@ -4,20 +4,18 @@ GitOps Reverser publishes live cluster state to Git. Flux or Argo CD applies Git cluster. Run both and a resource can be changed from either side. This guide covers who decides what, how to configure each actor, and which parts are proven by tests. -**The short version.** GitOps Reverser decides when the cluster is applied from Git. Flux keeps -doing what it is good at (fetching, rendering, applying, pruning, reporting health), but it applies -when GitOps Reverser asks, not on its own schedule. You configure that in three places: - -1. Name each folder's **applier** on its `GitTarget`: the Flux `Kustomization` that applies it. See - [declaring the applier](#declaring-the-applier). -2. Switch off Flux's own triggers: short intervals and any Git-host webhook into a Flux `Receiver`. - See [configure Flux](#configure-flux). -3. Point your Git host's push webhook at GitOps Reverser instead. See +**The short version.** GitOps Reverser decides when the cluster is applied from Git. Flux or Argo +CD keeps doing what it is good at (fetching, rendering, applying, pruning, reporting health), but it +applies when GitOps Reverser asks, not on its own schedule. You configure that in three places: + +1. Name each folder's **applier** on its `GitTarget`: the Flux `Kustomization` or the Argo CD + `Application` that applies it. See [declaring the applier](#declaring-the-applier). +2. Switch off the applier's own triggers: for Flux, short intervals and any Git-host webhook into a + Flux `Receiver` ([configure Flux](#configure-flux)); for Argo CD, automated sync + ([configure Argo CD](#configure-argo-cd)). +3. Point your Git host's push webhook at GitOps Reverser. See [telling GitOps Reverser that somebody else pushed](#telling-gitops-reverser-that-somebody-else-pushed). -Argo CD cannot be driven this way yet; [configure Argo CD](#configure-argo-cd) says what that costs -in the meantime. - There is no merge and no lock. What remains exposed is small and named in [what is still exposed](#what-is-still-exposed). @@ -26,7 +24,7 @@ flowchart LR subgraph CLUSTER["Kubernetes cluster"] API["Kubernetes API"] REV["GitOps Reverser"] - APP["Applier
Flux Kustomization"] + APP["Applier
Flux Kustomization
or Argo CD Application"] end GIT[("Git branch")] HUMAN(["Somebody pushes to Git"]) @@ -49,7 +47,7 @@ flowchart LR | Kubernetes API | Takes edits from people and controllers | | | GitOps Reverser | Mirrors every edit into Git, and decides when the applier may apply | Apply anything to the cluster itself | | Git host | Stores the branch, and tells GitOps Reverser when somebody else pushed | Notify the applier | -| Applier (a Flux `Kustomization`) | Fetches, renders, applies and prunes, when asked | Apply on a short timer or a webhook of its own | +| Applier (a Flux `Kustomization` or an Argo CD `Application`) | Fetches, renders, applies and prunes, when asked | Apply on a short timer, a webhook or an automated sync of its own | | A human | Can still run `flux reconcile` or press Sync | | ## Why GitOps Reverser owns the trigger @@ -60,7 +58,8 @@ or in the pending writes. An apply at that moment writes Git's older value over would then mirror that revert faithfully, as a commit authored by the applier, and the edit would be gone. (An edit that lands while an apply GitOps Reverser asked for is running keeps its value in Git: the applier's write over it is not mirrored. See -[the Flux guide](flux.md#what-gitops-reverser-asks-and-when).) +[the Flux guide](flux.md#what-gitops-reverser-asks-and-when) and +[the Argo CD guide](argocd.md#what-gitops-reverser-asks-and-when).) Only GitOps Reverser can see those pending edits. The applier cannot. So an apply nobody asked GitOps Reverser about is an apply nobody checked, and every trigger the applier runs by itself is @@ -89,11 +88,12 @@ branch worker. That is the moment nothing live is missing from the tip. | --- | --- | --- | | `ForeignCommit` | The branch gained a commit GitOps Reverser did not push | Every applier declared on the branch | | `Refusal` | An edit was refused and the target has `onRefusal: Reapply` | The refusing target's applier | -| `CatchUp` | The branch has been quiet for `spec.applier.quietPeriod` after GitOps Reverser's own pushes | Every applier on the branch | +| `CatchUp` | The branch has been quiet for `spec.applier.quietPeriod` after GitOps Reverser's own pushes | Every Flux `Kustomization` on the branch | The cluster already holds the values of GitOps Reverser's own commits, because they were read from it, so applying them changes nothing. Flux is asked to catch up with them anyway, and -[configure Flux](#configure-flux) says why. +[configure Flux](#configure-flux) says why. Argo CD is not: with automated sync off it applies +nothing by itself, so nothing it holds goes stale. A foreign commit that arrives while edits are pending does not wait for their windows to close. GitOps Reverser closes them, replays the pending writes onto the foreign commit, pushes, and asks @@ -128,7 +128,8 @@ it applied. ## Declaring the applier -`spec.applier` names exactly one applier per `GitTarget`: +`spec.applier` names exactly one applier per `GitTarget`, a Flux `Kustomization` or an Argo CD +`Application`: ```yaml apiVersion: configbutler.ai/v1alpha3 @@ -148,48 +149,58 @@ spec: onRefusal: Reapply ``` +```yaml +spec: + applier: + applicationRef: + namespace: argocd + name: checkout +``` + GitOps Reverser looks the applier up in the **source cluster**, the cluster it applies to, with -the identity the `ClusterProvider` already uses. That identity needs permission to read and patch -the applier; the chart grants it for the operator's own cluster with `applierAccess.flux: true`. +the identity the `ClusterProvider` already uses, or in the cluster `spec.applier.clusterProviderRef` +names. That identity needs permission to read and write the applier; the chart grants it for the +operator's own cluster with `applierAccess.flux: true` or `applierAccess.argocd: true`. The applier must apply this folder's repository and branch, or the target reports `ApplierConfigured=False`. The [configuration reference](configuration.md#who-applies-this-folder-specapplier) has every field and reason. -Several `GitTarget`s may name the same `Kustomization`, for example one whose path covers several -folders. Requests to one `Kustomization` are sent once. +Several `GitTarget`s may name the same applier, for example a `Kustomization` whose path covers +several folders. Requests to one applier are sent once. ## Configure Argo CD -GitOps Reverser cannot drive an Argo CD `Application` yet; `spec.applier` names a Flux -`Kustomization` only, and Argo CD is next. Until then an `Application` keeps its own triggers, with -the exposure [why GitOps Reverser owns the trigger](#why-gitops-reverser-owns-the-trigger) -describes: the timed refresh syncs each of GitOps Reverser's pushes two to three minutes later, over -whatever is pending at that moment. - -The configuration that keeps the most of a live edit in the meantime: +[Using GitOps Reverser with Argo CD](argocd.md) is the complete setup, with a worked example a unit +test holds to the code. The standard is full control: the `Application` named by +`spec.applier.applicationRef` has **automated sync off**, and every sync it runs is one GitOps +Reverser started, pinned to a commit it checked. ```yaml spec: syncPolicy: automated: - selfHeal: false - prune: true + enabled: false # automated sync off; GitOps Reverser starts every sync + prune: true # kept for the syncs GitOps Reverser starts ``` -With `selfHeal: true` Argo CD reverts a live edit in about a second, usually before GitOps Reverser -can publish it, and Git records a two-commit flap. `selfHeal: false` only suppresses repeating a -sync of a commit already synced, so a new commit or a manual sync still applies Git over an -unpublished API edit. Use `prune: true` only when a deletion in Git should delete the live object, -and point the Git host's webhook at `https:///api/webhook` so a Git-side push is -synced in seconds rather than on the timer. +An `Application` with automated sync on is `ApplierConfigured=False` with `AutomatedSyncEnabled`, +and GitOps Reverser never syncs it. `selfHeal` means nothing with automated sync off. Argo CD's +timed refresh and its `/api/webhook` may stay: with automated sync off they only compare. A sync +GitOps Reverser starts is the one automated sync would (`prune`, `syncOptions` and `retry` from the +`Application`), with the revision pinned and `initiatedBy` set to `gitops-reverser`; it waits for a +sync already running rather than replace it. + +Without `spec.applier`, an `Application` with automated sync keeps its own triggers, with the +exposure [why GitOps Reverser owns the trigger](#why-gitops-reverser-owns-the-trigger) describes: +each of GitOps Reverser's pushes is synced over whatever is pending at that moment, and with +`selfHeal: true` a live edit is reverted in about a second, before GitOps Reverser can publish it. ### The split-ownership alternative -`ignoreDifferences` on a field, with `RespectIgnoreDifferences=true` in `syncOptions`, keeps -`selfHeal: true` everywhere else and hands that one field to the API. Its differences alone do not -make the `Application` `OutOfSync`, and the option copies the live value into the apply target, so an -ordinary sync preserves it. +`ignoreDifferences` on a field, with `RespectIgnoreDifferences=true` in `syncOptions`, hands that +one field to the API. Its differences alone do not make the `Application` `OutOfSync`, and the +option copies the live value into the apply target, so an ordinary sync preserves it. The cost is that the field stops being Git-driven on an existing object: Git seeds it at creation, but later Git edits to it do not reach the object while the ignore is active. It protects no other @@ -348,6 +359,7 @@ the seconds an apply takes: `quietPeriod` of silence on a Flux branch. It no longer follows every commit GitOps Reverser makes. - **Flux's backstop timers** still apply on their own, at the tip as the last catch-up left it. + Argo CD with automated sync off has none. - **A human** running `flux reconcile` or pressing Sync applies whenever they do. ## Why the loop stops @@ -402,13 +414,13 @@ until something applies Git over it. | `Ignore` (default) | Report the refusal | Leave recovery to a human or a later apply | | `Reapply` | Ask the target's applier to apply the current tip, once the branch is quiet | The refused edit is reverted; nothing still waiting to be published is | -`Reapply` needs a declared applier, and the request goes to that applier alone. When the applier's -`GitRepository` has to fetch to reach the tip, though, every `Kustomization` on that -`GitRepository` applies the new revision, exactly as it would for any push: moving the source is -the only way to apply the tip, and Flux wakes everything that reads it. Give the folder its own -`GitRepository` if the apply must reach that folder alone. Waiting for a quiet branch means an -allowed edit still in a commit window is published first, so the apply reverts only what has no -home in Git. +`Reapply` needs a declared applier, and the request goes to that applier alone: an `Application` +syncs the tip. When a `Kustomization`'s `GitRepository` has to fetch to reach the tip, though, every +`Kustomization` on that `GitRepository` applies the new revision, exactly as it would for any push: +moving the source is the only way to apply the tip, and Flux wakes everything that reads it. Give +the folder its own `GitRepository` if the apply must reach that folder alone. Waiting for a quiet +branch means an allowed edit still in a commit window is published first, so the apply reverts only +what has no home in Git. Eligibility is narrow: the folder must be accepted, the refused edit must concern a document the folder already holds, and the write must remove no document, because re-applying would put back an @@ -458,7 +470,10 @@ concurrent writers. | A SOPS-encrypted Secret round-trips without a reverse commit | Flux | | A Git revert removes the live object and the committed file | Flux | | Argo CD bookkeeping never reaches Git, asserted as zero commits during a sync | [Argo CD](../test/e2e/argocd_bi_directional_e2e_test.go) | -| `selfHeal: true` loses the API edit, and the flap is exactly two commits; `selfHeal: false` keeps it | Argo CD | +| Two API edits to one field seconds apart keep the later value with no further commit, and Argo CD is never asked to catch up | Argo CD | +| A Git-side push is synced by a `ForeignCommit` request, pinned to that commit | Argo CD | +| A refused edit with `onRefusal: Reapply` is reverted by an Argo CD sync | Argo CD | +| An `Application` with automated sync on is `AutomatedSyncEnabled` and is never synced | Argo CD | **No spec tests simultaneous writes to one field from both sides**, because the outcome depends on arrival order the corner does not control. The replay mechanism is covered; a conflict policy is diff --git a/docs/configuration.md b/docs/configuration.md index e37e4767..4a66b2a7 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -32,9 +32,9 @@ The usual flow is: 3. Create one or more `WatchRule` or `ClusterWatchRule` objects that point at that target. 4. Create a `CommitRequest` only when you want a save: your writes in one commit, on the request's timers rather than the target's, and optionally with your message. -5. When Flux also applies the folder back to the cluster, name its `Kustomization` on the - `GitTarget` and give the `GitProvider` a push receiver, so GitOps Reverser decides when Git is - applied. See [who applies this folder](#who-applies-this-folder-specapplier) and the +5. When Flux or Argo CD also applies the folder back to the cluster, name its `Kustomization` or + `Application` on the `GitTarget` and give the `GitProvider` a push receiver, so GitOps Reverser + decides when Git is applied. See [who applies this folder](#who-applies-this-folder-specapplier) and the [bi-directional guide](bi-directional.md). That means one repository connection can back multiple targets, and one target can be fed by @@ -97,7 +97,7 @@ The destination fields are immutable: to move a target, delete it and create a n | `placement.useKustomize` | off | Keep the folder a kustomize folder. See [keeping the folder a kustomize folder](#keeping-the-folder-a-kustomize-folder-specplacementusekustomize) | | `serializeNamespace` | inferred per document | Whether a committed document carries its own `metadata.namespace`. An explicit `false` admits exactly one source namespace. See [whether documents carry their namespace](#whether-documents-carry-their-namespace-specserializenamespace) | | `prune.mode` | `OnEvent` | Which deletions reach Git. See [deletion policy](#deletion-policy-specprunemode) | -| `applier` | Flux applies on its own schedule | The Flux `Kustomization` that applies this folder (`kustomizationRef`), which GitOps Reverser then asks to apply, and how long the branch stays quiet before it catches up (`quietPeriod`, `30s`). See [who applies this folder](#who-applies-this-folder-specapplier) | +| `applier` | the applier applies on its own schedule | The Flux `Kustomization` (`kustomizationRef`) or Argo CD `Application` (`applicationRef`) that applies this folder, which GitOps Reverser then asks to apply, and for a `Kustomization` how long the branch stays quiet before it catches up (`quietPeriod`, `30s`). See [who applies this folder](#who-applies-this-folder-specapplier) | | `onRefusal` | `Ignore` | What happens to an edit with no legal destination. See [reverting a refused edit](#reverting-a-refused-edit-speconrefusal) | | `encryption` | off | SOPS + age encryption for sensitive resources | | `suspend` | `false` | Stop writing without deleting the target. See [stopping a target from writing](#stopping-a-target-from-writing-specsuspend) | @@ -634,8 +634,8 @@ The important fields are: omit it and each document's namespace is inferred from the folder - `spec.prune`: which deletion paths may remove documents from this target's folder (see [Deletion policy](#deletion-policy-specprunemode)); omit it for the safe default -- `spec.applier`: the Flux `Kustomization` that applies this folder, so GitOps Reverser decides when - it applies (see +- `spec.applier`: the Flux `Kustomization` or Argo CD `Application` that applies this folder, so + GitOps Reverser decides when it applies (see [Who applies this folder](#who-applies-this-folder-specapplier)); omit it when nothing applies the folder back - `spec.onRefusal`: whether a refused edit asks the applier to reapply Git (see @@ -721,7 +721,7 @@ summary reason, while the domain conditions name the axis to fix: | `ClusterProviderReady=False` or `SourceClusterReachable=False` | Source-cluster credentials and access. | | `GitPathAccepted=False` | Repository files under `spec.path`; the message includes the issue kind, path, and actor when known. | | `RenderMatchesLive=False` | Fields that no longer match the rendered source, often transformer-owned values. | -| `ApplierConfigured=False` | The applier named in `spec.applier`, its source, and the permission to patch it. Commits still flow; apply requests may not. | +| `ApplierConfigured=False` | The applier named in `spec.applier`, its source, its automated sync (an `Application`), and the permission to write it. Commits still flow; apply requests may not. | A green `GitTarget` with a red `WatchRule` or `ClusterWatchRule` means the destination may be fine, but the selected source objects are not being watched or authorized. Inspect the rule's `Ready`, @@ -1105,7 +1105,8 @@ When a GitOps tool applies this folder back to the cluster, `spec.applier` names does it, and GitOps Reverser becomes the one that decides when it applies. The tool keeps doing the fetching, rendering, applying and pruning. The reason it has to be this way is the [apply trigger design](design/apply-trigger.md); the [bi-directional guide](bi-directional.md) is -the model, and [Using GitOps Reverser with Flux](flux.md) the whole Flux setup. +the model, [Using GitOps Reverser with Flux](flux.md) the whole Flux setup, and +[Using GitOps Reverser with Argo CD](argocd.md) the whole Argo CD setup. ```yaml spec: @@ -1116,26 +1117,35 @@ spec: quietPeriod: 30s ``` +```yaml +spec: + applier: + applicationRef: + namespace: argocd # default: this GitTarget's namespace + name: checkout +``` + | Field | Default | What it does | | --- | --- | --- | -| `kustomizationRef` | **required** | A Flux `Kustomization` whose `sourceRef` is a `GitRepository` on this target's repository and branch | -| `quietPeriod` | `30s` | How long the branch must stay quiet after GitOps Reverser's own pushes before a `CatchUp` request. At most `1h` | +| `kustomizationRef` | one of the two | A Flux `Kustomization` whose `sourceRef` is a `GitRepository` on this target's repository and branch | +| `applicationRef` | one of the two | An Argo CD `Application` on a single source, with this target's repository and branch, and automated sync off | +| `quietPeriod` | `30s` | How long the branch must stay quiet after GitOps Reverser's own pushes before a `CatchUp` request. At most `1h`. Only with a `kustomizationRef`: rejected with an `applicationRef` | -**The appliers.** A Flux `Kustomization` (`kustomizationRef`) is built. Two more are the aim, and -the API does not accept them yet: +**The appliers.** A Flux `Kustomization` (`kustomizationRef`) and an Argo CD `Application` +(`applicationRef`); exactly one of the two. The `Application` runs with automated sync off +(`syncPolicy.automated.enabled: false`, or no `automated` block), and every sync it runs is one +GitOps Reverser starts, pinned to the commit it checked. It is never asked to catch up with GitOps +Reverser's own commits, since with automated sync off it applies nothing by itself. One with +automated sync on is `ApplierConfigured=False` with `AutomatedSyncEnabled`, and is never synced. -- **An Argo CD `Application` (`applicationRef`).** The `Application` runs with - `syncPolicy.automated.enabled: false`, and a request sets its sync operation exactly as automated - sync would, pinned to the commit GitOps Reverser checked. It is never asked to catch up with - GitOps Reverser's own commits, so `quietPeriod` will not apply to it. Until it lands, an - `Application` keeps its own triggers. -- **A webhook (`webhook`)** for any other tool: a POST of `{"repository", "branch", "commit", - "reason"}`, signed the way Flux's `generic-hmac` `Receiver` checks. +A third is the aim, and the API does not accept it yet: **a webhook (`webhook`)** for any other tool, +a POST of `{"repository", "branch", "commit", "reason"}`, signed the way Flux's `generic-hmac` +`Receiver` checks. **Where the reference is looked up.** In the cluster `spec.applier.clusterProviderRef` names, with the identity that [`ClusterProvider`](#clusterprovider) uses. Omitted, it is the `GitTarget`'s own -`spec.clusterProviderRef`, the source cluster. Name another provider when Flux runs in a management -cluster and applies to the source cluster from there; a provider without `spec.kubeConfig` is the +`spec.clusterProviderRef`, the source cluster. Name another provider when Flux or Argo CD runs in a +management cluster and applies to the source cluster from there; a provider without `spec.kubeConfig` is the operator's own cluster. That provider's `spec.accessFrom` must admit the `GitTarget`'s namespace, as for mirroring, or `ApplierConfigured` is `False` with `NamespaceNotAuthorized` (or `ClusterProviderNotFound`) and nothing is asked. @@ -1152,10 +1162,12 @@ spec: name: prod-eu-1-apps ``` -The identity needs `get` and `patch` on `kustomizations` (`kustomize.toolkit.fluxcd.io`) and -`gitrepositories` (`source.toolkit.fluxcd.io`), and `create` on `selfsubjectaccessreviews` to check -it. For the operator's own cluster the chart grants them with `applierAccess.flux: true`; for a -remote cluster, grant the same to the kubeconfig's identity. +For a `Kustomization`, the identity needs `get` and `patch` on `kustomizations` +(`kustomize.toolkit.fluxcd.io`) and `gitrepositories` (`source.toolkit.fluxcd.io`); for an +`Application`, `get` and `update` on `applications` (`argoproj.io`); and `create` on +`selfsubjectaccessreviews` to check it. For the operator's own cluster the chart grants them with +`applierAccess.flux: true` and `applierAccess.argocd: true`; for a remote cluster, grant the same to +the kubeconfig's identity. **When a request is made.** Only on a quiet branch: no open commit window and no pending writes on the branch worker, or on any other worker writing the same repository and branch through another @@ -1163,9 +1175,9 @@ the branch worker, or on any other worker writing the same repository and branch | `reason` | When | Sent to | | --- | --- | --- | -| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Open windows are closed and pending writes replayed and pushed first, so the request names that pushed commit | Every applier declared on the branch, once per `Kustomization` | +| `ForeignCommit` | The branch gained a commit GitOps Reverser did not push. Open windows are closed and pending writes replayed and pushed first, so the request names that pushed commit | Every applier declared on the branch, once per applier | | `Refusal` | An eligible refused edit, with `spec.onRefusal: Reapply` | This target's applier | -| `CatchUp` | The branch has been quiet for `quietPeriod` since GitOps Reverser's own pushes, or since the operator started | Every applier declared on the branch | +| `CatchUp` | The branch has been quiet for `quietPeriod` since GitOps Reverser's own pushes, or since the operator started | Every `Kustomization` declared on the branch; never an `Application` | Flux applies on its own timers and on a controller restart, from whatever artifact the `GitRepository` holds. `CatchUp` keeps that artifact at the tip, so such an apply applies values the @@ -1174,7 +1186,7 @@ cluster already holds; an artifact left behind would revert edits that are alrea `GitTarget`s name one `Kustomization`, it waits for the longest of their quiet periods, since its apply reaches all their folders, and it is asked once for all of them. -**What a request does.** What `flux reconcile kustomization --with-source` does: stamp +**What a request does to a `Kustomization`.** What `flux reconcile kustomization --with-source` does: stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository` when its artifact does not hold the requested commit; the new artifact wakes the `Kustomization` by itself. When the artifact already holds the commit, only a `Refusal` stamps the `Kustomization`, to revert drift at an unchanged @@ -1182,21 +1194,27 @@ commit; a `ForeignCommit` or `CatchUp` stamps nothing, since that artifact alrea is stamped when, by the time of the patch, the commit is no longer the branch tip or an edit is waiting to be published: the request is decided again at the next quiet moment. +**What a request does to an `Application`.** It sets the `Application`'s `operation` to the sync +Argo CD's automated sync would start (`prune` from `syncPolicy.automated`, `syncOptions` and `retry` +from `syncPolicy`), with `revision` pinned to the requested commit and `initiatedBy` +`{username: gitops-reverser, automated: true}`. It never replaces an operation already running: it +waits for it to end, at most 5 minutes. The details are in [the Argo CD guide](argocd.md#what-gitops-reverser-asks-and-when). + A request asks; it never claims the apply happened. It runs off the publication path, and a failed request never fails a publication that succeeded. Read the `Kustomization`'s -`status.lastAppliedRevision` for what it applied. +`status.lastAppliedRevision`, or the `Application`'s `status.operationState`, for what it applied. **Every attempt checks again.** Just before it patches, GitOps Reverser reads each `GitTarget` again and asks nothing for one that has been deleted or recreated, no longer names this applier on this branch, is no longer admitted by its `ClusterProvider`, or, for a `Refusal`, no longer consents (`onRefusal: Reapply`, not suspended). It never asks for something weaker in place of what it was -told; when no target still holds the request's reason it asks nothing. And nothing is stamped unless -the `Kustomization`'s `GitRepository` fetches this repository and branch, whatever -`ApplierConfigured` last said. +told; when no target still holds the request's reason it asks nothing. And nothing is asked unless +the `Kustomization`'s `GitRepository` fetches this repository and branch, or the `Application` syncs +it with automated sync off, whatever `ApplierConfigured` last said. **A request that fails is decided again from scratch.** The branch worker keeps what the applier is owed until a request delivers it. After a failure it waits out that applier's own backoff -(2 seconds doubling to 5 minutes, at least 15 seconds when the `Kustomization` cannot take a +(2 seconds doubling to 5 minutes, at least 15 seconds when the applier cannot take a request), then waits for a quiet branch like any first request, and asks for the tip as it is then. A refusal that recovered in the meantime is not asked for again. @@ -1219,14 +1237,15 @@ status: | Reason | Means | | --- | --- | -| `ApplierNotFound` | No such `Kustomization` in the applier's cluster, or Flux is not installed there | -| `SourceMismatch` | The `Kustomization` applies a different repository or branch, or a source that is not a `GitRepository` | -| `Forbidden` | The applier cluster's identity may not patch the `Kustomization` or its `GitRepository` | +| `ApplierNotFound` | No such `Kustomization` or `Application` in the applier's cluster, or Flux or Argo CD is not installed there | +| `SourceMismatch` | The `Kustomization` applies a different repository or branch, or a source that is not a `GitRepository`; or the `Application` syncs a different repository or revision, or more than one source | +| `AutomatedSyncEnabled` | The `Application` has automated sync on | +| `Forbidden` | The applier cluster's identity may not patch the `Kustomization` or its `GitRepository`, or update the `Application` | | `NamespaceNotAuthorized` | `spec.applier.clusterProviderRef` names a provider whose `accessFrom` does not admit this namespace | | `ClusterProviderNotFound` | `spec.applier.clusterProviderRef` names a provider that does not exist | -It is checked on the target's reconcile tick rather than watched, so a cluster without the Flux CRDs -installed costs nothing. Each delivered request also records an `ApplyRequested` Event on the +It is checked on the target's reconcile tick rather than watched, so a cluster without the Flux or +Argo CD CRDs installed costs nothing. Each delivered request also records an `ApplyRequested` Event on the `GitTarget`, and a request that could not be delivered records `ApplyRequestFailed` once, not again each time the worker asks again. @@ -1250,12 +1269,12 @@ spec: | `Reapply` | Once the branch is quiet, this target's [applier](#who-applies-this-folder-specapplier) is asked to apply the current tip, with reason `Refusal`. The refused edit is reverted when the applier applies it; the request asks, and the applier's own status says whether it did | `Reapply` requires `spec.applier`; a target without one is rejected at admission. The request goes -to this target's applier alone, and it waits for a quiet branch, so an allowed edit still in a commit -window is published before the apply and is not reverted with the refused one. When the applier's -`GitRepository` has to fetch to reach the tip, every `Kustomization` on that `GitRepository` applies -the new revision, as it would for any push; give the folder its own `GitRepository` if the apply -must reach it alone. The [bi-directional guide](bi-directional.md#choosing-speconrefusal) -covers when to choose it. +to this target's applier alone, and it waits for a quiet branch, so an allowed edit still in a +commit window is published before the apply and is not reverted with the refused one. An +`Application` is asked to sync the tip. When a `Kustomization`'s `GitRepository` has to fetch to +reach the tip, every `Kustomization` on that `GitRepository` applies the new revision, as it would +for any push; give the folder its own `GitRepository` if the apply must reach it alone. The +[bi-directional guide](bi-directional.md#choosing-speconrefusal) covers when to choose it. **It only fires for an edit to a document the folder already holds, and never for a write that removes one.** That is the pruning fence. Re-applying corrects drift on an object Git manages, diff --git a/docs/design/apply-trigger.md b/docs/design/apply-trigger.md index e775cbfb..462240d5 100644 --- a/docs/design/apply-trigger.md +++ b/docs/design/apply-trigger.md @@ -1,9 +1,10 @@ # The apply trigger -> **design**: decided 2026-10-16, extended 2026-10-17; the Flux applier (in any cluster), the push -> receiver, `onRefusal: Reapply` and the apply window are built, Argo CD and the webhook applier are -> the aim. Index: [`../INDEX.md`](../INDEX.md). -> How to use it: [`../flux.md`](../flux.md) and [`../bi-directional.md`](../bi-directional.md). +> **design**: decided 2026-10-16, extended 2026-10-17; the Flux and Argo CD appliers (in any +> cluster), the push receiver, `onRefusal: Reapply` and the apply window are built, the webhook +> applier is the aim. Index: [`../INDEX.md`](../INDEX.md). +> How to use it: [`../flux.md`](../flux.md), [`../argocd.md`](../argocd.md) and +> [`../bi-directional.md`](../bi-directional.md). > Fields: [`../configuration.md`](../configuration.md#who-applies-this-folder-specapplier). **GitOps Reverser decides when Git is applied to the cluster.** Flux and Argo CD keep fetching, @@ -102,10 +103,12 @@ flowchart LR end EXEC["Dispatcher
one attempt, no retries"] FLUX["Flux
GitRepository · Kustomization"] + ARGO["Argo CD
Application"] OBS --> OWE --> QUIET QUIET -- "yes, and due" --> EXEC EXEC -- "recheck targets, check source,
stamp requestedAt, wait for it" --> FLUX + EXEC -- "recheck targets, check source,
set a pinned sync, wait for it" --> ARGO EXEC -. "report: Delivered · Current ·
Failed · Rejected · Dropped" .-> OWE ``` @@ -123,11 +126,13 @@ flowchart LR another), and for a `Refusal` still consent. If no target still holds the request's reason, it asks nothing (`Dropped`). Just before the stamp it asks the worker whether the decision still holds (the commit is still the tip, nothing is waiting to be published); if not, it stamps - nothing (`Stale`, no failure). Then Flux: stamp `reconcile.fluxcd.io/requestedAt` on the + nothing (`Stale`, no failure). Then the applier's kind decides. Flux: stamp `reconcile.fluxcd.io/requestedAt` on the `GitRepository` when its artifact lacks the commit; when the artifact holds it, only a `Refusal` - stamps the `Kustomization`, because the artifact already woke it for anything else. A source on - another repository or branch is `Rejected`. A delivered attempt is reported only once Flux has - finished the apply, or after 5 minutes, so the attempt is out for exactly the apply window. + stamps the `Kustomization`, because the artifact already woke it for anything else. Argo CD: set + the `Application`'s sync operation, pinned to the commit ([below](#the-argo-cd-applier)). A + source on another repository or branch, or an `Application` with automated sync on, is + `Rejected`. A delivered attempt is reported only once the applier has finished the apply, or + after 5 minutes, so the attempt is out for exactly the apply window. - **Settling.** A delivered attempt is done. A stale one gives back what it carried and is decided again at the next quiet moment. Any other outcome gives back what the attempt carried, a refusal only if it still stands, and starts that applier's backoff (2s doubling to 5m, at least @@ -148,42 +153,72 @@ The quiet-branch rule confines the window to three moments: right after somebody after a refusal the target asked to have reverted, and after `quietPeriod` of silence on a branch whose artifact is behind. Inside it, **the API wins**, as it does for any object both sides changed: -- **The window is known.** The dispatcher reports a delivered attempt only once Flux has finished - the apply: `status.lastHandledReconcileAt` carries the stamp of a stamped `Kustomization`, or the - `Kustomization` attempted a new revision after a source stamp (or the source handled the stamp - with no new artifact). At most 5 minutes; the worker keeps the window open 10 seconds more, - because Flux's writes reach it through the watch. -- **The late apply is withheld.** A write whose latest `managedFields` entry is - `kustomize-controller` alone, inside the window, to an object in the applier's folders that the - API side edited after the attempt was sent, is not mirrored. Git keeps the edit, and the - `CatchUp` its push owes applies it back. The cluster runs the reverted value until then. -- **Only proof withholds.** `managedFields` times resolve to the second. When `kustomize-controller` - ties with another manager, the write could be the edit itself, and withholding an edit loses it +- **The window is known.** The dispatcher reports a delivered attempt only once the applier has + finished the apply: for Flux, `status.lastHandledReconcileAt` carries the stamp of a stamped + `Kustomization`, or the `Kustomization` attempted a new revision after a source stamp (or the + source handled the stamp with no new artifact); for Argo CD, the operation state carrying the + request's token has completed. At most 5 minutes; the worker keeps the window open 10 seconds + more, because the applier's writes reach it through the watch. +- **The late apply is withheld.** A write whose latest `managedFields` entries are the applier's + own managers alone (`kustomize-controller`; `argocd-controller` or + `argocd-application-controller`), inside the window, to an object in the applier's folders that + the API side edited after the attempt was sent, is not mirrored. Git keeps the edit; for Flux the + `CatchUp` its push owes applies it back, for Argo CD the next sync does. The cluster runs the + reverted value until then. +- **Only proof withholds.** `managedFields` times resolve to the second. When the applier ties + with another manager, the write could be the edit itself, and withholding an edit loses it outright; it is mirrored as before. A DELETE carries nothing about its deleter and is never withheld. Both are logged. Every collision is logged with the object, the applier and the requested commit, and counted in `apply_window_collisions_total` (`withheld` or `ambiguous`). What remains: an edit reverted on the -cluster for up to `quietPeriod` until the `CatchUp`, an ambiguous tie mirrored as the applier's -commit, and a Flux write that reaches the watch more than 10 seconds after Flux reported done. +cluster until the next apply (for Flux, up to `quietPeriod`), an ambiguous tie mirrored as the +applier's commit, and an applier's write that reaches the watch more than 10 seconds after it +reported done. -## The aim beyond Flux +## The Argo CD applier -**Argo CD (`applicationRef`).** The `Application` runs with `syncPolicy.automated.enabled: false` -(Argo CD 3.1 and later), so its refresh only compares and nothing syncs it but us. A request sets -`operation.sync` exactly as Argo CD's automated sync builds it, with `prune`, `syncOptions` and -`retry` from the `Application`, and **`revision` pinned to the commit the worker checked**. Three -consequences: +**GitOps Reverser takes full control of when Argo CD syncs.** The `Application` named by +`spec.applier.applicationRef` has automated sync off (`syncPolicy.automated.enabled: false`, or no +`automated` block), so its timed refresh and its webhook only compare, and every sync it runs is +one GitOps Reverser asked for, pinned to a commit the branch worker checked. This is the supported +setup for bi-directional editing with Argo CD, not an option next to automated sync: an +`Application` with automated sync on is `ApplierConfigured=False` (`AutomatedSyncEnabled`) and is +never synced. The user guide is [`../argocd.md`](../argocd.md). -- **No `CatchUp`.** With automated sync off Argo CD applies nothing on its own, so there is no stale - artifact to catch up, and every sync runs the `Application`'s hooks. -- **The commit is pinned**, so a sync applies exactly what the worker checked. -- **A running operation is waited for, never replaced**, as Argo CD's own `SetAppOperation` does. +Argo CD is the easier half. Flux has no fetch-only mode, so it needs a `CatchUp` to keep its +artifact from going stale. With automated sync off Argo CD applies nothing on its own, so it holds +nothing that can go stale, and **it gets no `CatchUp`**: `quietPeriod` is rejected with an +`applicationRef`, and the worker never owes an `Application` one. -A hub Argo CD keeps `Application`s in a management cluster, not the cluster they deploy to. That is -answered as for Flux: `spec.applier.clusterProviderRef` names the cluster the `Application` lives -in, and the `Application`'s own field manager (`argocd-controller`) takes `kustomize-controller`'s -place in the apply window. +Every rule the client relies on is Argo CD's own, read from its source (3.5.0): + +| Fact | Consequence | +| --- | --- | +| Automated sync is on when `syncPolicy.automated` is present and its `enabled` is unset or true (`IsAutomatedSyncEnabled`) | `AutomatedSyncEnabled` is exactly this rule; `automated: {enabled: false, prune: …}` is the off switch that keeps prune | +| Automated sync builds `Operation{Sync{Revision, Prune: automated.prune, SyncOptions, Source}, InitiatedBy{Automated: true}, Retry: syncPolicy.retry or {Limit: 5}}` (`autoSync`) | A request builds the same, with `Revision` pinned to the checked commit | +| A sync starts by setting `operation`, never while one is set, in an update that also clears `status.operationState` (`SetAppOperation`; the CRD has no status subresource) | A request updates the whole `Application`, guarded by the `resourceVersion` it read, and waits out a running operation instead of replacing it | +| The controller clears `operation` when the operation completes and leaves the result in `status.operationState`; a failed operation that will be retried is `Running` again, with a `finishedAt` | An attempt's window closes when the operation state carrying its token has completed (`Succeeded`, `Failed`, `Error`); a pending retry is still running | +| A retry with `retry.refresh: true` drops the operation's revision and syncs the newest one | That one field is dropped from the copied retry: a retry stays pinned | +| Every apply writes as field manager `argocd-controller`, client-side and server-side alike; a `Replace=true` sync creates and replaces with no field manager, so the API server takes it from the user agent, `argocd-application-controller` | The apply window knows both as Argo CD's managers | +| Sync windows are checked for every operation, by `InitiatedBy.Automated` | A sync GitOps Reverser starts obeys the project's windows for automated syncs | + +Three choices were decided (2026-10-17): + +1. **Prune** comes from `syncPolicy.automated.prune` when `automated` is present with + `enabled: false`. Without the block, a sync does not prune, and the guide says so. There is no + `spec.applier.prune`. +2. **The initiator** is `InitiatedBy{Username: "gitops-reverser", Automated: true}`. GitOps Reverser + replaces automated sync, so a sync window that blocks automated syncs blocks its syncs too. +3. **Hooks run on every sync**, so on every `ForeignCommit`, which asks every applier on the + branch. Narrowing it by changed paths waits for a measured complaint. + +A hub Argo CD keeps `Application`s in a management cluster: `spec.applier.clusterProviderRef` +reaches it as it reaches a Flux in another cluster. Out of scope: multi-source `Application`s (and +`sourceHydrator`), which are `SourceMismatch`; `ApplicationSet`s, which own the spec of the +`Application`s they generate; and Argo CD's own sharding and HA. + +## The aim beyond Flux and Argo CD **A webhook applier** for any other tool: POST `{"repository", "branch", "commit", "reason"}`, signed `X-Signature: sha256=` the way Flux's `generic-hmac` `Receiver` checks. It gets @@ -204,25 +239,27 @@ in-process plugin: it would run with every credential the operator holds. - **Every applier on the branch is asked on a `ForeignCommit`**, whether or not the push touched its folder. Harmless for Flux; for Argo CD it runs hooks. Narrowing it needs the applier's own source paths, so it waits for a measured complaint. +- **A sync somebody else started holds an Argo CD request**, for up to 5 minutes, and then fails + it; the worker asks again after its backoff. ## Where this goes next The Flux applier PR (#419) carries what gitops-api, the first adopter, needs before it can drop its own gate: both reviews' findings and the customer review's three blockers (stale stamps, a late webhook cutting a save's window, a `Kustomization` in the management cluster via -`spec.applier.clusterProviderRef`), the hardened receiver, and the apply window. After it, in order, -each one pull request: +`spec.applier.clusterProviderRef`), the hardened receiver, the apply window, and the Argo CD +applier with automated sync off as the standard. After it, in order, each one pull request: -1. **The Argo CD applier**, as described above, with automated sync off as the standard. The - implementation plan is [`argocd-applier-plan.md`](argocd-applier-plan.md). -2. **The webhook applier.** -3. **More than one replica.** Leader election is not switched on, and the chart runs one replica. +1. **The webhook applier.** +2. **More than one replica.** Leader election is not switched on, and the chart runs one replica. Standbys need ownership coordination first (the HA plan in `docs/future/`), and deliveries then need routing to the pod that owns the branch. -Test gaps to close along the way: a remote source cluster and an applier in another cluster in e2e, -`Forbidden` against real RBAC, the receiver and applier role installed by the Helm chart, two -`GitTarget`s sharing one `Kustomization` in e2e, an apply-window collision in e2e, and `-race` in CI. +Test gaps to close along the way: a remote source cluster and an applier in another cluster (a hub +Argo CD among them) in e2e, `Forbidden` against real RBAC, the receiver and applier roles installed +by the Helm chart, two `GitTarget`s sharing one `Kustomization` in e2e, an apply-window collision in +e2e, a `Replace=true` Argo CD sync's field manager measured rather than read from the source, and +`-race` in CI. **What adopting it changes for gitops-api.** Its gate shrinks to "one save, one window", which protects commit messages and has nothing to do with Flux. Its push handler relays each push to the @@ -244,6 +281,11 @@ been quiet; `quietPeriod` is the knob for that trade. | `SourceMismatch` is never stamped and is delivered once fixed; a Gitea webhook and the refresh alone each get a push applied | [Manager Flux Applier](../../test/e2e/flux_applier_e2e_test.go) | | A refused edit is reverted by its `Kustomization` | [Refusal reapply](../../test/e2e/refusal_reapply_e2e_test.go) | | The Flux guide's example is accepted by the code | [`flux_guide_test.go`](../../internal/applier/flux_guide_test.go) | +| An Argo CD sync is the one automated sync builds, pinned to the commit, with a pinned retry; a running operation is waited out and never replaced; an operation set between the read and the update makes it fail; every `ApplierConfigured` reason, `AutomatedSyncEnabled` among them; when a sync counts as finished | [`argocd_test.go`](../../internal/applier/argocd_test.go), [`dispatcher_test.go`](../../internal/applier/dispatcher_test.go) | +| An `Application` is never owed a `CatchUp`; Argo CD's late write over a newer edit is withheld by its own managers | [`apply_trigger_test.go`](../../internal/git/apply_trigger_test.go), [`apply_window_test.go`](../../internal/git/apply_window_test.go) | +| Exactly one of `kustomizationRef` and `applicationRef`, and no `quietPeriod` with an `applicationRef` | [`gittarget_applier_schema_test.go`](../../internal/controller/gittarget_applier_schema_test.go) | +| With Argo CD: two API edits keep the later value with no extra commit; a Git-side push is synced by a pinned `ForeignCommit`; a refused edit is reverted by a sync; an `Application` with automated sync on is `AutomatedSyncEnabled` and never synced | [bi-directional Argo CD corner](../../test/e2e/argocd_bi_directional_e2e_test.go) | +| The Argo CD guide's example is accepted by the code | [`argocd_guide_test.go`](../../internal/applier/argocd_guide_test.go) | The history of how this was decided, with the options weighed and the Git-side reference the code cites, is in [`../finished/push-notification-and-reconcile-trigger.md`](../finished/push-notification-and-reconcile-trigger.md). diff --git a/docs/design/argocd-applier-plan.md b/docs/design/argocd-applier-plan.md deleted file mode 100644 index 375325b3..00000000 --- a/docs/design/argocd-applier-plan.md +++ /dev/null @@ -1,140 +0,0 @@ -# The Argo CD applier: implementation plan - -> **plan**: written 2026-10-17, to be built in a fresh session on #419's branch -> (`docs-bi-directional-right`), before it merges. Delete it -> once the PR lands; the lasting design goes into [`apply-trigger.md`](apply-trigger.md). -> Index: [`../INDEX.md`](../INDEX.md). - -**The standard: GitOps Reverser takes full control of when Argo CD syncs.** An `Application` named -by a `GitTarget` has automated sync switched off, and every sync it runs is one GitOps Reverser -asked for, pinned to a commit it checked. This is the supported setup for bi-directional editing -with Argo CD, not an option next to automated sync. - -Argo CD is the easier half of the apply trigger. Flux has no fetch-only mode: a new artifact applies -at once. With automated sync off, Argo CD's timed refresh and its Git webhook only **compare**, so -they can stay on and do no harm, and nothing syncs but us. There is no stale artifact to catch up, -so Argo CD gets no `CatchUp` (already decided; do not re-raise). - -## Start here (new session) - -- Read [`apply-trigger.md`](apply-trigger.md) first: the worker owns every obligation per applier, - the dispatcher makes one attempt and reports back, and quietness is per repository and branch. - None of that changes. -- The Flux applier is the template to copy: [`internal/applier/flux.go`](../../internal/applier/flux.go) - (`Validate`, `Request`, `Handled`), [`internal/applier/dispatcher.go`](../../internal/applier/dispatcher.go), - [`internal/git/apply_trigger.go`](../../internal/git/apply_trigger.go), - [`internal/git/apply_window.go`](../../internal/git/apply_window.go), - [`internal/controller/gittarget_applier.go`](../../internal/controller/gittarget_applier.go). -- The Argo CD upstream checkout is in the gitignored `external-sources/argo-cd` (3.5.0 at - `3cbae653d`). Read it as ground truth; never link to it from a tracked doc. -- The e2e corner that installs Argo CD is `task test-e2e-bi-directional` - ([`argocd_bi_directional_e2e_test.go`](../../test/e2e/argocd_bi_directional_e2e_test.go)). It - occasionally dies on an ECR rate limit pulling `argocd-redis`; rerun later, that is not the diff. - -## Facts verified in the upstream source - -| Fact | Where (external-sources/argo-cd) | Consequence | -| --- | --- | --- | -| Every apply writes as field manager `argocd-controller`, client-side and server-side alike | `gitops-engine/pkg/sync/sync_context.go` passes `serverSideApplyManager` to `ApplyResource`; `gitops-engine/pkg/utils/kube/resource_ops.go:532` sets it as `FieldManager` | The apply window works as for Flux, with `argocd-controller` in place of `kustomize-controller`. `kubectl-client-side-apply` is only the manager Argo CD migrates *from* | -| Autosync builds `Operation{Sync{Revision, Prune: automated.GetPrune(), SyncOptions, Source(s), Revisions}, InitiatedBy{Automated: true}, Retry: spec.syncPolicy.retry or {Limit: 5}}` | `controller/appcontroller.go:2628` | Build ours the same way, with `Revision` pinned to the checked commit | -| Setting an operation refuses while one is set (`ErrAnotherOperationInProgress`) | `util/argo/argo.go` `SetAppOperation` | Never replace a running operation; wait for it | -| Automated sync is on when `syncPolicy.automated` is present and `enabled` is unset or true | `pkg/apis/application/v1alpha1/types.go` `IsAutomatedSyncEnabled` | `AutomatedSyncEnabled` is exactly this rule; `automated: {enabled: false, prune: …}` is the off switch that keeps prune | -| Sync windows are checked for every operation, manual or not, by `InitiatedBy.Automated` | `controller/sync.go` `syncWindowPreventsSync` | A sync we set obeys the project's windows; with `Automated: true` (decision 2) the automated rules hold | - -To verify while building: the `Replace=true` and create paths (`UpdateResource`, `CreateResource`, -`ReplaceResource` in `sync_context.go`) may write under a different manager; hooks create objects -(`Job`s) during a sync. - -## What to build - -### API - -- **`spec.applier.applicationRef`** (`meta.NamespacedObjectReference`), the Argo CD `Application`. - Exactly one of `kustomizationRef` and `applicationRef` (CEL on `GitTargetApplier`); the namespace - defaults to the `GitTarget`'s, like `kustomizationRef` (the user decided: no namespace gate). -- **`clusterProviderRef`** works unchanged. It is how a hub Argo CD is reached: the `Application` - lives in the management cluster. -- **`quietPeriod`** means nothing without `CatchUp`: refuse it together with `applicationRef` (CEL), - rather than accept a knob that does nothing. -- **`onRefusal: Reapply`** works with `applicationRef`. That closes the known gap where Argo CD - folders lost the automatic refusal revert when `PushEmptyCommit` was removed. - -### The applier interface - -- `git.ApplierRef` gains a `Kind` (`Kustomization` | `Application`), part of the key. -- The worker skips `CatchUp` for an `Application` applier: `sendDue` never owes it one, and its quiet - period is not consulted. `ForeignCommit` and `Refusal` work as for Flux. -- The window's field manager comes from the applier's kind (`FluxFieldManager` today, add - `ArgoCDFieldManager = "argocd-controller"`). -- The dispatcher picks a client by kind behind one small interface (`Validate`, `Request`, - `Handled`), so `attempt` and `awaitHandled` stay as they are. Metrics already carry - `applier_kind`; add `application`. - -### The Argo CD client (`internal/applier/argocd.go`) - -- **Validate** (for `ApplierConfigured`): - - `ApplierNotFound`: no such `Application`, or the CRD is not served. - - `SourceMismatch`: `spec.source.repoURL` is not this repository (`repourl.Same`), or - `targetRevision` is not this branch (a branch name or `refs/heads/`; `HEAD` and tags - mismatch). A multi-source `Application` is refused here in v1, with a message that says so. - - **`AutomatedSyncEnabled`** (new, stalled): `spec.syncPolicy.automated` is present and not - `enabled: false` (`IsAutomatedSyncEnabled`). The standard is full control, so this is a - misconfiguration, not a warning. - - `Forbidden`: a SelfSubjectAccessReview for `patch` on `applications.argoproj.io` fails. -- **Request**: - 1. Read the `Application`. Check the source and the automated-sync rule again (a published - verdict can be older than the object). Either failing is `Rejected`. - 2. If an operation is set or `status.operationState.phase` is `Running`, wait for it to end, - bounded by the apply-window timeout, then read again. Never replace it. - 3. Ask `Current()`. False is `Stale`, as for Flux. - 4. Update the `Application` with `operation` set as autosync builds it (see the facts table), - `Revision` = the commit, `InitiatedBy{Username: "gitops-reverser", Automated: true}`, - `Prune` per decision 1. Use an update guarded by - `resourceVersion`, so a sync somebody else started in between makes it fail instead of - replacing theirs. -- **Handled**: `status.operationState` is for our operation (`initiatedBy.username` and the revision - match) and has `finishedAt`, whatever the phase. A failed sync still closes the window; Argo CD's - own retry belongs to the operation. -- There is no `OutcomeCurrent` for Argo CD: every `ForeignCommit` and `Refusal` syncs. - -### Controller, RBAC and docs - -- `ApplierConfigured` per kind; `applierReadiness` treats `AutomatedSyncEnabled` as stalled. -- RBAC: `config/rbac/argocd-applier-role.yaml` (`get` and `update` on `applications.argoproj.io`) - and Helm `applierAccess.argocd`, off by default, mirroring the Flux grant. -- Docs: a `docs/argocd.md` guide shaped like [`../flux.md`](../flux.md), with the standard spelled - out (automated sync off; refresh and the Argo CD webhook may stay). Update the Argo CD section of - [`../bi-directional.md`](../bi-directional.md), [`../configuration.md`](../configuration.md), - the support-boundary [`argocd-bi-directional.md`](support-boundary/argocd-bi-directional.md), - the metric rows in [`../interpreting-metrics.md`](../interpreting-metrics.md), and the "aim beyond - Flux" section of [`apply-trigger.md`](apply-trigger.md). - -## Tests - -- **Unit**: the operation equals what autosync would build except the pinned revision and the - initiator; a running operation is waited for and never replaced; a concurrent operation makes the - guarded update fail; every `Validate` reason; `Handled` for success, failure and somebody else's - operation; the worker owes an `Application` no `CatchUp`; the window withholds an - `argocd-controller` write over a newer edit. -- **e2e, bi-directional corner**: move the existing Argo CD specs to the standard (template - `Automated: false`, `spec.applier.applicationRef`). Prove: - - two API edits seconds apart keep the later value, with no further commit; - - a Git-side push is synced by a `ForeignCommit` request, pinned to that commit; - - a refused edit with `Reapply` is reverted by an Argo CD sync; - - an `Application` with automated sync on is `AutomatedSyncEnabled` and is never synced. - -## Decided (2026-10-17, do not re-raise) - -1. **Prune** comes from `syncPolicy.automated.prune` when `automated` is present with - `enabled: false` (the off switch that keeps the block). Without the block, the sync does not - prune, and the guide says so. No `spec.applier.prune` field. -2. **The initiator** is `InitiatedBy{Username: "gitops-reverser", Automated: true}`: we replace - autosync, so a sync window that blocks automated syncs blocks ours too. -3. **Hooks on every foreign commit** ship as they are. `ForeignCommit` keeps asking every applier on - the branch; narrowing by changed paths waits for a measured complaint. - -## Out of scope - -Multi-source `Application`s, `ApplicationSet`s (they generate `Application`s whose spec they own), -Argo CD's `syncPolicy.automated.selfHeal` (meaningless with automated sync off), and Argo CD -sharding or its own HA. diff --git a/docs/design/support-boundary/argocd-bi-directional.md b/docs/design/support-boundary/argocd-bi-directional.md index 5aa47360..f92e050c 100644 --- a/docs/design/support-boundary/argocd-bi-directional.md +++ b/docs/design/support-boundary/argocd-bi-directional.md @@ -9,12 +9,20 @@ amount of configuration reconciles them — because the moment a live write land Argo already has everything it needs to revert it, and there is no hook where an external system can say "wait, this one is legitimate." -The one Argo CD configuration that supports a shared field changing from **both** +The Argo CD configuration that supports a shared field changing from **both** directions is: -- **automated sync with `selfHeal: false`**, plus -- **a push webhook** (Gitea/GitHub → argocd-server `/api/webhook`) for the - Git → cluster direction. +- **automated sync off** (`syncPolicy.automated.enabled: false`), with the + `Application` named by the `GitTarget`'s `spec.applier.applicationRef`, so GitOps + Reverser starts every sync, pinned to a commit it checked + ([the Argo CD guide](../../argocd.md)), plus +- **a push webhook** into GitOps Reverser's receiver for the Git → cluster + direction. + +Automated sync with `selfHeal: false` was the answer before GitOps Reverser could +drive an `Application`, and is still the best an unnamed `Application` can do; it +leaves the timed refresh syncing GitOps Reverser's own pushes over whatever is +pending ([why](#why-selfheal-false-is-not-enough)). Everything else that looks like a solution — `ignoreDifferences`, `managedFieldsManagers`, three-way merge, PreSync hooks, self-heal timing knobs — @@ -145,8 +153,19 @@ setting. Verdict: **cannot be used to be slow.** ## The configuration that works -Turn self-heal **off** and let the two directions flow without a reverting -controller in the middle: +Turn automated sync **off** and let GitOps Reverser start every sync. It is +`selfHeal: false` taken to its end: nothing in Argo CD applies on its own, so the +only syncs are ones started on a quiet branch, pinned to a commit with nothing live +missing from it. Argo CD's own rules make it safe to drive this way: a sync is +started by setting the `Application`'s `operation` as automated sync would +(`controller/appcontroller.go` `autoSync`), and never while another operation is set +(`util/argo/argo.go` `SetAppOperation`). With automated sync off, a refresh only +compares (`autoSync` returns at once when `IsAutomatedSyncEnabled` is false). + +### Why `selfHeal: false` is not enough + +The intermediate step, still useful for an `Application` no `GitTarget` names, is +automated sync with self-heal **off**: - **Cluster → Git.** With `selfHeal: false`, Argo does not revert live drift — it marks the app `OutOfSync` and waits. GitOps Reverser captures the live change to @@ -163,10 +182,13 @@ controller in the middle: └──────────── apply ◀── Argo CD (selfHeal off) ◀─push webhook┘ ``` -This is genuinely bi-directional on a shared field, and it is exactly the -configuration the e2e corner exercises once you drop the `selfHeal` flag. The -webhook is what makes it comfortable rather than sluggish — without it, the -Git → cluster direction waits on Argo's 120s poll. +This is bi-directional on a shared field, with one hole: automated sync still +applies **every new commit**, including each one GitOps Reverser pushes, two to three +minutes later on the timed refresh (or seconds later with Argo CD's webhook), over +whatever is pending at that moment. Two API edits a few seconds apart are enough to +have the first commit synced over the second edit. Only switching automated sync +off and letting GitOps Reverser start the syncs closes it, which is what the e2e +corner now exercises. **What you give up** is self-heal's one unique service: reverting drift that *should not* be captured (a rogue or accidental change). That loss is inherent — @@ -194,12 +216,14 @@ forgotten. | PreSync hook (delay) | **No** | target frozen at operation start; reverts stale later | | PreSync hook (veto) | **No** | also aborts legitimate deploys; can't tell them apart | | self-heal timeout / backoff | **No** | throttles repeats; first drift reverted immediately | -| `selfHeal: false` + push webhook | **Yes** | nothing reverts; both directions flow, fast | +| `selfHeal: false` + push webhook | **Mostly** | nothing reverts drift, but every new commit, GitOps Reverser's own included, is synced over pending edits | +| automated sync off + `spec.applier.applicationRef` | **Yes** | GitOps Reverser starts every sync, on a quiet branch, pinned to a checked commit | | admission gate on Argo's writes | (recovers "revert unauthorized") | per-write authority decision; external to Argo | ## References -- User-facing guidance: [`../../bi-directional.md`](../../bi-directional.md) +- User-facing guidance: [`../../bi-directional.md`](../../bi-directional.md) and + [`../../argocd.md`](../../argocd.md) - E2E corner design: [`../e2e-bi-directional-corner.md`](../../spec/e2e-bi-directional-corner.md) - Exercised behavior: [`../../../test/e2e/argocd_bi_directional_e2e_test.go`](../../../test/e2e/argocd_bi_directional_e2e_test.go) - Argo CD source (vendored): `external-sources/argo-cd` — cited inline by file:line. diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index ce745f43..e06a5500 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -816,20 +816,21 @@ are unbounded, and both are in the log line at the write site. ## Apply requests and the push receiver -With `GitTarget.spec.applier` set, the branch worker decides when the Flux `Kustomization` applies, -and the push receiver tells it that somebody else pushed. See +With `GitTarget.spec.applier` set, the branch worker decides when the Flux `Kustomization` or Argo +CD `Application` applies, and the push receiver tells it that somebody else pushed. See [who applies this folder](configuration.md#who-applies-this-folder-specapplier). | Metric | Type | Labels | Meaning | | --- | --- | --- | --- | -| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per attempt to one applier. `applier_kind` is `kustomization`; `reason` is `ForeignCommit` / `Refusal` / `CatchUp`; `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not ask, including a `Kustomization` that applies another repository or branch; the branch worker keeps the obligation and decides the retry on a quiet branch, so a failure that persists keeps counting at its backoff) or `dropped` (no `GitTarget` may still ask for that reason: each was deleted or recreated, moved to another branch or applier, is no longer admitted by its `ClusterProvider`, or withdrew its consent to a `Refusal`) or `stale` (asked nothing because, by the time of the patch, the commit was no longer the branch tip or an edit was waiting to be published; the worker decides again at the next quiet moment) | -| `apply_window_collisions_total` | counter | `provider_namespace`, `provider_name`, `branch`, `outcome` | One per write by the applier, while an apply it was asked for runs, to an object the API side edited after the request was sent: the apply arriving late over a newer edit. `outcome` is `withheld` (proven to be the applier's write, so not mirrored: Git keeps the API's value and the next apply restores it on the cluster) or `ambiguous` (the applier and another field manager wrote in the same second, so it was mirrored as ever; if it was the apply, the revert is in Git as the applier's commit) | +| `apply_requests_total` | counter | `applier_kind`, `reason`, `outcome` | One per attempt to one applier. `applier_kind` is `kustomization` (a Flux `Kustomization`) or `application` (an Argo CD `Application`); `reason` is `ForeignCommit` / `Refusal` / `CatchUp` (never `CatchUp` for an `application`); `outcome` is `sent` (delivered, or a `CatchUp` that found Flux already current), `failed` (could not ask, including an applier that applies another repository or branch, an `Application` with automated sync on, or one whose running sync did not end within 5 minutes; the branch worker keeps the obligation and decides the retry on a quiet branch, so a failure that persists keeps counting at its backoff) or `dropped` (no `GitTarget` may still ask for that reason: each was deleted or recreated, moved to another branch or applier, is no longer admitted by its `ClusterProvider`, or withdrew its consent to a `Refusal`) or `stale` (asked nothing because, by the time of the write, the commit was no longer the branch tip or an edit was waiting to be published; the worker decides again at the next quiet moment) | +| `apply_window_collisions_total` | counter | `provider_namespace`, `provider_name`, `branch`, `outcome` | One per write by the applier, while an apply it was asked for runs, to an object the API side edited after the request was sent: the apply arriving late over a newer edit. `outcome` is `withheld` (proven to be the applier's write, by its field manager: `kustomize-controller`, or `argocd-controller` / `argocd-application-controller`; so not mirrored: Git keeps the API's value and the next apply restores it on the cluster) or `ambiguous` (the applier and another field manager wrote in the same second, so it was mirrored as ever; if it was the apply, the revert is in Git as the applier's commit) | | `receiver_deliveries_total` | counter | `type`, `outcome` | One per verified-or-refused delivery to `/git-push/`. `type` is the `GitProvider.spec.receiver.type`; `outcome` is `notified` (handed to the branch worker), `unmatched` (no worker for that branch), `ignored` (a ping, a tag, another branch's event type, or a `generic-hmac` body naming another repository), `rejected` (bad signature or an oversized body) or `invalid` (verified but unparseable) | A delivery counted `notified` is only a hint: the worker reads the advertisement, and fetches under `git_fetches_total{reason="refresh"}` only when the branch actually moved. A `ForeignCommit` request follows when it did. So on a branch where only GitOps Reverser pushes, `notified` deliveries are -followed by `CatchUp` requests and never by `ForeignCommit`. +followed by `CatchUp` requests (to a `Kustomization`; an `Application` gets none) and never by +`ForeignCommit`. ```promql # Requests that could not be delivered, by reason. A failure that persists keeps counting, at the diff --git a/hack/crdfields/fields.yaml b/hack/crdfields/fields.yaml index 0a53d785..44f288b4 100644 --- a/hack/crdfields/fields.yaml +++ b/hack/crdfields/fields.yaml @@ -138,11 +138,12 @@ kinds: Which deletions reach Git. See [deletion policy](#deletion-policy-specprunemode) - path: applier - default: Flux applies on its own schedule + default: the applier applies on its own schedule doc: >- - The Flux `Kustomization` that applies this folder (`kustomizationRef`), which GitOps - Reverser then asks to apply, and how long the branch stays quiet before it catches up - (`quietPeriod`, `30s`). See [who applies this folder](#who-applies-this-folder-specapplier) + The Flux `Kustomization` (`kustomizationRef`) or Argo CD `Application` (`applicationRef`) + that applies this folder, which GitOps Reverser then asks to apply, and for a + `Kustomization` how long the branch stays quiet before it catches up (`quietPeriod`, + `30s`). See [who applies this folder](#who-applies-this-folder-specapplier) - path: onRefusal doc: >- What happens to an edit with no legal destination. See diff --git a/internal/applier/argocd_guide_test.go b/internal/applier/argocd_guide_test.go new file mode 100644 index 00000000..6dc75cf5 --- /dev/null +++ b/internal/applier/argocd_guide_test.go @@ -0,0 +1,76 @@ +// SPDX-License-Identifier: Apache-2.0 + +package applier + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/types" + + "github.com/ConfigButler/gitops-reverser/api/v1alpha3" +) + +// argoCDGuide is the Argo CD user guide. Its "A complete example" section is the configuration +// this file holds to the code: a reader copies it, so it must be accepted as written. +const argoCDGuide = "../../docs/argocd.md" + +// TestArgoCDGuide_TheCompleteExampleIsAccepted reads the guide's complete example and checks it +// against the code that validates and drives an Argo CD applier: the GitOps Reverser objects decode +// into the API types with no unknown field, ApplierConfigured comes out True for the Application as +// written, and a request sets a sync pinned to the commit. It also holds the example to the guide's +// own advice: automated sync off, prune kept, and a recursive directory source. +func TestArgoCDGuide_TheCompleteExampleIsAccepted(t *testing.T) { + objects := guideExampleObjects(t, argoCDGuide) + + var provider v1alpha3.GitProvider + var target v1alpha3.GitTarget + var secret corev1.Secret + decodeStrict(t, objects, "GitProvider", &provider) + decodeStrict(t, objects, "GitTarget", &target) + decodeStrict(t, objects, "Secret", &secret) + app := one(t, objects, kindApplication) + + // The GitOps Reverser objects are wired to each other. + assert.Equal(t, provider.Name, target.Spec.GitProviderRef.Name) + assert.Equal(t, provider.Namespace, target.Namespace, "a GitTarget names a GitProvider in its own namespace") + require.NotNil(t, provider.Spec.Receiver, "the example serves a push receiver") + assert.Equal(t, secret.Name, provider.Spec.Receiver.SecretRef.Name) + assert.NotEmpty(t, secret.StringData[v1alpha3.ReceiverTokenKey], "the receiver reads the token key") + require.NotNil(t, target.Spec.Applier) + assert.Equal(t, v1alpha3.ApplierKindApplication, target.Spec.Applier.Kind()) + assert.Nil(t, target.Spec.Applier.QuietPeriod, "quietPeriod is rejected with an applicationRef") + assert.Equal(t, v1alpha3.RefusalActionReapply, target.Spec.OnRefusal) + + // The Application follows the guide's advice. + assert.False(t, automatedSyncEnabled(app), "the standard is automated sync off") + assert.True(t, automatedPrune(app), "the off switch keeps prune for the syncs GitOps Reverser starts") + recurse, _, _ := unstructured.NestedBool(app.Object, "spec", "source", "directory", "recurse") + assert.True(t, recurse, "GitOps Reverser writes nested folders") + + // The applier, as written, is accepted and driven. + ref := types.NamespacedName{ + Namespace: target.Spec.Applier.ApplicationRef.Namespace, Name: target.Spec.Applier.ApplicationRef.Name, + } + want := Binding{RepositoryURL: provider.Spec.URL, Branch: target.Spec.Branch} + client := newSourceClient(app) + argo := pinnedArgo(client) + verdict := argo.Validate(context.Background(), ref, want) + assert.True(t, verdict.OK, "ApplierConfigured for the guide's example: %s: %s", verdict.Reason, verdict.Message) + + stamped, err := argo.Request(context.Background(), ref, want, testCommit, v1alpha3.ApplyReasonForeignCommit, nil) + require.NoError(t, err) + assert.Equal(t, OutcomeSyncRequested, stamped.Outcome) + written, err := client.Resource(applicationGVR()).Namespace(ref.Namespace). + Get(context.Background(), ref.Name, metav1.GetOptions{}) + require.NoError(t, err) + revision, _, _ := unstructured.NestedString(written.Object, "operation", "sync", "revision") + prune, _, _ := unstructured.NestedBool(written.Object, "operation", "sync", "prune") + assert.Equal(t, testCommit, revision) + assert.True(t, prune) +} diff --git a/internal/applier/flux_guide_test.go b/internal/applier/flux_guide_test.go index d6bbe737..8f650177 100644 --- a/internal/applier/flux_guide_test.go +++ b/internal/applier/flux_guide_test.go @@ -35,7 +35,7 @@ const fluxGuide = "../../docs/flux.md" // as written, and a request stamps the GitRepository. It also holds the example to the guide's own // advice, so the page cannot recommend one thing and show another. func TestFluxGuide_TheCompleteExampleIsAccepted(t *testing.T) { - objects := guideExampleObjects(t) + objects := guideExampleObjects(t, fluxGuide) var provider v1alpha3.GitProvider var target v1alpha3.GitTarget @@ -95,17 +95,17 @@ func TestFluxGuide_TheCompleteExampleIsAccepted(t *testing.T) { assert.NotEmpty(t, requestedAt(t, client, gitRepositoryGVR(), repository.GetNamespace(), repository.GetName())) } -// guideExampleObjects returns every object in the yaml blocks of the guide's "A complete example" +// guideExampleObjects returns every object in the yaml blocks of a guide's "A complete example" // section, by kind. -func guideExampleObjects(t *testing.T) map[string][]*unstructured.Unstructured { +func guideExampleObjects(t *testing.T, guide string) map[string][]*unstructured.Unstructured { t.Helper() - content, err := os.ReadFile(fluxGuide) + content, err := os.ReadFile(guide) require.NoError(t, err) text := string(content) const heading = "\n## A complete example\n" start := strings.Index(text, heading) - require.GreaterOrEqual(t, start, 0, "%s must keep its %q section", fluxGuide, strings.TrimSpace(heading)) + require.GreaterOrEqual(t, start, 0, "%s must keep its %q section", guide, strings.TrimSpace(heading)) section := text[start+len(heading):] if end := strings.Index(section, "\n## "); end >= 0 { section = section[:end] diff --git a/internal/controller/gittarget_applier.go b/internal/controller/gittarget_applier.go index 51b12d71..de6a477f 100644 --- a/internal/controller/gittarget_applier.go +++ b/internal/controller/gittarget_applier.go @@ -13,16 +13,18 @@ import ( configbutleraiv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" "github.com/ConfigButler/gitops-reverser/internal/applier" "github.com/ConfigButler/gitops-reverser/internal/authz" + "github.com/ConfigButler/gitops-reverser/internal/git" "github.com/ConfigButler/gitops-reverser/internal/types" ) // GitTargetConditionApplierConfigured reports whether spec.applier names an object GitOps Reverser // can ask to apply: it exists in the applier's cluster, applies this target's repository and branch, -// and may be patched. Present only while spec.applier is set. +// may be written, and (an Argo CD Application) has automated sync off. Present only while +// spec.applier is set. const GitTargetConditionApplierConfigured = "ApplierConfigured" -// applierCheckTimeout bounds the ApplierConfigured check: two GETs and two access reviews against -// the source cluster. Its client has no request timeout, because it also carries the watches, and +// applierCheckTimeout bounds the ApplierConfigured check: at most two GETs and two access reviews +// against the applier's cluster. Its client has no request timeout, because it also carries the watches, and // this controller runs one reconcile worker, so a source API server that accepts the connection and // never answers would otherwise stop every GitTarget's reconcile, not only this one's. const applierCheckTimeout = 10 * time.Second @@ -39,8 +41,8 @@ type ApplyRequestLedger interface { // condition for the readiness trio. present is false when the target names no applier, in which // case both are removed. // -// The check reads the applier on this reconcile rather than watching it. Flux may not be installed -// in a source cluster at all, and an informer on a kind that is not served would be a failure to +// The check reads the applier on this reconcile rather than watching it. Flux or Argo CD may not be +// installed in the applier's cluster at all, and an informer on a kind that is not served would be a failure to // report rather than a check that answers "not found"; the cadence is the target's own. func (r *GitTargetReconciler) observeApplier( ctx context.Context, @@ -74,14 +76,16 @@ func (r *GitTargetReconciler) applierVerdict( target *configbutleraiv1alpha3.GitTarget, gitProvider *configbutleraiv1alpha3.GitProvider, ) conditionValue { - ref := applierReference(target) + applierRef := git.ApplierOf(target) + ref := k8stypes.NamespacedName{Namespace: applierRef.Namespace, Name: applierRef.Name} + kind := applierRef.Kind ctx, cancel := context.WithTimeout(ctx, applierCheckTimeout) defer cancel() if r.ApplierClients == nil || gitProvider == nil { return conditionValue{ Status: metav1.ConditionUnknown, Reason: GitTargetReasonNotChecked, - Message: fmt.Sprintf("Kustomization %s has not been checked: no cluster client is wired", ref), + Message: fmt.Sprintf("%s %s has not been checked: no cluster client is wired", kind, ref), } } cluster := target.ApplierCluster() @@ -97,20 +101,20 @@ func (r *GitTargetReconciler) applierVerdict( client, err := r.ApplierClients(ctx, cluster) if err != nil { return keptOrUnknown(target, - fmt.Sprintf("Cannot reach cluster %q to check Kustomization %s: %v", cluster, ref, err)) + fmt.Sprintf("Cannot reach cluster %q to check %s %s: %v", cluster, kind, ref, err)) } - newFlux := r.newFlux - if newFlux == nil { - newFlux = applier.NewFlux + newApplier := r.newApplier + if newApplier == nil { + newApplier = applier.NewClient } - verdict := newFlux(client).Validate(ctx, ref, + verdict := newApplier(kind, client).Validate(ctx, ref, applier.Binding{RepositoryURL: gitProvider.Spec.URL, Branch: target.Spec.Branch}) switch { case verdict.OK: return conditionValue{ Status: metav1.ConditionTrue, Reason: ReasonSucceeded, - Message: fmt.Sprintf("Kustomization %s applies this repository and branch, and can be asked to apply", ref), + Message: fmt.Sprintf("%s %s applies this repository and branch, and can be asked to apply", kind, ref), } case verdict.Err != nil: return keptOrUnknown(target, verdict.Message) @@ -129,20 +133,10 @@ func keptOrUnknown(target *configbutleraiv1alpha3.GitTarget, message string) con return conditionValue{Status: metav1.ConditionUnknown, Reason: ReasonProgressing, Message: message} } -// applierReference is spec.applier.kustomizationRef, its namespace defaulting to the GitTarget's. -func applierReference(target *configbutleraiv1alpha3.GitTarget) k8stypes.NamespacedName { - ref := target.Spec.Applier.Reference() - namespace := ref.Namespace - if namespace == "" { - namespace = target.Namespace - } - return k8stypes.NamespacedName{Namespace: namespace, Name: ref.Name} -} - -// applierReadiness contributes ApplierConfigured to the trio. A Kustomization that applies another -// repository, or that the operator may not patch, needs a human; one that is missing may be on its -// way (Flux objects are often created after the GitTarget), and an unknown verdict is still -// settling. +// applierReadiness contributes ApplierConfigured to the trio. An applier that applies another +// repository, that the operator may not write, or (an Argo CD Application) that has automated sync +// on, needs a human; one that is missing may be on its way (Flux and Argo CD objects are often +// created after the GitTarget), and an unknown verdict is still settling. func applierReadiness(rd *readiness, value conditionValue, present bool) { if !present { return diff --git a/internal/controller/gittarget_applier_test.go b/internal/controller/gittarget_applier_test.go index 0e5149c2..ef6dca79 100644 --- a/internal/controller/gittarget_applier_test.go +++ b/internal/controller/gittarget_applier_test.go @@ -56,6 +56,7 @@ func applierFakeSource(objs ...runtime.Object) *dynamicfake.FakeDynamicClient { map[schema.GroupVersionResource]string{ {Group: "kustomize.toolkit.fluxcd.io", Version: "v1", Resource: "kustomizations"}: "KustomizationList", {Group: "source.toolkit.fluxcd.io", Version: "v1", Resource: "gitrepositories"}: "GitRepositoryList", + {Group: "argoproj.io", Version: "v1alpha1", Resource: "applications"}: "ApplicationList", }, objs...) } @@ -87,9 +88,15 @@ func applierTarget(applierSet bool) *configbutleraiv1alpha3.GitTarget { func applierReconciler(source dynamic.Interface, sourceErr error, allowed bool) *GitTargetReconciler { return &GitTargetReconciler{ ApplierClients: func(context.Context, string) (dynamic.Interface, error) { return source, sourceErr }, - newFlux: func(client dynamic.Interface) *applier.Flux { + newApplier: func(kind string, client dynamic.Interface) applier.Client { + canI := func(context.Context, authorizationv1.ResourceAttributes) (bool, error) { return allowed, nil } + if kind == configbutleraiv1alpha3.ApplierKindApplication { + argo := applier.NewArgoCD(client) + argo.CanI = canI + return argo + } flux := applier.NewFlux(client) - flux.CanI = func(context.Context, authorizationv1.ResourceAttributes) (bool, error) { return allowed, nil } + flux.CanI = canI return flux }, } @@ -147,6 +154,74 @@ func TestObserveApplier_Verdicts(t *testing.T) { } } +// applierArgoApplication is an Application argocd/checkout syncing branch of the test repository, +// with a syncPolicy when one is given. +func applierArgoApplication(branch string, syncPolicy map[string]any) runtime.Object { + app := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "argoproj.io/v1alpha1", + "kind": "Application", + "metadata": map[string]any{"namespace": "argocd", "name": "checkout"}, + "spec": map[string]any{ + "source": map[string]any{ + "repoURL": "https://github.com/acme/platform", "targetRevision": branch, "path": "apps", + }, + }, + }} + if syncPolicy != nil { + _ = unstructured.SetNestedMap(app.Object, syncPolicy, "spec", "syncPolicy") + } + return app +} + +// TestObserveApplier_ArgoCDVerdicts: an applicationRef is checked as an Argo CD Application, with +// automated sync on as a verdict of its own. +func TestObserveApplier_ArgoCDVerdicts(t *testing.T) { + cases := []struct { + name string + objects []runtime.Object + allowed bool + wantStatus metav1.ConditionStatus + wantReason string + wantMessage string + }{ + {name: "automated sync off", objects: []runtime.Object{applierArgoApplication("main", nil)}, allowed: true, + wantStatus: metav1.ConditionTrue, wantReason: ReasonSucceeded, + wantMessage: "Application argocd/checkout applies this repository and branch"}, + {name: "automated sync switched off, keeping prune", allowed: true, + objects: []runtime.Object{applierArgoApplication("main", map[string]any{ + "automated": map[string]any{"enabled": false, "prune": true}, + })}, + wantStatus: metav1.ConditionTrue, wantReason: ReasonSucceeded}, + {name: "automated sync on", allowed: true, + objects: []runtime.Object{applierArgoApplication("main", map[string]any{ + "automated": map[string]any{"prune": true}, + })}, + wantStatus: metav1.ConditionFalse, wantReason: applier.ReasonAutomatedSyncEnabled}, + {name: "another branch", objects: []runtime.Object{applierArgoApplication("release", nil)}, allowed: true, + wantStatus: metav1.ConditionFalse, wantReason: applier.ReasonSourceMismatch}, + {name: "no such Application", allowed: true, + wantStatus: metav1.ConditionFalse, wantReason: applier.ReasonApplierNotFound}, + {name: "an identity that may not update it", objects: []runtime.Object{applierArgoApplication("main", nil)}, + wantStatus: metav1.ConditionFalse, wantReason: applier.ReasonForbidden}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + target := applierTarget(true) + target.Spec.Applier = &configbutleraiv1alpha3.GitTargetApplier{ + ApplicationRef: &meta.NamespacedObjectReference{Namespace: "argocd", Name: "checkout"}, + } + + value, present := applierReconciler(applierFakeSource(tc.objects...), nil, tc.allowed). + observeApplier(context.Background(), beginStatus(nil, nil, target), target, applierProvider()) + + require.True(t, present) + assert.Equal(t, tc.wantStatus, value.Status, value.Message) + assert.Equal(t, tc.wantReason, value.Reason) + assert.Contains(t, value.Message, tc.wantMessage) + }) + } +} + // TestObserveApplier_AnUnreachableSourceKeepsThePublishedVerdict: a blip reading the source cluster // says nothing about the applier, so it must not flip a True to Unknown and back on every hiccup. func TestObserveApplier_AnUnreachableSourceKeepsThePublishedVerdict(t *testing.T) { @@ -175,7 +250,7 @@ func TestObserveApplier_AHangingSourceIsBounded(t *testing.T) { _, sawDeadline = ctx.Deadline() return applierFakeSource(applierFluxObjects("main")...), nil }, - newFlux: func(client dynamic.Interface) *applier.Flux { + newApplier: func(_ string, client dynamic.Interface) applier.Client { flux := applier.NewFlux(client) flux.CanI = func(ctx context.Context, _ authorizationv1.ResourceAttributes) (bool, error) { deadline, ok := ctx.Deadline() @@ -231,6 +306,8 @@ func TestApplierReadiness_StallsOnlyWhatNeedsAHuman(t *testing.T) { value: conditionValue{Status: metav1.ConditionFalse, Reason: applier.ReasonForbidden}}, {name: "not checked yet", present: true, value: conditionValue{Status: metav1.ConditionUnknown, Reason: ReasonProgressing}}, + {name: "an Application with automated sync on", present: true, wantStalled: true, + value: conditionValue{Status: metav1.ConditionFalse, Reason: applier.ReasonAutomatedSyncEnabled}}, } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) { diff --git a/internal/controller/gittarget_controller.go b/internal/controller/gittarget_controller.go index e177daeb..8af66998 100644 --- a/internal/controller/gittarget_controller.go +++ b/internal/controller/gittarget_controller.go @@ -159,9 +159,9 @@ type GitTargetReconciler struct { // ApplierConfigured Unknown and status.applier unpublished (tests, and the CLI). ApplierClients applier.SourceClients ApplyRequests ApplyRequestLedger - // newFlux builds the client that checks a Kustomization; nil is applier.NewFlux. A seam for - // tests, whose fake source cluster cannot answer an access review. - newFlux func(dynamic.Interface) *applier.Flux + // newApplier builds the client that checks an applier of a kind; nil is applier.NewClient. A + // seam for tests, whose fake source cluster cannot answer an access review. + newApplier func(kind string, client dynamic.Interface) applier.Client } // clockNow is the reconciler's wall clock. @@ -178,9 +178,10 @@ func (r *GitTargetReconciler) clockNow() time.Time { // +kubebuilder:rbac:groups="",resources=secrets,verbs=get;create;update // +kubebuilder:rbac:groups="",resources=events,verbs=create;patch -// The Flux kinds a spec.applier names are NOT granted here: no GitTarget needs them until one names -// an applier, and an install that does not use Flux should not hold patch rights on Flux objects. -// config/rbac/flux-applier-role.yaml and the chart's applierAccess.flux grant them. +// The Flux and Argo CD kinds a spec.applier names are NOT granted here: no GitTarget needs them +// until one names an applier, and an install that does not use Flux or Argo CD should not hold +// write rights on their objects. config/rbac/flux-applier-role.yaml and argocd-applier-role.yaml, +// and the chart's applierAccess.flux and applierAccess.argocd, grant them. // newGitTargetReadiness starts the accumulator that owns this kind's kstatus trio. Its converged // outcome is what Ready reports when no gate objected. From cc49a4d736bacb0d67effc5d98f4e9ee40871c7b Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 07:24:19 +0000 Subject: [PATCH 33/36] test(e2e): drive the Argo CD corner under the standard, automated sync off The Argo CD specs now run the standard: the Application has automated sync off and is named by spec.applier.applicationRef, and its GitProvider has a receiver Gitea pushes to. They prove that a Git-side push is synced by a ForeignCommit request pinned to that commit (and Argo CD's bookkeeping still makes no commit); that two quick API edits keep the later value with no further commit and no sync; that a refused edit with onRefusal: Reapply is reverted by an Argo CD sync; and that an Application with automated sync on is AutomatedSyncEnabled, stalls its GitTarget, and is never synced by GitOps Reverser. The phases that characterized automated sync (selfHeal losing an edit, ignoreDifferences, selfHeal: false with Argo CD's webhook) are gone with it. Argo CD's webhook stays wired, and is shown only to refresh. The receiver wiring is shared with the Flux applier spec. Co-Authored-By: Claude Opus 5.5 (1M context) --- .../support-boundary/argocd-bi-directional.md | 3 +- docs/spec/e2e-bi-directional-corner.md | 244 +++--- test/e2e/argocd_bi_directional_e2e_test.go | 742 +++++++++--------- test/e2e/flux_applier_e2e_test.go | 71 +- .../bi-directional/argocd-application.tmpl | 29 +- 5 files changed, 494 insertions(+), 595 deletions(-) diff --git a/docs/design/support-boundary/argocd-bi-directional.md b/docs/design/support-boundary/argocd-bi-directional.md index f92e050c..73da3813 100644 --- a/docs/design/support-boundary/argocd-bi-directional.md +++ b/docs/design/support-boundary/argocd-bi-directional.md @@ -32,7 +32,8 @@ is not folklore. All citations are against the Argo CD tree vendored at `external-sources/argo-cd` (`VERSION` = 3.5.0 dev; the e2e installs the v3.4.x -line, which is materially identical for these paths). Exercised end-to-end in +line, which is materially identical for these paths). The configuration it leads +to is exercised end-to-end in [`../../../test/e2e/argocd_bi_directional_e2e_test.go`](../../../test/e2e/argocd_bi_directional_e2e_test.go); corner design in [`../e2e-bi-directional-corner.md`](../../spec/e2e-bi-directional-corner.md). diff --git a/docs/spec/e2e-bi-directional-corner.md b/docs/spec/e2e-bi-directional-corner.md index 3a85d949..00958606 100644 --- a/docs/spec/e2e-bi-directional-corner.md +++ b/docs/spec/e2e-bi-directional-corner.md @@ -302,7 +302,7 @@ devcontainer**: | Operation | kubectl equivalent | | --- | --- | | refresh | a push fires the webhook (§5); no manual refresh needed | -| sync | patch `.operation.sync` on the `Application` (manual-sync phase only) | +| sync | GitOps Reverser sets `.operation` itself; the specs patch it by hand only to deploy the refusal spec's overlay before any `GitTarget` names it | | read state | `.status.sync.status`, `.status.sync.revision`, `.status.operationState.phase`, `.status.conditions` | ### 4. What moves @@ -352,162 +352,98 @@ host — so it matches. A `localhost` ROOT_URL would silently fail to match and app would refresh; that is why this is called out. A matched webhook triggers a **`normal`** refresh (`argo.RefreshApp` with -`RefreshTypeNormal`). For an automated app that is `OutOfSync`, that refresh then -auto-syncs — which is how phases 2 and 3 apply the new commit with no `kubectl` -sync at all. For the manual-sync app in phase 1 the webhook only refreshes; the -apply is still an explicit `.operation` patch. +`RefreshTypeNormal`). The specs run the standard (automated sync off), so the +webhook only refreshes: it makes Argo CD compare sooner, and the specs prove it +never syncs. Every sync is one GitOps Reverser starts, learning of a push through +its own receiver, which the corner registers as a second Gitea webhook. Only Spec E +turns automated sync on, and there the webhook's refresh does auto-sync. ## Test design: `argocd_bi_directional_e2e_test.go` -Four scenarios, sharing one Gitea repo and one `IceCreamOrder` CRD applied via -`applyIceCreamCRD(crdGroupArgoBiDirectional)`. - -**They are implemented as four phases of a single `Ordered` `It`, driving one -Application whose `syncPolicy` is re-applied between phases — not as separate -specs with an Application each.** (Specs A–C plus Spec E below; Spec D remains -unimplemented — see [Not done](#not-done).) Two Applications pointed at the same live path -would each stamp their own tracking-id on the same objects, and by the very -mechanism this spec documents, each would then see the other's id as foreign and -raise `SharedResourceWarning`. The specs would fight. One Application, one order -per phase. - -The Argo `Application` lives in the `argocd` namespace, targets +Every spec runs the standard of [`docs/argocd.md`](../argocd.md): the `Application` +has automated sync off (`automated: {enabled: false, prune: true}`) and is named by +the `GitTarget`'s `spec.applier.applicationRef`, so every sync it runs is one GitOps +Reverser started, pinned to a commit it checked. The earlier phases that ran +automated sync (`selfHeal: true` losing the edit, `ignoreDifferences`, and +`selfHeal: false` with the webhook) are gone with it: they characterized an Argo CD +that synced on its own, which the standard no longer allows. + +Specs A, B and E are three `It`s of one `Ordered` describe, sharing one Gitea repo, +one `IceCreamOrder` CRD (`applyIceCreamCRD(crdGroupArgoBiDirectional)`) and one +`Application`, with one order per spec. Two Applications pointed at the same live +path would each stamp their own tracking-id on the same objects and raise +`SharedResourceWarning`. Spec C has its own describe, repo, namespace and +`Application`. + +The `Application` lives in the `argocd` namespace, targets `spec.destination.namespace = `, `spec.source.path = `, -`targetRevision: main`, `directory.recurse: true`. - -Ordering matters: `SetupRepo` leaves the Gitea repo **empty**, so `main` does not -exist on the remote until something pushes it, and `git pull` cannot fetch a ref -that is not there. The spec therefore seeds the branch first and only then stands -up the reverser pipeline — which also sidesteps the question of whether a -`GitTarget` aimed at an empty repository bootstraps a commit of its own. The Flux -spec depends on the same ordering. - -### Spec A — Argo CD's stamps land in Git - -`syncPolicy: {}` (manual). This is the characterization test: it records exactly -what Argo writes and exactly what survives sanitization. - -1. Commit a clean `order-1` (no Argo metadata). Refresh + sync via `kubectl`. -2. Assert the **live** object has annotation `argocd.argoproj.io/tracking-id` - equal to `:/IceCreamOrder:/order-1` — proves default - `annotation` tracking, exact value format. -3. Assert the **live** object has `kubectl.kubernetes.io/last-applied-configuration` - — proves client-side apply is the default. -4. Assert the **committed file** contains none of it: no - `argocd.argoproj.io/tracking-id`, no `kubectl.kubernetes.io/last-applied-configuration`, - no `managedFields`, no `resourceVersion`. -5. Assert, with `Consistently`, that the Argo sync produced **zero commits**. - -Step 5 is the strongest form of the assertion, and it only holds *because* of the -sanitize fix. If sanitization is complete, the sanitized live object is byte-equal -to the file already in Git, so the Reverser has nothing to write. Before the fix -the tracking-id leaked, the sanitized object differed from Git, and the count grew -by one. The commit count is therefore a sharper detector of a metadata leak than -any substring assertion — it catches keys nobody thought to name. - -(The same shape is what lets the Flux spec assert exact counts: Flux's stamps are -stripped too, so its applies are commit-neutral.) - -### Spec B — `selfHeal: true` destroys the API-side change - -`syncPolicy.automated: {prune: true, selfHeal: true}`. - -1. Git holds `order-2` with `container: Cone`. Sync. Live = `Cone`. -2. Patch the live object through the Kubernetes API: `container: WaffleBowl`. -3. `Eventually` assert live is back to **`Cone`**, then `Consistently` assert it - stays `Cone`. -4. `Eventually` assert the committed file is back to `Cone`; then `Consistently` - assert the commit count is stable. - -Expected timeline, from the verified constants: self-heal's **first** revert is -sub-second (backoff is zero on attempt 0; -[`argocd-bi-directional.md`](../design/support-boundary/argocd-bi-directional.md)), replaying the -stale cached revision. The Reverser commits `WaffleBowl`, then observes the revert -and commits `Cone`. At the 180s refresh Argo finds `Cone` in Git and is already -Synced. **The user's change is lost, and Git history flaps.** - -The flap is **exactly two commits, deterministically** — and the spec asserts that -`+2`. It is *not* a race, despite self-heal being sub-second: the Reverser watches -the object through the Kubernetes API **watch**, which delivers every edit in order -and never collapses them, so it observes both the `WaffleBowl` edit and the `Cone` -revert; with `commitWindow=0` each observed edit finalizes as its own commit. So the -self-heal round trip writes precisely two commits (the change, then its revert), -which the spec asserts alongside the terminal `Cone` state. (This differs from the -Flux spec only in that Flux is driven as a manual applier; both assert exact -counts.) - -### Spec C — split ownership (NOT bi-directional; see the caveat below) - -`syncPolicy.automated: {selfHeal: true}` **plus**: - -```yaml -ignoreDifferences: - - group: argo-bi-directional.e2e.example.com - kind: IceCreamOrder - jsonPointers: ["/spec/scoops"] -syncOptions: ["RespectIgnoreDifferences=true"] -``` - -1. Git holds `order-3` with `scoops: [{Vanilla, 2}]`. Sync. -2. API-patch `scoops: [{MintChip, 4}]`. -3. `Consistently` assert live **stays** `MintChip` despite `selfHeal: true` — - the field is ignored, so no drift, so no heal. -4. Assert the `Application` remains `Synced`. -5. Assert the Reverser commits `MintChip`. -6. After an Argo refresh, assert still `Synced`, still `MintChip`, commit count stable. - -This is the Argo CD analogue of split ownership (mode 3 in -[`bi-directional.md`](../bi-directional.md#the-split-ownership-alternative)) — and it is **not** -bi-directional, which the spec asserts precisely so no one mistakes it for one. -`ignoreDifferences` removes `/spec/scoops` from Argo's comparison in *both* -directions: the API can now own it, but a **Git-side** change to `/spec/scoops` -would no longer reach the cluster either. That is the whole point of split -ownership, and the whole reason it is not the bi-directional answer. The genuine -bi-directional configuration is `selfHeal: false` + the webhook (see -[`argocd-bi-directional.md`](../design/support-boundary/argocd-bi-directional.md)); Spec C exists -to pin the *contrast* — the difference from Spec B is one `ignoreDifferences` -stanza, and it is the difference between the cluster losing the change and the -cluster keeping it by taking the field out of GitOps. - -`RespectIgnoreDifferences=true` is belt-and-braces rather than load-bearing here: -because the Reverser keeps Git current, target and live agree anyway. It closes -the window in which an unrelated Git change triggers a sync *between* the API -patch and the Reverser's commit, which would otherwise reset the field -(`controller/sync.go:250-261`). Documented, not separately asserted. - -### Spec E — the recommended shared-field loop (`selfHeal: false` + webhook) - -`syncPolicy.automated: {prune: true, selfHeal: false}`, no `ignoreDifferences`. - -This is the genuine bi-directional configuration from -[`argocd-bi-directional.md`](../design/support-boundary/argocd-bi-directional.md): the field stays -fully GitOps-managed (unlike Spec C, which carves it out) and changes from **both** -sides. It is the deterministic counterpart to Spec B — nothing reverts live drift, -so every step can be asserted directly, with no timing race. - -1. Git holds `order-4` with `container: Cone`. The webhook drives the auto-sync; - live = `Cone`. -2. **API side.** Patch the live object: `container: WaffleBowl`. -3. `Consistently` assert live **stays** `WaffleBowl` — with `selfHeal: false`, - Argo marks the app `OutOfSync` but does not revert. (Contrast Spec B, where - `selfHeal: true` reverts within a second.) The Reverser committing `WaffleBowl` - fires the webhook, which only ever converges Argo *to* `WaffleBowl`. -4. Assert the Reverser commits `WaffleBowl`. -5. **Git side of the same field.** Commit `container: Cup` (a different valid enum - value — `spec.container` is constrained to `{Cup, Cone, WaffleBowl}`) and push. -6. Assert the webhook drives automated sync to apply it: live becomes `Cup`. This - is the step `ignoreDifferences` (Spec C) would have silently blocked — proof the - shared field is still driven from Git. (Confirmed against Argo's own logs: - `Received push event … touchedHead: true` → `Updated sync status: Synced -> - OutOfSync` → `Initiated automated sync`. The webhook re-resolves the fresh - commit and auto-sync applies it, exactly as the recommended mode promises.) -7. Assert both loops settle in agreement (`Cup` live and committed, commit count - stable). - -Spec B and Spec E together are the payoff of the whole corner: the *only* -difference between them is the `selfHeal` flag, and it is the difference between -the cluster losing a shared-field edit and keeping it while Git can still drive -the same field. +`targetRevision: main`, `directory.recurse: true`. The `GitProvider` has a +`gitea` receiver, registered as a Gitea webhook next to Argo CD's own. + +Ordering matters: `SetupRepo` leaves the Gitea repo **empty**, so the spec seeds +`main` first, creates the `Application` (nothing to sync yet), and only then stands +up the reverser pipeline, whose `GitTarget` needs the `Application` to report +`ApplierConfigured=True`. + +### Spec A — a Git-side push is synced through a pinned `ForeignCommit` + +1. Assert Argo CD has run **no operation**: GitOps Reverser's own bootstrap push, + and its start, ask for nothing. +2. Push a clean `order-1` from the Git side, and nothing else. +3. Assert the `Application`'s operation state is GitOps Reverser's: initiated by + `gitops-reverser` with `automated: true`, reason `ForeignCommit` in its info, + `sync.revision` and `syncResult.revision` both the pushed commit, `Succeeded`, + with resources. The `GitTarget`'s `status.applier` names the same commit. +4. Assert the **live** object carries `argocd.argoproj.io/tracking-id` (exact + value format) and `kubectl.kubernetes.io/last-applied-configuration`, and the + **committed file** carries neither, nor `managedFields` or `resourceVersion`. +5. Assert, with `Consistently`, that the sync produced **zero commits**. + +Step 5 is the strongest form of the bookkeeping assertion, and it only holds +*because* of the sanitize fix: if sanitization is complete, the sanitized live +object is byte-equal to the file already in Git, so the Reverser has nothing to +write. The commit count is a sharper detector of a metadata leak than any +substring assertion — it catches keys nobody thought to name. + +### Spec B — two quick API edits keep the later value + +The race the apply trigger exists for. + +1. Push `order-4` from the Git side; it is synced by a `ForeignCommit`. +2. Patch `container: Cup` through the API, wait for its commit, then patch + `container: WaffleBowl` and wait for that commit. +3. `Consistently` assert live stays `WaffleBowl`, the commit count is exactly + baseline + 2, and the `Application`'s last `finishedAt` has not moved: GitOps + Reverser's own commits owe Argo CD no sync. +4. Assert Argo CD, refreshed by its webhook, reports the `Application` `Synced`: + there was nothing to catch up. + +### Spec C — a refused edit is reverted by an Argo CD sync + +Its own describe, repo and namespace. Corpus shape 8, as in the Flux refusal +spec: an overlay whose `Deployment` comes from `../../base`, and a live edit to an +env var no overlay declaration can express. + +1. Seed the base and overlay, create the `Application` on the overlay + (kustomize), and deploy it with one manual sync, before any `GitTarget` names it. +2. Drift `LOG_LEVEL` to `debug`. +3. Create the `GitTarget` on the overlay with `applicationRef` and + `onRefusal: Reapply`, and a `WatchRule` on `deployments`. +4. Assert the write is refused at the boundary (`WriteBoundaryRefused`), and that + GitOps Reverser's `Refusal` sync, pinned to the tip, reverts `LOG_LEVEL` to + `info`, with no commit for it. + +### Spec E — an `Application` with automated sync on is never synced + +1. Re-apply the `Application` with automated sync on (an `automated` block with no + `enabled`), and request a `GitTarget` reconcile, since the applier is read on + reconcile rather than watched. +2. Assert `ApplierConfigured=False` with `AutomatedSyncEnabled`, and `Stalled`. +3. Push a Git-side change. Assert an `ApplyRequestFailed` Event: the `ForeignCommit` + was decided and refused at the `Application`. +4. `Consistently` assert no operation initiated by `gitops-reverser` names the + pushed commit, and `status.applier` is unchanged. Argo CD's own automated sync + applies the push; that is not GitOps Reverser's. ### Spec D — arming the landmine (phase 2) @@ -619,7 +555,7 @@ neither is `Serial`. | `_argocd-installed` `helm pull`s the chart from the argo-helm repo at install time — a network dependency | Retried 3× with an explicit error message. If CI flakes, mirror the `_ghcr-preflight` node as an `_argocd-preflight`, or vendor the chart tarball | | The chart could ship `NetworkPolicy` objects that k3s enforces | `global.networkPolicy.create` is off by default (we do not enable it), so no NPs are created. Port-forward goes kubelet→pod and bypasses NP anyway | | Argo's app-controller watches cluster-wide and will observe the Flux spec's objects | Only objects carrying a tracking-id are compared. Harmless, but confirm no `SharedResourceWarning` noise across specs | -| Spec B asserts an exact `+2` commit delta | Deterministic, not timing-derived: the Kubernetes watch delivers every edit and `commitWindow=0` makes each its own commit. See Spec B | +| Spec B asserts an exact `+2` commit delta | Deterministic, not timing-derived: the Kubernetes watch delivers every edit, `commitWindow=0` makes each its own commit, and nothing syncs GitOps Reverser's own commits | | A chart bump may move the tracking defaults | The defaults are asserted directly in Spec A. A behaviour change breaks the spec loudly, which is the point. README lists the re-verify checklist | ## Decisions diff --git a/test/e2e/argocd_bi_directional_e2e_test.go b/test/e2e/argocd_bi_directional_e2e_test.go index 8937a763..cb3868a1 100644 --- a/test/e2e/argocd_bi_directional_e2e_test.go +++ b/test/e2e/argocd_bi_directional_e2e_test.go @@ -21,9 +21,15 @@ import ( // Argo CD is only installed for this corner, by the `_argocd-installed` Taskfile // node. `task test-e2e` skips these specs; `task test-e2e-bi-directional` runs them. // -// Everything here is driven through kubectl — refresh via the -// `argocd.argoproj.io/refresh` annotation, sync via the Application's `.operation` -// field — so the devcontainer needs no `argocd` binary. +// Every spec runs the standard (docs/argocd.md): the Application has automated sync +// off and is named by the GitTarget's spec.applier.applicationRef, so every sync it +// runs is one GitOps Reverser started, pinned to a commit it checked. Argo CD's own +// Gitea webhook stays wired, and is shown to be harmless: with automated sync off it +// only refreshes. The last spec turns automated sync on and proves GitOps Reverser +// then refuses to drive the Application. +// +// Everything is driven through kubectl, so the devcontainer needs no `argocd` +// binary. const ( // Argo CD stamps this onto every non-CRD object it applies. Its default // resource-tracking method is `annotation` (NOT `label`), so this annotation — @@ -40,19 +46,19 @@ const ( // argocd-secret; the Gitea hook is created with the same value. argoWebhookSecret = "e2e-argocd-webhook-secret" - // Self-heal reacts to live drift on a ~2s backoff, so 90s is generous. The - // spec never waits on Argo's 180s timed refresh (see setup/argocd/values.yaml). - argoSelfHealTimeout = 90 * time.Second - argoSyncTimeout = 120 * time.Second - // Bounded so it can only pass via the webhook: comfortably longer than webhook - // delivery + refresh + auto-sync, but well under Argo's 180s timed - // reconciliation. If the webhook path is broken, this times out. - argoWebhookSyncTimeout = 20 * time.Second // measured ~4s end-to-end in the corner runs - // Long enough that a self-heal would certainly have fired if the field were not ignored. - argoNoSelfHealWindow = 10 * time.Second // measured: self-heal reverts a non-ignored field in ~1s + // The initiator GitOps Reverser's syncs carry (applier.ArgoCDInitiator). + argoReverserInitiator = "gitops-reverser" + + argoSyncTimeout = 120 * time.Second + // A Git-side push reaches GitOps Reverser's receiver, which decides a + // ForeignCommit on a quiet branch, and Argo CD then syncs: seconds, far below the + // 10m refresher, so this can only pass through the receiver. + argoRequestedSyncTimeout = 90 * time.Second + // Long enough for a sync GitOps Reverser should not have asked for to show up. + argoNoSyncWindow = 20 * time.Second ) -// argoBiDirectionalRepo holds the file-local repo fixtures for this describe block. +// argoBiDirectionalRepo holds the file-local repo fixtures for the main describe block. var argoBiDirectionalRepo *RepoArtifacts type argoBiDirectionalRun struct { @@ -67,20 +73,20 @@ type argoBiDirectionalRun struct { appName string repoSecretName string - gitProviderName string - gitTargetName string - watchRuleName string - livePath string + gitProviderName string + gitTargetName string + watchRuleName string + receiverSecretName string + livePath string - // One order per phase, so the phases never contend for the same object. - syncedOrderName string - selfHealOrderName string - ignoredOrderName string - recommendedOrderName string + // One order per spec, so the specs never contend for the same object. + syncedOrderName string + editedOrderName string + automatedOrderName string } -// argoAppConfig is the knob set the spec turns between phases. The Application is -// re-applied (not patched) each time, so dropping a field really removes it. +// argoAppConfig is the knob set the specs turn. The Application is re-applied (not +// patched) each time, so dropping a field really removes it. type argoAppConfig struct { Name string ArgoNamespace string @@ -89,19 +95,16 @@ type argoAppConfig struct { Path string DestinationNamespace string + // Kustomize renders the path as a kustomization instead of a recursive directory. + Kustomize bool + // Automated turns automated sync ON, which the standard forbids. Automated bool - Prune bool - SelfHeal bool - - SyncOptions []string - IgnoreGroup string - IgnoreKind string - IgnoreJSONPointers []string } var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), Ordered, func() { var run argoBiDirectionalRun var testNs string + var receiverHookID int64 BeforeAll(func() { skipUnlessBiDirectionalEnabled() @@ -123,7 +126,7 @@ var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") applySOPSAgeKeyToNamespace(testNs) - run = newArgoBiDirectionalRun(testNs) + run = newArgoBiDirectionalRun(testNs, argoBiDirectionalRepo, "argo-bi") run.assertCheckoutReady() By("installing the IceCreamOrder CRD directly") @@ -136,7 +139,7 @@ var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), By("registering the Gitea repository with Argo CD") run.applyArgoRepoSecret() - By("wiring a Gitea -> Argo CD webhook so pushes reconcile without waiting for the poll") + By("wiring a Gitea -> Argo CD webhook, which with automated sync off only refreshes") run.configureArgoWebhookSecret() run.ensureArgoWebhook() }) @@ -150,59 +153,37 @@ var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), // so the IceCreamOrders die with the namespace. cleanupNamespacedResource(run.argoNs, "application.argoproj.io", run.appName) cleanupNamespacedResource(run.argoNs, "secret", run.repoSecretName) + deleteRepoHook(run.repoName, receiverHookID) cleanupWatchRule(run.watchRuleName, testNs) cleanupGitTarget(run.gitTargetName, testNs) cleanupNamespacedResource(testNs, "gitprovider", run.gitProviderName) + cleanupNamespacedResource(testNs, "secret", run.receiverSecretName) cleanupClusterResource("crd", iceCreamCRDName(crdGroupArgoBiDirectional)) cleanupNamespace(testNs) }) - It("should keep Argo CD's bookkeeping out of Git, and prove selfHeal on vs off is the difference "+ - "between losing and keeping a shared-field edit", func() { - // Ordering is load-bearing, and both halves of it are. - // - // 1. SetupRepo leaves the Gitea repo EMPTY. There is no `main` on the - // remote until something pushes one, and `git pull` cannot fetch a ref - // that does not exist. So seed the branch first. - // - // 2. The reverser's first reconcile makes its managed path match LIVE - // state: it writes its bootstrap files and DELETES anything it does not - // own. An IceCreamOrder committed before that reconcile is pruned before - // Argo CD ever sees it — Argo then syncs an empty path and truthfully - // reports Succeeded, having applied nothing. - // - // So: seed the branch, let the reverser settle, and only then write orders. - // The Flux spec depends on exactly the same sequence. - By("seeding the branch so the GitTarget has a main to bootstrap") + // A Git-side push reaches the cluster only through GitOps Reverser: its receiver + // hears of the push, the worker decides a ForeignCommit on a quiet branch, and the + // sync it starts is pinned to that commit. Nothing GitOps Reverser pushes itself + // is synced. And what Argo CD stamps on the objects it applies stays out of Git. + It("syncs a Git-side push through a pinned ForeignCommit, and keeps Argo CD's bookkeeping out of Git", func() { + // SetupRepo leaves the Gitea repo EMPTY, so seed the branch first: the + // GitTarget needs a main to bootstrap. Argo CD has nothing to sync until the + // Git-side push below, and with automated sync off it would not sync it anyway. + By("seeding the branch, and creating the Application with automated sync off") Expect(run.commitAllAndPush("argo bi-directional: seed main")).To(Succeed()) + run.applyArgoApp(run.baseAppConfig()) - By("enabling gitops-reverser on the shared live path") - run.startReverserPipeline() + By("enabling gitops-reverser on the shared live path, naming the Application as the applier") + receiverHookID = run.startReverserPipeline() - // Absorbs the reverser's bootstrap/prune commit, so the zero-churn - // assertions below measure Argo CD's effect and nothing else. + // Absorbs the reverser's bootstrap commit, so the counts below measure the + // Git-side push and Argo CD's sync, and nothing else. baselineCommitCount := run.waitForStableRemoteCommitCount(biStableCountShortWait) + Expect(run.operationFinishedAt()).To(BeEmpty(), + "GitOps Reverser's own pushes and its start ask Argo CD for nothing") - // --------------------------------------------------------------------- - // Phase 1 — Argo CD's stamps must not reach Git. - // - // Argo's repo-server stamps `argocd.argoproj.io/tracking-id` onto every - // non-CRD manifest it renders, and its default client-side apply writes - // `last-applied-configuration`. Both land on the LIVE object, which is - // exactly what gitops-reverser mirrors back to Git. - // - // Neither is user intent, so neither may be committed. Regression guard for - // internal/sanitize: before that fix, the tracking-id was committed, which - // (a) polluted Git with a provenance string naming one Application, and - // (b) armed a real failure — Argo's GetAppName never validates the id - // against the object, so a manifest carrying a foreign tracking-id makes a - // second Application fail to sync with "Shared resource found". - // - // The strongest form of the assertion is a commit COUNT: if sanitization is - // complete, the sanitized live object equals the file already in Git, so - // Argo syncing a reverser-watched path produces ZERO commits. - // --------------------------------------------------------------------- - By("committing a clean IceCreamOrder through normal GitOps") + By("pushing a clean IceCreamOrder from the Git side, and nothing else") run.writeLiveOrder(iceCreamOrderFile{ Name: run.syncedOrderName, Namespace: testNs, @@ -212,13 +193,16 @@ var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), Toppings: []string{"Sprinkles"}, }) Expect(run.commitAllAndPush("argo bi-directional: add icecream order")).To(Succeed()) - syncedHead := run.gitHEAD() - run.expectRemoteCommitCount(baselineCommitCount + 1) + head := run.gitHEAD() - By("creating a manually-synced Argo CD Application for the shared live path") - run.applyArgoApp(run.baseAppConfig()) - run.syncToRevision(syncedHead) + By("GitOps Reverser asks Argo CD to sync exactly that commit, as a ForeignCommit") + run.waitForReverserSync(head, "ForeignCommit") run.waitForOrderContainer(run.syncedOrderName, "Cone") + Eventually(func(g Gomega) { + reason, commit := run.gitTargetApplierStatus(g) + g.Expect(reason).To(Equal("ForeignCommit")) + g.Expect(commit).To(Equal(head)) + }, biEventuallyTimeout, biPollInterval).Should(Succeed()) By("verifying Argo CD stamped its bookkeeping onto the live object") liveAnnotations := run.liveOrderAnnotations(run.syncedOrderName) @@ -228,270 +212,210 @@ var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), Expect(liveAnnotations).To(HaveKey(kubectlLastAppliedAnnotation), "Argo CD's default apply is client-side, which writes last-applied-configuration") - By("verifying none of that bookkeeping is committed, and no commit happens at all") + By("verifying none of that bookkeeping is committed, and the sync made no commit at all") // If any of it leaked, the sanitized live object would differ from the file // already in Git and the reverser would commit — so the count would grow. run.consistentlyExpectRemoteCommitCount(baselineCommitCount+1, biStableCountLongWait) - committed := run.readCommittedOrder(run.syncedOrderName) Expect(committed).NotTo(ContainSubstring(argoTrackingIDAnnotation), "Argo CD's tracking-id is controller bookkeeping and must never reach Git") Expect(committed).NotTo(ContainSubstring(kubectlLastAppliedAnnotation)) Expect(committed).NotTo(ContainSubstring("managedFields")) Expect(committed).NotTo(ContainSubstring("resourceVersion")) + }) - By("re-syncing Argo CD with no Git change and expecting no churn") - run.syncToRevision(syncedHead) - run.consistentlyExpectRemoteCommitCount(baselineCommitCount+1, biStableCountMediumWait) - - // --------------------------------------------------------------------- - // Phase 2 — selfHeal destroys the API-side change, and Git history flaps. - // - // Two clocks: self-heal reverts live drift essentially immediately — - // sub-second on the FIRST drift, because the backoff is zero on attempt 0 - // (docs/design/support-boundary/argocd-bi-directional.md) — while a new Git revision is - // only noticed on the 180s timed refresh. So Argo replays its stale cached - // revision long before it ever looks at what the reverser committed. The - // user's change is lost. - // - // The flap is TWO commits in quick succession: the reverser captures the API - // edit (WaffleBowl), then self-heal reverts to the Git-side value and the - // reverser captures THAT too (Cone). This is deterministic, not a race: the - // Kubernetes API watch the reverser runs delivers every edit in order and - // never collapses them, and with commitWindow=0 each edit is its own commit. - // So even though self-heal is sub-second, both edits are observed and both - // are committed — asserted as an exact +2 delta at the end of the phase. - // --------------------------------------------------------------------- - By("switching Argo CD to automated sync with selfHeal enabled") - selfHealApp := run.baseAppConfig() - selfHealApp.Automated = true - selfHealApp.Prune = true - selfHealApp.SelfHeal = true - run.applyArgoApp(selfHealApp) - - By("committing a second IceCreamOrder through normal GitOps") + // The race the apply trigger exists for (docs/design/apply-trigger.md): two API + // edits to one field a few seconds apart. With automated sync, Argo CD synced the + // first edit's commit over the second edit. Under the standard nothing syncs + // GitOps Reverser's own commits at all, so the later value survives, Git ends on + // it with no further commit, and Argo CD, refreshed by its webhook on each push, + // only reports the Application Synced. + It("keeps the later of two quick API edits, with no further commit and no sync", func() { + By("committing a third IceCreamOrder from the Git side, synced by a ForeignCommit") run.writeLiveOrder(iceCreamOrderFile{ - Name: run.selfHealOrderName, + Name: run.editedOrderName, Namespace: testNs, - CustomerName: "Bob", + CustomerName: "Dana", Container: "Cone", - Scoops: []iceCreamScoop{{Flavor: "Strawberry", Quantity: 1}}, - Toppings: []string{"WhippedCream"}, + Scoops: []iceCreamScoop{{Flavor: "Vanilla", Quantity: 2}}, + Toppings: []string{"Sprinkles"}, }) - Expect(run.commitAllAndPush("argo bi-directional: add self-heal icecream order")).To(Succeed()) - - By("letting the Gitea -> Argo CD webhook drive the sync — no manual refresh, no manual sync") - // The push notifies argocd-server, which refreshes the automated app and - // auto-syncs the new commit. This lands well inside argoWebhookSyncTimeout, - // far below Argo's 180s poll — so it can only pass via the webhook. - Eventually(func(g Gomega) { - value, err := run.orderContainer(run.selfHealOrderName) - g.Expect(err).NotTo(HaveOccurred()) - g.Expect(value).To(Equal("Cone")) - }, argoWebhookSyncTimeout, biPollInterval).Should(Succeed(), - "the webhook should drive Argo CD to auto-sync the new commit") - - By("recording the commit count just before the API edit, to observe the self-heal flap") - Expect(run.gitPull()).To(Succeed()) - preFlapCount, err := run.gitMainCommitCount() - Expect(err).NotTo(HaveOccurred()) - - By("changing that IceCreamOrder through the Kubernetes API") - run.patchOrderContainer(run.selfHealOrderName, "WaffleBowl") - - By("verifying Argo CD self-heals the change away — the API-side edit is LOST") - Eventually(func(g Gomega) { - g.Expect(run.orderContainer(run.selfHealOrderName)).To(Equal("Cone")) - }, argoSelfHealTimeout, biPollInterval).Should(Succeed(), - "selfHeal should replay the stale cached revision over the API-side change") + Expect(run.commitAllAndPush("argo bi-directional: add the order two edits will race on")).To(Succeed()) + run.waitForReverserSync(run.gitHEAD(), "ForeignCommit") + run.waitForOrderContainer(run.editedOrderName, "Cone") + baseline := run.waitForStableRemoteCommitCount(biStableCountMediumWait) + lastSync := run.operationFinishedAt() + + By("editing the order through the API, and again seconds after that edit is pushed") + run.patchOrderContainer(run.editedOrderName, "Cup") + run.waitForCommittedOrderContainer(run.editedOrderName, "Cup") + run.patchOrderContainer(run.editedOrderName, "WaffleBowl") + run.waitForCommittedOrderContainer(run.editedOrderName, "WaffleBowl") + + By("the later value survives, with no commit after it, and Argo CD is never asked to sync") Consistently(func(g Gomega) { - g.Expect(run.orderContainer(run.selfHealOrderName)).To(Equal("Cone")) - }, biStableCountMediumWait, biPollInterval).Should(Succeed()) + g.Expect(run.orderContainer(run.editedOrderName)).To(Equal("WaffleBowl")) + g.Expect(run.gitPull()).To(Succeed()) + count, err := run.gitMainCommitCount() + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(count).To(Equal(baseline+2), + "a sync of the first edit's commit over the second edit would show as more commits") + g.Expect(run.operationFinishedAt()).To(Equal(lastSync), + "GitOps Reverser's own commits owe Argo CD no sync: there is no catch-up") + }, 30*time.Second, 2*time.Second).Should(Succeed()) - By("verifying Git converges back to the Git-side value and then stops moving") + By("and Argo CD, which only compared, finds Git and the cluster in agreement") Eventually(func(g Gomega) { - g.Expect(run.gitPull()).To(Succeed()) - g.Expect(run.readCommittedOrder(run.selfHealOrderName)).To(ContainSubstring("container: Cone")) - }, biEventuallyTimeout, biPollInterval).Should(Succeed()) + g.Expect(run.syncStatus(g)).To(Equal("Synced")) + }, argoSyncTimeout, biPollInterval).Should(Succeed()) + }) - settled, err := run.gitMainCommitCount() - Expect(err).NotTo(HaveOccurred()) - run.consistentlyExpectRemoteCommitCount(settled, biStableCountMediumWait) - - // The flap is exactly two commits, and this is deterministic — not a race. - // gitops-reverser watches the IceCreamOrder through the Kubernetes API watch, - // which delivers EVERY edit in order and never collapses them: it sees the - // API edit (WaffleBowl), then self-heal's revert to the Git-side value (Cone). - // With commitWindow=0 each observed edit finalizes as its own commit, so the - // self-heal round trip always writes precisely two commits. - By(fmt.Sprintf("verifying the self-heal flap wrote exactly two commits (%d -> %d)", preFlapCount, settled)) - Expect(settled-preFlapCount).To(Equal(2), - "the watch delivers both the API edit and self-heal's revert, and each is one commit") - - // --------------------------------------------------------------------- - // Phase 3 — the safe recipe: ignoreDifferences gives a field to the API. - // - // Identical to phase 2 except for `ignoreDifferences` on /spec/scoops. That - // one stanza is the difference between losing the change and keeping it: - // the field never registers as drift, so selfHeal never fires, and the - // reverser publishes the API-side value to Git. - // - // `RespectIgnoreDifferences=true` is belt-and-braces here rather than - // load-bearing: because the reverser keeps Git current, target and live - // agree anyway. It closes the window where an unrelated Git change triggers - // a sync between the API patch and the reverser's commit, which would - // otherwise reset the ignored field. - // --------------------------------------------------------------------- - By("giving /spec/scoops to the API via ignoreDifferences, keeping selfHeal on") - ignoredApp := run.baseAppConfig() - ignoredApp.Automated = true - ignoredApp.Prune = true - ignoredApp.SelfHeal = true - ignoredApp.SyncOptions = []string{"RespectIgnoreDifferences=true"} - ignoredApp.IgnoreGroup = crdGroupArgoBiDirectional - ignoredApp.IgnoreKind = "IceCreamOrder" - ignoredApp.IgnoreJSONPointers = []string{"/spec/scoops"} - run.applyArgoApp(ignoredApp) - - By("committing a third IceCreamOrder through normal GitOps") + // The standard is full control, so an Application with automated sync on is a + // configuration error, not a mode: the GitTarget reports it and stalls, and GitOps + // Reverser refuses every request to it. Argo CD's automated sync still applies the + // push on its own; what must never appear is a sync GitOps Reverser started. + It("never syncs an Application with automated sync on", func() { + By("turning automated sync on") + automated := run.baseAppConfig() + automated.Automated = true + run.applyArgoApp(automated) + // The applier is read on the GitTarget's reconcile, not watched; a converged + // target reconciles every few minutes, so ask for one now. + _, err := kubectlRunInNamespace(testNs, "annotate", "gittarget", run.gitTargetName, "--overwrite", + "reconcile.configbutler.ai/requestedAt="+strconv.FormatInt(time.Now().UnixNano(), 10)) + Expect(err).NotTo(HaveOccurred(), "failed to request a GitTarget reconcile") + verifyResourceCondition("gittarget", run.gitTargetName, testNs, "ApplierConfigured", "False", + "AutomatedSyncEnabled", "has automated sync on") + verifyResourceCondition("gittarget", run.gitTargetName, testNs, "Stalled", "True", + "AutomatedSyncEnabled", "") + + _, requestedBefore := run.gitTargetApplierStatus(Default) + + By("pushing a Git-side change, which GitOps Reverser hears of and refuses to sync") run.writeLiveOrder(iceCreamOrderFile{ - Name: run.ignoredOrderName, + Name: run.automatedOrderName, Namespace: testNs, - CustomerName: "Charlie", - Container: "Cup", - Scoops: []iceCreamScoop{{Flavor: "Vanilla", Quantity: 2}}, - Toppings: []string{"HotFudge"}, + CustomerName: "Bob", + Container: "Cone", + Scoops: []iceCreamScoop{{Flavor: "Strawberry", Quantity: 1}}, + Toppings: []string{"WhippedCream"}, }) - Expect(run.commitAllAndPush("argo bi-directional: add split-ownership icecream order")).To(Succeed()) - - By("letting the webhook drive the sync of the third order") + Expect(run.commitAllAndPush("argo bi-directional: a push under automated sync")).To(Succeed()) + head := run.gitHEAD() Eventually(func(g Gomega) { - value, err := run.orderScoopFlavor(run.ignoredOrderName) - g.Expect(err).NotTo(HaveOccurred()) - g.Expect(value).To(Equal("Vanilla")) - }, argoWebhookSyncTimeout, biPollInterval).Should(Succeed(), - "the webhook should drive Argo CD to auto-sync the third commit") - - By("changing the ignored field through the Kubernetes API") - run.patchOrderScoop(run.ignoredOrderName, "MintChip", 4) + failed := gitTargetEventMessages(g, testNs, run.gitTargetName, "ApplyRequestFailed") + g.Expect(failed).To(ContainSubstring("(ForeignCommit)")) + g.Expect(failed).To(ContainSubstring("has automated sync on")) + }, argoRequestedSyncTimeout, biPollInterval).Should(Succeed(), + "the ForeignCommit is decided, and refused at the Application") - By("verifying selfHeal does NOT revert an ignored field") + By("no sync GitOps Reverser started ever names that commit") Consistently(func(g Gomega) { - g.Expect(run.orderScoopFlavor(run.ignoredOrderName)).To(Equal("MintChip")) - }, argoNoSelfHealWindow, biPollInterval).Should(Succeed(), - "ignoreDifferences should keep /spec/scoops out of the drift comparison") + initiator, revision := run.lastOperation(g) + if initiator == argoReverserInitiator { + g.Expect(revision).NotTo(Equal(head), + "GitOps Reverser must not sync an Application that syncs on its own") + } + _, requested := run.gitTargetApplierStatus(g) + g.Expect(requested).To(Equal(requestedBefore), "no request was delivered") + }, argoNoSyncWindow, biPollInterval).Should(Succeed()) + }) +}) - By("verifying Argo CD still reports the Application as Synced") - Expect(run.syncStatus()).To(Equal("Synced"), - "an ignored field must not put the Application OutOfSync") +// A refused edit on a target with onRefusal: Reapply is reverted by an Argo CD sync, +// as it is by a Flux Kustomization in refusal_reapply_e2e_test.go, and with the same +// refusal: corpus shape 8, a live edit to an env var a kustomize base owns, which no +// declaration in the overlay can express. Its own repository, Application and +// namespace, because a second Application on the first spec's path would fight it +// over the tracking-id. +var _ = Describe("Bi Directional (Argo CD) refusal", Label("bi-directional", "argocd"), Ordered, func() { + const overlayPath = "apps/checkout/overlays/prod" + var ( + run argoBiDirectionalRun + testNs string + repo *RepoArtifacts + ) - By("verifying gitops-reverser publishes the API-side value to Git") - Eventually(func(g Gomega) { - g.Expect(run.gitPull()).To(Succeed()) - g.Expect(run.readCommittedOrder(run.ignoredOrderName)).To(ContainSubstring("flavor: MintChip")) - }, biEventuallyTimeout, biPollInterval).Should(Succeed()) + BeforeAll(func() { + skipUnlessBiDirectionalEnabled() + requireArgoCDInstalled() - By("verifying the system is at rest") - finalCount, err := run.gitMainCommitCount() - Expect(err).NotTo(HaveOccurred()) - run.consistentlyExpectRemoteCommitCount(finalCount, biStableCountMediumWait) - Expect(run.orderScoopFlavor(run.ignoredOrderName)).To(Equal("MintChip")) - - // --------------------------------------------------------------------- - // Phase 4 — the recommended shared-field loop: selfHeal:false + webhook. - // - // This is the ONE Argo CD configuration docs/bi-directional.md recommends - // for a genuinely shared field (docs/design/support-boundary/argocd-bi-directional.md). - // It is the deterministic counterpart to phase 2: phase 2 proved selfHeal - // LOSES the API edit; this proves selfHeal:false KEEPS it, and — unlike the - // ignoreDifferences of phase 3 — the field stays fully GitOps-driven, so a - // Git-side change to the SAME field still reaches the cluster. - // - // One field, both directions: - // - API side: an edit to spec.container is captured to Git and, because - // selfHeal is off, never reverted; - // - Git side: a later commit to the SAME spec.container is applied back - // to the cluster through the webhook. - // - // No race here: nothing reverts live drift, so every step is deterministic - // and can be asserted directly. - // --------------------------------------------------------------------- - By("switching Argo CD to automated sync with selfHeal DISABLED (the recommended shared-field mode)") - recommendedApp := run.baseAppConfig() - recommendedApp.Automated = true - recommendedApp.Prune = true - recommendedApp.SelfHeal = false - run.applyArgoApp(recommendedApp) - - By("committing a fourth IceCreamOrder through normal GitOps") - run.writeLiveOrder(iceCreamOrderFile{ - Name: run.recommendedOrderName, - Namespace: testNs, - CustomerName: "Dana", - Container: "Cone", - Scoops: []iceCreamScoop{{Flavor: "Vanilla", Quantity: 2}}, - Toppings: []string{"Sprinkles"}, - }) - Expect(run.commitAllAndPush("argo bi-directional: add shared-field selfHeal-off order")).To(Succeed()) + testNs = testNamespaceFor("argo-refusal") + _, _ = kubectlRun("create", "namespace", testNs) + repo = SetupRepo(resolveE2EContext(), testNs, fmt.Sprintf("e2e-argo-refusal-%d", GinkgoRandomSeed())) + _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to test namespace") + applySOPSAgeKeyToNamespace(testNs) - By("letting the webhook drive the sync of the fourth order") - Eventually(func(g Gomega) { - value, containerErr := run.orderContainer(run.recommendedOrderName) - g.Expect(containerErr).NotTo(HaveOccurred()) - g.Expect(value).To(Equal("Cone")) - }, argoWebhookSyncTimeout, biPollInterval).Should(Succeed(), - "the webhook should drive Argo CD to auto-sync the fourth commit") - - // --- API side of the shared field: captured, not reverted --- - By("changing the shared field spec.container through the Kubernetes API") - run.patchOrderContainer(run.recommendedOrderName, "WaffleBowl") - - By("verifying selfHeal:false leaves the API edit in place (the contrast with phase 2)") - // Long enough that a self-heal would certainly have fired had it been on; - // the reverser committing WaffleBowl (which fires the webhook) only ever - // converges Argo TO WaffleBowl, never reverts it. - Consistently(func(g Gomega) { - g.Expect(run.orderContainer(run.recommendedOrderName)).To(Equal("WaffleBowl")) - }, argoNoSelfHealWindow, biPollInterval).Should(Succeed(), - "with selfHeal off, Argo must not revert the live edit before the reverser captures it") + run = newArgoBiDirectionalRun(testNs, repo, "argo-refusal") + run.livePath = overlayPath + run.applyArgoRepoSecret() + createGitProviderWithURLInNamespace(run.gitProviderName, testNs, repo.GitSecretHTTP, run.repoURL) + verifyResourceStatus("gitprovider", run.gitProviderName, testNs, "True", "Succeeded", "") + }) - By("verifying gitops-reverser captures the API edit to Git") - Eventually(func(g Gomega) { - g.Expect(run.gitPull()).To(Succeed()) - g.Expect(run.readCommittedOrder(run.recommendedOrderName)).To(ContainSubstring("container: WaffleBowl")) - }, biEventuallyTimeout, biPollInterval).Should(Succeed()) + AfterAll(func() { + if !biDirectionalEnabled() { + return + } + cleanupNamespacedResource(run.argoNs, "application.argoproj.io", run.appName) + cleanupNamespacedResource(run.argoNs, "secret", run.repoSecretName) + cleanupWatchRule(run.watchRuleName, testNs) + cleanupGitTarget(run.gitTargetName, testNs) + cleanupNamespacedResource(testNs, "gitprovider", run.gitProviderName) + cleanupNamespace(testNs) + }) - // --- Git side of the SAME shared field: applied back through the webhook --- - By("changing the SAME field spec.container from the Git side and pushing") - run.writeLiveOrder(iceCreamOrderFile{ - Name: run.recommendedOrderName, - Namespace: testNs, - CustomerName: "Dana", - // A different valid enum value from the API-side WaffleBowl and the - // initial Cone; spec.container is constrained to {Cup, Cone, WaffleBowl}. - Container: "Cup", - Scoops: []iceCreamScoop{{Flavor: "Vanilla", Quantity: 2}}, - Toppings: []string{"Sprinkles"}, - }) - Expect(run.commitAllAndPush("argo bi-directional: git-side change to the shared field")).To(Succeed()) + It("reverts a refused edit with an Argo CD sync", func() { + By("seeding a base-and-overlay folder that renders into this test's namespace") + seedFolder := writeBaseOwnedFieldFolder(testNs) + DeferCleanup(func() { _ = os.RemoveAll(seedFolder) }) + seedRenderedFolderIntoRepo(repo, testNs, seedFolder, "apps/checkout") + Expect(run.gitPull()).To(Succeed()) - By("verifying the webhook applies the Git-side change to the cluster — the field is still GitOps-driven") + // Before any GitTarget names it, the Application is deployed the way a human + // would: one manual sync. Every sync after this one is GitOps Reverser's. + By("deploying the overlay once with a manual sync, automated sync off") + app := run.baseAppConfig() + app.Kustomize = true + run.applyArgoApp(app) + run.syncToRevision(remoteHeadSHA(Default, repo.CheckoutDir)) Eventually(func(g Gomega) { - value, containerErr := run.orderContainer(run.recommendedOrderName) - g.Expect(containerErr).NotTo(HaveOccurred()) - g.Expect(value).To(Equal("Cup")) - }, argoWebhookSyncTimeout, biPollInterval).Should(Succeed(), - "a Git-side change to the shared field must reach the cluster (ignoreDifferences would have blocked it)") - - By("verifying both loops settle in agreement on the shared field") + g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info")) + }, 2*time.Minute, 2*time.Second).Should(Succeed()) + + By("drifting the base-owned env var away from what Git says") + _, err := kubectlRunInNamespace(testNs, "set", "env", "deployment/checkout", "LOG_LEVEL=debug") + Expect(err).NotTo(HaveOccurred(), "failed to drift the base-owned env var") + emptyBefore := emptyCommitCount(repo.CheckoutDir) + + By("creating the GitTarget on the overlay, naming the Application and asking it to reapply") + createGitTarget(run.gitTargetName, testNs, run.gitProviderName, overlayPath, "main") + _, err = kubectlRunInNamespace(testNs, "patch", "gittarget", run.gitTargetName, "--type=merge", "-p", + fmt.Sprintf(`{"spec":{"applier":{"applicationRef":{"namespace":%q,"name":%q}},"onRefusal":"Reapply"}}`, + run.argoNs, run.appName)) + Expect(err).NotTo(HaveOccurred(), "failed to set spec.applier and spec.onRefusal") + verifyResourceCondition("gittarget", run.gitTargetName, testNs, "ApplierConfigured", "True", "Succeeded", "") + Expect(applyFromTemplate("test/e2e/templates/manager/watchrule-resources.tmpl", struct { + Name string + Namespace string + DestinationName string + Resources string + }{Name: run.watchRuleName, Namespace: testNs, DestinationName: run.gitTargetName, + Resources: `"deployments"`}, + testNs)).To(Succeed(), "failed to apply the WatchRule") + + By("the write is refused at the boundary, and Argo CD is asked to sync the tip") + waitForGitTargetGitPathRefused(run.gitTargetName, testNs, "WriteBoundaryRefused") + run.waitForReverserSync("", "Refusal") + _, pinned := run.lastOperation(Default) + Expect(pinned).To(Equal(remoteHeadSHA(Default, repo.CheckoutDir)), "the sync applies the tip") Eventually(func(g Gomega) { - g.Expect(run.gitPull()).To(Succeed()) - g.Expect(run.readCommittedOrder(run.recommendedOrderName)).To(ContainSubstring("container: Cup")) - }, biEventuallyTimeout, biPollInterval).Should(Succeed()) - recommendedSettled, err := run.gitMainCommitCount() - Expect(err).NotTo(HaveOccurred()) - run.consistentlyExpectRemoteCommitCount(recommendedSettled, biStableCountMediumWait) - Expect(run.orderContainer(run.recommendedOrderName)).To(Equal("Cup")) + g.Expect(checkoutLogLevel(g, testNs)).To(Equal("info"), + "a refused edit on a Reapply target must be reverted by its Application") + }, 3*time.Minute, 2*time.Second).Should(Succeed()) + Expect(emptyCommitCount(repo.CheckoutDir)).To(Equal(emptyBefore), + "the request goes to the applier; nothing moves the branch for it") }) }) @@ -513,25 +437,25 @@ func argoNamespace() string { return "argocd" } -func newArgoBiDirectionalRun(testNs string) argoBiDirectionalRun { +func newArgoBiDirectionalRun(testNs string, repo *RepoArtifacts, prefix string) argoBiDirectionalRun { testID := strconv.FormatInt(time.Now().UnixNano(), 10) return argoBiDirectionalRun{ - gitCheckout: newGitCheckout(argoBiDirectionalRepo, testNs), - testID: testID, - testNs: testNs, - argoNs: argoNamespace(), - repoName: argoBiDirectionalRepo.RepoName, - repoURL: argoBiDirectionalRepo.RepoURLHTTP, - appName: fmt.Sprintf("argo-bi-app-%s", testID), - repoSecretName: fmt.Sprintf("argo-bi-repo-%s", testID), - gitProviderName: fmt.Sprintf("argo-bi-provider-%s", testID), - gitTargetName: fmt.Sprintf("argo-bi-target-%s", testID), - watchRuleName: fmt.Sprintf("argo-bi-watchrule-%s", testID), - livePath: fmt.Sprintf("argo-bi-directional/%s/live", testID), - syncedOrderName: fmt.Sprintf("argo-alice-order-%s", testID), - selfHealOrderName: fmt.Sprintf("argo-bob-order-%s", testID), - ignoredOrderName: fmt.Sprintf("argo-charlie-order-%s", testID), - recommendedOrderName: fmt.Sprintf("argo-dana-order-%s", testID), + gitCheckout: newGitCheckout(repo, testNs), + testID: testID, + testNs: testNs, + argoNs: argoNamespace(), + repoName: repo.RepoName, + repoURL: repo.RepoURLHTTP, + appName: fmt.Sprintf("%s-app-%s", prefix, testID), + repoSecretName: fmt.Sprintf("%s-repo-%s", prefix, testID), + gitProviderName: fmt.Sprintf("%s-provider-%s", prefix, testID), + gitTargetName: fmt.Sprintf("%s-target-%s", prefix, testID), + watchRuleName: fmt.Sprintf("%s-watchrule-%s", prefix, testID), + receiverSecretName: fmt.Sprintf("%s-receiver-%s", prefix, testID), + livePath: fmt.Sprintf("argo-bi-directional/%s/live", testID), + syncedOrderName: fmt.Sprintf("argo-alice-order-%s", testID), + editedOrderName: fmt.Sprintf("argo-dana-order-%s", testID), + automatedOrderName: fmt.Sprintf("argo-bob-order-%s", testID), } } @@ -564,15 +488,22 @@ func (r argoBiDirectionalRun) waitForCRDEstablished() { }, 20*time.Second, biPollInterval).Should(Succeed()) } -// startReverserPipeline brings up GitProvider -> GitTarget -> WatchRule on the -// shared live path. Called after the branch exists, not in BeforeAll. -func (r argoBiDirectionalRun) startReverserPipeline() { +// startReverserPipeline brings up GitProvider (with a receiver Gitea pushes to) -> +// GitTarget (naming the Application as its applier) -> WatchRule on the shared live +// path. Called after the branch and the Application exist. It returns the receiver +// hook's id. +func (r argoBiDirectionalRun) startReverserPipeline() int64 { GinkgoHelper() createGitProviderWithURLInNamespace( r.gitProviderName, r.testNs, argoBiDirectionalRepo.GitSecretHTTP, r.repoURL) verifyResourceStatus("gitprovider", r.gitProviderName, r.testNs, "True", "Succeeded", "") + hookID := registerOperatorReceiver(r.testNs, r.gitProviderName, r.receiverSecretName, r.repoName) createGitTarget(r.gitTargetName, r.testNs, r.gitProviderName, r.livePath, "main") + _, err := kubectlRunInNamespace(r.testNs, "patch", "gittarget", r.gitTargetName, "--type=merge", "-p", + fmt.Sprintf(`{"spec":{"applier":{"applicationRef":{"namespace":%q,"name":%q}}}}`, r.argoNs, r.appName)) + Expect(err).NotTo(HaveOccurred(), "failed to set spec.applier") + verifyResourceCondition("gittarget", r.gitTargetName, r.testNs, "ApplierConfigured", "True", "Succeeded", "") Expect(applyFromTemplate("test/e2e/templates/watchrule-crd.tmpl", struct { Name string @@ -593,6 +524,7 @@ func (r argoBiDirectionalRun) startReverserPipeline() { // applies anything; otherwise the first live object could be folded into an // unattributed baseline reconcile instead of arriving as its own event. waitForStreamsRunning(r.gitTargetName, r.testNs) + return hookID } func (r argoBiDirectionalRun) applyArgoRepoSecret() { @@ -633,8 +565,7 @@ func (r argoBiDirectionalRun) configureArgoWebhookSecret() { // ensureArgoWebhook registers a Gitea webhook that notifies argocd-server on push, // so Argo refreshes immediately instead of waiting for its timed reconciliation. -// This mirrors the Flux receiver webhook wiring (ensureRepoWebhook), but Argo has -// a single /api/webhook endpoint rather than a per-repo Receiver CR. +// With automated sync off, a refresh only compares: the specs show it never syncs. // // The match is exact host+path: Argo compiles a regex from the payload's // repository HTMLURL (derived from Gitea's ROOT_URL) and tests it against the @@ -664,12 +595,13 @@ func (r argoBiDirectionalRun) ensureArgoWebhook() { Expect(err).NotTo(HaveOccurred(), "failed to create the Gitea -> Argo CD webhook") } -// syncToRevision drives a sync through the Application's `.operation` field — -// the kubectl-only equivalent of `argocd app sync --revision `. Passing the -// revision explicitly means the sync does not depend on a refresh having landed. +// syncToRevision is a human pressing Sync: it sets the Application's `.operation` +// to a sync of revision, initiated by "e2e", the kubectl-only equivalent of +// `argocd app sync --revision `. Only the refusal spec's setup uses it, before +// any GitTarget names the Application. func (r argoBiDirectionalRun) syncToRevision(revision string) { GinkgoHelper() - By(fmt.Sprintf("syncing Argo CD Application %q to revision %s", r.appName, revision[:8])) + By(fmt.Sprintf("syncing Argo CD Application %q to revision %s by hand", r.appName, revision[:8])) // Argo clears `.operation` when it finishes and leaves the result in // `.status.operationState`. Without pinning the previous finishedAt, a @@ -689,33 +621,85 @@ func (r argoBiDirectionalRun) syncToRevision(revision string) { Eventually(func(g Gomega) { app, err := r.appObject() g.Expect(err).NotTo(HaveOccurred()) - - finishedAt, _, err := unstructured.NestedString(app.Object, "status", "operationState", "finishedAt") - g.Expect(err).NotTo(HaveOccurred()) + finishedAt, _, _ := unstructured.NestedString(app.Object, "status", "operationState", "finishedAt") g.Expect(finishedAt).NotTo(Equal(previousFinishedAt), "Argo CD has not finished a NEW sync operation yet") - - phase, _, err := unstructured.NestedString(app.Object, "status", "operationState", "phase") - g.Expect(err).NotTo(HaveOccurred()) + phase, _, _ := unstructured.NestedString(app.Object, "status", "operationState", "phase") g.Expect(phase).To(Equal("Succeeded"), "Argo CD sync operation did not succeed") - - synced, _, err := unstructured.NestedString(app.Object, "status", "operationState", "syncResult", "revision") - g.Expect(err).NotTo(HaveOccurred()) + synced, _, _ := unstructured.NestedString(app.Object, "status", "operationState", "syncResult", "revision") g.Expect(synced).To(Equal(revision), "Argo CD synced a different revision") + }, argoSyncTimeout, biPollInterval).Should(Succeed()) +} - // A sync that applies nothing still reports Succeeded. Guard against the - // empty-path failure mode: if the reverser pruned the manifest before Argo - // read it, `resources` is empty and the spec would otherwise sail past. - resources, found, err := unstructured.NestedSlice( - app.Object, "status", "operationState", "syncResult", "resources") +// waitForReverserSync waits for a sync GitOps Reverser started, for reason, to +// succeed: initiated by gitops-reverser with automated: true, carrying the reason in +// its info, and pinned to revision (an empty revision accepts any commit), which is +// also the revision Argo CD synced. +func (r argoBiDirectionalRun) waitForReverserSync(revision, reason string) { + GinkgoHelper() + Eventually(func(g Gomega) { + app, err := r.appObject() g.Expect(err).NotTo(HaveOccurred()) - g.Expect(found).To(BeTrue(), "Argo CD sync produced no syncResult.resources") + state, found, _ := unstructured.NestedMap(app.Object, "status", "operationState") + g.Expect(found).To(BeTrue(), "Argo CD has run no operation yet") + + username, _, _ := unstructured.NestedString(state, "operation", "initiatedBy", "username") + automated, _, _ := unstructured.NestedBool(state, "operation", "initiatedBy", "automated") + g.Expect(username).To(Equal(argoReverserInitiator), "the last sync was not GitOps Reverser's") + g.Expect(automated).To(BeTrue(), "GitOps Reverser's syncs stand in for automated sync") + g.Expect(operationInfo(state, "gitops-reverser reason")).To(Equal(reason)) + + pinned, _, _ := unstructured.NestedString(state, "operation", "sync", "revision") + synced, _, _ := unstructured.NestedString(state, "syncResult", "revision") + g.Expect(pinned).NotTo(BeEmpty(), "a sync GitOps Reverser starts is always pinned") + if revision != "" { + g.Expect(pinned).To(Equal(revision), "the sync must be pinned to the commit GitOps Reverser checked") + } + g.Expect(synced).To(Equal(pinned), "Argo CD synced a revision other than the pinned one") + + phase, _, _ := unstructured.NestedString(state, "phase") + g.Expect(phase).To(Equal("Succeeded"), "the sync GitOps Reverser asked for did not succeed") + resources, _, _ := unstructured.NestedSlice(state, "syncResult", "resources") g.Expect(resources).NotTo(BeEmpty(), "Argo CD synced zero resources — is the source path empty?") - }, argoSyncTimeout, biPollInterval).Should(Succeed()) + }, argoRequestedSyncTimeout, biPollInterval).Should(Succeed()) +} + +// operationInfo is the value of the named entry in an operation state's info. +func operationInfo(state map[string]any, name string) string { + info, _, _ := unstructured.NestedSlice(state, "operation", "info") + for _, item := range info { + entry, ok := item.(map[string]any) + if ok && entry["name"] == name { + value, _ := entry["value"].(string) + return value + } + } + return "" +} + +// lastOperation is who started the Application's last operation, and the revision +// it was pinned to; both empty when it has run none. +func (r argoBiDirectionalRun) lastOperation(g Gomega) (string, string) { + app, err := r.appObject() + g.Expect(err).NotTo(HaveOccurred()) + username, _, _ := unstructured.NestedString(app.Object, + "status", "operationState", "operation", "initiatedBy", "username") + revision, _, _ := unstructured.NestedString(app.Object, + "status", "operationState", "operation", "sync", "revision") + return username, revision +} + +// gitTargetApplierStatus is the GitTarget's status.applier: the reason and the +// commit of the last request delivered. +func (r argoBiDirectionalRun) gitTargetApplierStatus(g Gomega) (string, string) { + out, err := kubectlRunInNamespace(r.testNs, "get", "gittarget", r.gitTargetName, + "-o", "jsonpath={.status.applier.reason}|{.status.applier.requestedCommit}") + g.Expect(err).NotTo(HaveOccurred()) + reason, commit, _ := strings.Cut(strings.TrimSpace(out), "|") + return reason, commit } // operationFinishedAt returns "" when the Application has never run an operation. func (r argoBiDirectionalRun) operationFinishedAt() string { - GinkgoHelper() output, err := kubectlRunInNamespace(r.argoNs, "get", "application.argoproj.io", r.appName, "-o", "jsonpath={.status.operationState.finishedAt}") if err != nil { @@ -742,13 +726,10 @@ func (r argoBiDirectionalRun) appObject() (*unstructured.Unstructured, error) { return &obj, nil } -func (r argoBiDirectionalRun) syncStatus() string { - GinkgoHelper() +func (r argoBiDirectionalRun) syncStatus(g Gomega) string { app, err := r.appObject() - Expect(err).NotTo(HaveOccurred()) - - status, _, err := unstructured.NestedString(app.Object, "status", "sync", "status") - Expect(err).NotTo(HaveOccurred()) + g.Expect(err).NotTo(HaveOccurred()) + status, _, _ := unstructured.NestedString(app.Object, "status", "sync", "status") return status } @@ -791,29 +772,6 @@ func (r argoBiDirectionalRun) orderContainer(name string) (string, error) { return value, nil } -func (r argoBiDirectionalRun) orderScoopFlavor(name string) (string, error) { - order, err := r.liveOrder(name) - if err != nil { - return "", err - } - scoops, found, err := unstructured.NestedSlice(order.Object, "spec", "scoops") - if err != nil { - return "", err - } - if !found || len(scoops) == 0 { - return "", fmt.Errorf("IceCreamOrder %q has no scoops", name) - } - first, ok := scoops[0].(map[string]interface{}) - if !ok { - return "", fmt.Errorf("IceCreamOrder %q has a malformed scoop", name) - } - flavor, ok := first["flavor"].(string) - if !ok { - return "", fmt.Errorf("IceCreamOrder %q scoop has no flavor", name) - } - return flavor, nil -} - func (r argoBiDirectionalRun) waitForOrderContainer(name, container string) { GinkgoHelper() Eventually(func(g Gomega) { @@ -823,19 +781,20 @@ func (r argoBiDirectionalRun) waitForOrderContainer(name, container string) { }, biEventuallyTimeout, biPollInterval).Should(Succeed()) } -func (r argoBiDirectionalRun) patchOrderContainer(name, container string) { +// waitForCommittedOrderContainer waits for the order's committed file to hold container. +func (r argoBiDirectionalRun) waitForCommittedOrderContainer(name, container string) { GinkgoHelper() - _, err := kubectlRunInNamespace(r.testNs, "patch", iceCreamCRDName(crdGroupArgoBiDirectional), name, - "--type=merge", "--patch", fmt.Sprintf(`{"spec":{"container":%q}}`, container)) - Expect(err).NotTo(HaveOccurred(), "failed to patch IceCreamOrder through the API") + Eventually(func(g Gomega) { + g.Expect(r.gitPull()).To(Succeed()) + g.Expect(r.readCommittedOrder(name)).To(ContainSubstring("container: " + container)) + }, biEventuallyTimeout, biPollInterval).Should(Succeed()) } -func (r argoBiDirectionalRun) patchOrderScoop(name, flavor string, quantity int) { +func (r argoBiDirectionalRun) patchOrderContainer(name, container string) { GinkgoHelper() _, err := kubectlRunInNamespace(r.testNs, "patch", iceCreamCRDName(crdGroupArgoBiDirectional), name, - "--type=merge", "--patch", - fmt.Sprintf(`{"spec":{"scoops":[{"flavor":%q,"quantity":%d}]}}`, flavor, quantity)) - Expect(err).NotTo(HaveOccurred(), "failed to patch IceCreamOrder scoops through the API") + "--type=merge", "--patch", fmt.Sprintf(`{"spec":{"container":%q}}`, container)) + Expect(err).NotTo(HaveOccurred(), "failed to patch IceCreamOrder through the API") } func (r argoBiDirectionalRun) liveOrderPath(name string) string { @@ -844,6 +803,7 @@ func (r argoBiDirectionalRun) liveOrderPath(name string) string { func (r argoBiDirectionalRun) writeLiveOrder(order iceCreamOrderFile) { GinkgoHelper() + Expect(r.gitPull()).To(Succeed(), "the Git-side change is made on the current tip") content, err := renderTemplate("test/e2e/templates/icecreamorder-instance.tmpl", struct { Name string Namespace string diff --git a/test/e2e/flux_applier_e2e_test.go b/test/e2e/flux_applier_e2e_test.go index 0adbd305..38021b60 100644 --- a/test/e2e/flux_applier_e2e_test.go +++ b/test/e2e/flux_applier_e2e_test.go @@ -63,12 +63,7 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order }) AfterAll(func() { - if receiverHookID != 0 { - gitea := giteaTestInstance() - ctx, cancel := gitea.Context() - _ = gitea.Client().DeleteRepoHook(ctx, gitea.Org, repo.RepoName, receiverHookID) - cancel() - } + deleteRepoHook(repo.RepoName, receiverHookID) cleanupWatchRule(ruleName, testNs) cleanupGitTarget(destName, testNs) _, _ = kubectlRunInNamespace(testNs, "delete", "gitprovider", providerName, "--ignore-not-found=true") @@ -173,29 +168,7 @@ var _ = Describe("Manager Flux Applier", Label("manager", "flux-applier"), Order // API edit surfaces the push, and the refresher's tick is minutes away. It("applies a Git-side push the Git host reports, with nothing else to surface it", func() { By("configuring the GitProvider's receiver, and registering it as a Gitea webhook") - token := randomToken() - _, err := kubectlRunInNamespace(testNs, "create", "secret", "generic", receiverSecretName, - "--from-literal=token="+token) - Expect(err).NotTo(HaveOccurred(), "failed to create the receiver Secret") - _, err = kubectlRunInNamespace(testNs, "patch", "gitprovider", providerName, "--type=merge", "-p", - fmt.Sprintf(`{"spec":{"receiver":{"type":"gitea","secretRef":{"name":%q}}}}`, receiverSecretName)) - Expect(err).NotTo(HaveOccurred(), "failed to set spec.receiver") - var webhookPath string - Eventually(func(g Gomega) { - out, err := kubectlRunInNamespace(testNs, "get", "gitprovider", providerName, - "-o", "jsonpath={.status.webhookPath}") - g.Expect(err).NotTo(HaveOccurred()) - webhookPath = strings.TrimSpace(out) - g.Expect(webhookPath).NotTo(BeEmpty()) - }, time.Minute, 2*time.Second).Should(Succeed()) - gitea := giteaTestInstance() - ctx, cancel := gitea.Context() - hook, err := gitea.Client().CreateGiteaWebhook(ctx, gitea.Org, repo.RepoName, - fmt.Sprintf("http://gitops-reverser-receiver.%s.svc.cluster.local%s", resolveE2ENamespace(), webhookPath), - token, []string{"push"}) - cancel() - Expect(err).NotTo(HaveOccurred(), "failed to register the receiver as a Gitea webhook") - receiverHookID = hook.ID + receiverHookID = registerOperatorReceiver(testNs, providerName, receiverSecretName, repo.RepoName) Eventually(func(g Gomega) { g.Expect(appliedColor(g, appliedNs)).To(Equal("orange")) @@ -323,6 +296,46 @@ func gitTargetEventMessages(g Gomega, testNs, gitTarget, reason string) string { return out } +// registerOperatorReceiver gives a GitProvider a gitea receiver whose token is a new Secret, and +// registers the path it serves as a push webhook on the Gitea repository, pointed at the operator's +// receiver Service. It returns the hook's id, for deleteRepoHook. +func registerOperatorReceiver(testNs, providerName, secretName, repoName string) int64 { + GinkgoHelper() + token := randomToken() + _, err := kubectlRunInNamespace(testNs, "create", "secret", "generic", secretName, "--from-literal=token="+token) + Expect(err).NotTo(HaveOccurred(), "failed to create the receiver Secret") + _, err = kubectlRunInNamespace(testNs, "patch", "gitprovider", providerName, "--type=merge", "-p", + fmt.Sprintf(`{"spec":{"receiver":{"type":"gitea","secretRef":{"name":%q}}}}`, secretName)) + Expect(err).NotTo(HaveOccurred(), "failed to set spec.receiver") + var webhookPath string + Eventually(func(g Gomega) { + out, err := kubectlRunInNamespace(testNs, "get", "gitprovider", providerName, + "-o", "jsonpath={.status.webhookPath}") + g.Expect(err).NotTo(HaveOccurred()) + webhookPath = strings.TrimSpace(out) + g.Expect(webhookPath).NotTo(BeEmpty()) + }, time.Minute, 2*time.Second).Should(Succeed()) + gitea := giteaTestInstance() + ctx, cancel := gitea.Context() + defer cancel() + hook, err := gitea.Client().CreateGiteaWebhook(ctx, gitea.Org, repoName, + fmt.Sprintf("http://gitops-reverser-receiver.%s.svc.cluster.local%s", resolveE2ENamespace(), webhookPath), + token, []string{"push"}) + Expect(err).NotTo(HaveOccurred(), "failed to register the receiver as a Gitea webhook") + return hook.ID +} + +// deleteRepoHook deletes a Gitea webhook registerOperatorReceiver made; zero is none. +func deleteRepoHook(repoName string, hookID int64) { + if hookID == 0 { + return + } + gitea := giteaTestInstance() + ctx, cancel := gitea.Context() + defer cancel() + _ = gitea.Client().DeleteRepoHook(ctx, gitea.Org, repoName, hookID) +} + // randomToken is a receiver secret for one spec. func randomToken() string { GinkgoHelper() diff --git a/test/e2e/templates/bi-directional/argocd-application.tmpl b/test/e2e/templates/bi-directional/argocd-application.tmpl index d0b1b532..58872241 100644 --- a/test/e2e/templates/bi-directional/argocd-application.tmpl +++ b/test/e2e/templates/bi-directional/argocd-application.tmpl @@ -9,33 +9,22 @@ spec: repoURL: {{ .RepoURL }} targetRevision: {{ .Branch }} path: {{ .Path }} +{{- if not .Kustomize }} directory: # gitops-reverser mirrors into ////.yaml, # so Argo must walk subdirectories to see anything at all. recurse: true +{{- end }} destination: server: https://kubernetes.default.svc namespace: {{ .DestinationNamespace }} -{{- if or .Automated .SyncOptions }} syncPolicy: -{{- if .Automated }} automated: - prune: {{ .Prune }} - selfHeal: {{ .SelfHeal }} -{{- end }} -{{- if .SyncOptions }} - syncOptions: -{{- range .SyncOptions }} - - {{ . }} -{{- end }} -{{- end }} -{{- end }} -{{- if .IgnoreJSONPointers }} - ignoreDifferences: - - group: {{ .IgnoreGroup }} - kind: {{ .IgnoreKind }} - jsonPointers: -{{- range .IgnoreJSONPointers }} - - {{ . }} -{{- end }} +{{- if .Automated }} + # Automated sync ON: no `enabled`, which Argo CD reads as on. + prune: true +{{- else }} + # The standard: automated sync off, prune kept for the syncs GitOps Reverser starts. + enabled: false + prune: true {{- end }} From b295a977337f1d19f5a39496642499a6399b30b7 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 07:25:08 +0000 Subject: [PATCH 34/36] docs: the architecture overview and index name the Argo CD applier Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/INDEX.md | 2 +- docs/architecture.md | 28 ++++++++++++++++------------ 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/docs/INDEX.md b/docs/INDEX.md index aff9878e..39ce43eb 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -151,7 +151,7 @@ so you never have to read a proposal to find out it already shipped. | [`watch-and-catalog-architecture.md`](design/watch-and-catalog-architecture.md) | the target three-layer watch model — **needs a human call before building** | | [`metrics-observability-plan.md`](design/metrics-observability-plan.md) | **built.** the canonical metrics document, rewritten 2026-09 and shipped in the same wave. Reads the product as two ingestion paths — object state through watch, authorship through audit — joining at the resolver, and gives each stage one counter with a bounded `outcome` so the flow and its loss paths are drawable. Records what three review rounds corrected, including two reversals worth not re-litigating: the fact-loss counters must NOT be merged (an eviction counts a fact, a trim gap an occurrence, a decode error a whole batch), and a cumulative counter cannot replace a current count, which is why `status.streams` and `status.retainedDocuments` stayed. The instrument inventory lives in [`interpreting-metrics.md`](interpreting-metrics.md), not here. Open: the Grafana JSON and alert rules | | [`reconcile-triggering.md`](design/reconcile-triggering.md) | which controllers still fail to wake up | -| [`apply-trigger.md`](design/apply-trigger.md) | **built for Flux; Argo CD and a webhook applier are the aim.** Why GitOps Reverser decides when Git is applied: an apply is safe only when nothing live is missing from the commit, and only the branch's writers can see that. Carries what building it taught (timers are triggers, Flux has no fetch-only mode, a decision must hold at the patch, a retry is a decision so only the worker makes one) and the three units that make the 1:n cases work: the branch owns quietness, the applier owns obligations and retries, the `GitTarget` (by uid) owns consent. Ends with the apply window that is still racy and three ways to shrink it | +| [`apply-trigger.md`](design/apply-trigger.md) | **built for Flux and Argo CD; a webhook applier is the aim.** Why GitOps Reverser decides when Git is applied: an apply is safe only when nothing live is missing from the commit, and only the branch's writers can see that. Carries what building it taught (timers are triggers, Flux has no fetch-only mode, a decision must hold at the patch, a retry is a decision so only the worker makes one) and the three units that make the 1:n cases work: the branch owns quietness, the applier owns obligations and retries, the `GitTarget` (by uid) owns consent. Ends with the apply window that is still racy and three ways to shrink it | | [`push-cooldown.md`](design/push-cooldown.md) | **partly built.** Whether the fixed `5s` push cooldown still earns its place now that the commit window exists. Ordinary timer closures already space one author writing one target, but identity changes, saves with shorter timers and buffer-limit closures commit sooner, and serialization does not batch queued commits, so the cooldown is what batches them. Option C, the failure backoff, is built in #412 and shares the push timer; removing the success wait would now remove only `lastPushAt` and its branch, so that decision waits on the event-loop measurements in §9 | | [`multi-source-audit-ingress-hardening.md`](design/multi-source-audit-ingress-hardening.md) | how independent sources authenticate to a named audit route, when annotation routing is trustworthy, and how multi-provider ingestion remains fair | | [`release-image-reuse-plan.md`](design/release-image-reuse-plan.md) | PRs 2–5 unstarted | diff --git a/docs/architecture.md b/docs/architecture.md index 1f4330dc..06bed6d4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -132,8 +132,8 @@ that repository and branch: The worker owns each request until it is delivered, and decides every retry the same way. A request asks and never reports; it runs off the publication path and cannot fail a publication. **Built:** -the Flux `Kustomization` applier. **The aim:** an Argo CD `Application` and a webhook for any other -tool. See [the apply trigger design](design/apply-trigger.md). +the Flux `Kustomization` applier and the Argo CD `Application` applier, with automated sync off. +**The aim:** a webhook for any other tool. See [the apply trigger design](design/apply-trigger.md). **Redis/Valkey is optional but advised.** The default configured-author mode runs without it: a plain `helm install` comes up healthy and watches initialize from current state on startup. When an endpoint @@ -272,7 +272,7 @@ graph LR subgraph SOURCE["Source cluster selected by ClusterProvider"] API["Kubernetes API\ndiscovery · Namespace policy · watches"] - APPLIER["Applier\nFlux Kustomization"] + APPLIER["Applier\nFlux Kustomization\nor Argo CD Application"] end HOST[("Git host")] @@ -298,8 +298,9 @@ source context: client, discovery catalog, followability registry, source-namesp reachability state. Its `GitTarget`s open their source watches through that context. The contexts are keyed by provider name and are released when no `GitTarget` references them. -The applier is not one of the six kinds. It belongs to Flux, lives in the source cluster it applies -to, and is reached through the same source context and identity the watches use. +The applier is not one of the six kinds. It belongs to Flux or Argo CD, lives in the source cluster +it applies to or in the cluster `spec.applier.clusterProviderRef` names, and is reached through that +`ClusterProvider`'s identity. | CRD | Scope | One line role | |---|---|---| @@ -399,8 +400,10 @@ Key fields: - `spec.commit.window`: when this target's commit windows close: `idleTimeout` (default `5s`) of silence, or `maxDuration` (default `1m`) after the window opened. - `spec.commit.message`: `liveTemplate` / `reconcileTemplate` Go templates. -- `spec.applier`: optional; `kustomizationRef` (the Flux `Kustomization` that applies this folder, - and that GitOps Reverser asks to apply) and `quietPeriod` (default `30s`). +- `spec.applier`: optional; exactly one of `kustomizationRef` (the Flux `Kustomization` that applies + this folder, and that GitOps Reverser asks to apply) and `applicationRef` (the Argo CD + `Application`, automated sync off, whose syncs GitOps Reverser starts), with `quietPeriod` + (default `30s`) for a `Kustomization` only. - `spec.onRefusal`: `Ignore` (default) or `Reapply`, which asks the applier to reapply the tip when an edit to an existing document is refused. `Reapply` requires `spec.applier`. @@ -426,8 +429,9 @@ Status has a kstatus-compatible summary layer plus domain conditions: source API the data plane can currently reach. - `GitPathAccepted` explains the target side: the selected Git path is safe for the operator to materialize. -- `ApplierConfigured`, present only with `spec.applier`, says the applier exists in the source - cluster, applies this repository and branch, and can be patched. It is a gate. +- `ApplierConfigured`, present only with `spec.applier`, says the applier exists in its cluster, + applies this repository and branch, can be written, and (an `Application`) has automated sync off. + It is a gate. - `status.streams` is a bounded count summary, not a per-type list. - `status.applier` is an observation: the last apply request's commit, time and reason. @@ -1740,9 +1744,9 @@ Current limitations: - **GitOps Reverser is the fast path to a deployment.** With `spec.applier` set, the applier's own triggers are off, so while the operator is down a Git-side push reaches the cluster only through a manual sync, `flux reconcile`, or Flux's long backstop interval. -- **Only Flux can be an applier yet.** An Argo CD `Application` keeps its own triggers until - `applicationRef` lands. The `Kustomization` may live in any cluster a `ClusterProvider` reaches - (`spec.applier.clusterProviderRef`), such as a management cluster. +- **Only Flux and Argo CD can be appliers yet,** and an Argo CD `Application` only with automated + sync off: one that syncs on its own is refused, not driven. The applier may live in any cluster a + `ClusterProvider` reaches (`spec.applier.clusterProviderRef`), such as a management cluster. - **An apply request cannot reserve time.** An edit made while the requested apply runs, to a field the applier sets, is reverted on the cluster. Its value stays in Git, because the applier's write over it is not mirrored, and the next apply restores it; until then the cluster runs the older From bedeb2eed1c32d03016e649539a0661d2a311069 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 07:29:02 +0000 Subject: [PATCH 35/36] test(e2e): ask the GitTarget to reconcile before reading status.applier status.applier and the applier check are published on the GitTarget's reconcile, and a delivered request does not trigger one, so a converged target can take minutes to show it. Co-Authored-By: Claude Opus 5.5 (1M context) --- test/e2e/argocd_bi_directional_e2e_test.go | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/test/e2e/argocd_bi_directional_e2e_test.go b/test/e2e/argocd_bi_directional_e2e_test.go index cb3868a1..5e144720 100644 --- a/test/e2e/argocd_bi_directional_e2e_test.go +++ b/test/e2e/argocd_bi_directional_e2e_test.go @@ -198,6 +198,9 @@ var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), By("GitOps Reverser asks Argo CD to sync exactly that commit, as a ForeignCommit") run.waitForReverserSync(head, "ForeignCommit") run.waitForOrderContainer(run.syncedOrderName, "Cone") + // status.applier is published on the GitTarget's reconcile, which a delivered + // request does not trigger; a converged target reconciles every few minutes. + run.requestGitTargetReconcile() Eventually(func(g Gomega) { reason, commit := run.gitTargetApplierStatus(g) g.Expect(reason).To(Equal("ForeignCommit")) @@ -281,9 +284,7 @@ var _ = Describe("Bi Directional (Argo CD)", Label("bi-directional", "argocd"), run.applyArgoApp(automated) // The applier is read on the GitTarget's reconcile, not watched; a converged // target reconciles every few minutes, so ask for one now. - _, err := kubectlRunInNamespace(testNs, "annotate", "gittarget", run.gitTargetName, "--overwrite", - "reconcile.configbutler.ai/requestedAt="+strconv.FormatInt(time.Now().UnixNano(), 10)) - Expect(err).NotTo(HaveOccurred(), "failed to request a GitTarget reconcile") + run.requestGitTargetReconcile() verifyResourceCondition("gittarget", run.gitTargetName, testNs, "ApplierConfigured", "False", "AutomatedSyncEnabled", "has automated sync on") verifyResourceCondition("gittarget", run.gitTargetName, testNs, "Stalled", "True", @@ -688,6 +689,16 @@ func (r argoBiDirectionalRun) lastOperation(g Gomega) (string, string) { return username, revision } +// requestGitTargetReconcile asks the GitTarget controller to reconcile now, the way +// its reconcile-request annotation does: the applier check and status.applier are +// published on a reconcile, and nothing about an applier triggers one. +func (r argoBiDirectionalRun) requestGitTargetReconcile() { + GinkgoHelper() + _, err := kubectlRunInNamespace(r.testNs, "annotate", "gittarget", r.gitTargetName, "--overwrite", + "reconcile.configbutler.ai/requestedAt="+strconv.FormatInt(time.Now().UnixNano(), 10)) + Expect(err).NotTo(HaveOccurred(), "failed to request a GitTarget reconcile") +} + // gitTargetApplierStatus is the GitTarget's status.applier: the reason and the // commit of the last request delivered. func (r argoBiDirectionalRun) gitTargetApplierStatus(g Gomega) (string, string) { From 6efbf4d0cbf6dd9a62cf9c6936de1ce1b4ed2d91 Mon Sep 17 00:00:00 2001 From: Simon Koudijs Date: Sat, 17 Oct 2026 15:33:52 +0000 Subject: [PATCH 36/36] docs(argocd): list the requirements, and what not to do with Argo CD The Argo CD guide gains a requirements table (Argo CD 3.1 for automated.enabled, a single Git source on the branch, a recursive or kustomize source, annotation tracking, update RBAC, the receiver) and a "What not to do" section, each item read from Argo CD's source: manual, CI and rollback syncs; sync windows, which keep a blocked sync Running and run it later, still pinned to a commit that may be older than the tip; watching the kinds hooks create, which BeforeHookCreation deletes and recreates on every sync; switching automated sync back on; ApplicationSet-generated Applications; selfHeal and ignoreDifferences; Replace=true; and retry refresh. It also corrects the troubleshooting row that called a window-blocked sync "skipped". Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/argocd.md | 104 ++++++++++++++++++++++++++++++++++- docs/design/apply-trigger.md | 3 +- 2 files changed, 103 insertions(+), 4 deletions(-) diff --git a/docs/argocd.md b/docs/argocd.md index 674c9aa8..2237be18 100644 --- a/docs/argocd.md +++ b/docs/argocd.md @@ -57,6 +57,23 @@ Reverser's own commits are already live (they were read from the cluster), and a refresh the `Application` reports them `Synced`. So `spec.applier.quietPeriod` is rejected with an `applicationRef`. +## Requirements + +Running bi-directional with Argo CD is a supported setup, with a narrow configuration. Each row is +checked by GitOps Reverser or is a condition of the guarantees on this page. + +| Requirement | Why | When it is not met | +| --- | --- | --- | +| Argo CD 3.1 or later | `spec.syncPolicy.automated.enabled` exists from 3.1.0. It is the only way to switch automated sync off and keep `prune` | Before 3.1 the `Application` schema has no `enabled`, so the API server drops it, the block reads as on, and the target reports `AutomatedSyncEnabled`. Remove the `automated` block instead, and accept that a file deleted in Git leaves its object running | +| Automated sync off | GitOps Reverser starts every sync, on a quiet branch | `AutomatedSyncEnabled`; the `Application` is never synced by GitOps Reverser | +| A single Git source: `spec.source`, on the `GitProvider`'s repository | A sync is pinned to one commit of one repository | `SourceMismatch` for `spec.sources`, `spec.sourceHydrator`, or another repository | +| `targetRevision` is the `GitTarget`'s branch (`main` or `refs/heads/main`) | The tip GitOps Reverser checks is the tip of that branch | `SourceMismatch` for `HEAD`, an empty `targetRevision`, a tag or a commit | +| The source renders the folder GitOps Reverser writes: `directory.recurse: true` for a plain directory, or a kustomization at the path | GitOps Reverser writes `////.yaml` | A non-recursive directory sees no files, and the sync applies nothing | +| Resource tracking by `annotation`, Argo CD's default | GitOps Reverser strips `argocd.argoproj.io/tracking-id` before writing; it cannot strip `app.kubernetes.io/instance`, a label people set on purpose | With `label` or `annotation+label`, Argo CD's tracking label reaches Git on every object | +| `get` and `update` on the `Application`, for the identity of the `ClusterProvider` that reaches it | A sync is an update of the `Application` | `Forbidden` | +| The Git host's push webhook goes to GitOps Reverser's receiver | A Git-side push reaches the cluster in seconds | It still arrives, by the periodic refresh (`git.refreshInterval`, `10m` by default) | +| Nothing but GitOps Reverser syncs the `Application` while GitOps Reverser runs | See [what not to do](#what-not-to-do) | Not checked: an unchecked sync can revert live edits | + ## A complete example One folder, `apps/checkout` on branch `main` of `github.com/acme/platform`, published by GitOps @@ -239,8 +256,8 @@ these differences: - `sync.revision` is **pinned to the commit GitOps Reverser checked**, so the sync applies exactly that commit, not whatever the branch holds by then. - `initiatedBy` is `{username: gitops-reverser, automated: true}`. Argo CD's UI and history show - the sync as GitOps Reverser's, and a [sync window](https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/) - that blocks automated syncs blocks it too. + the sync as GitOps Reverser's, and [sync windows](#sync-windows-that-block-automated-syncs) treat + it as an automated sync. - `info` carries the request's reason, and a token that tells this sync's result from any other. - A `retry` with `refresh: true` is copied without it. Argo CD's refresh-on-retry drops the pinned revision and syncs the newest one, a commit GitOps Reverser never checked. @@ -279,6 +296,86 @@ The request runs off the branch worker, so a slow Argo CD never holds a publicat fails is decided again from scratch, after that `Application`'s own backoff, exactly as for [Flux](flux.md#what-gitops-reverser-asks-and-when). +## What not to do + +Each of these applies Git to the cluster without the check that nothing live is missing from it, +or makes GitOps Reverser mirror something that is not an edit. None is refused by the API; most +are invisible until an edit is lost. + +### Syncing by hand, from CI, or by rollback + +A sync somebody else starts applies the commit they chose, over whatever is pending at that moment. +A Sync button pressed while an edit sits in a commit window reverts the edit on the cluster, and +GitOps Reverser publishes the revert as Argo CD's commit. The same holds for `argocd app sync` in a +CI pipeline that runs on push: it is the Argo CD form of a Flux `Receiver`, and the push webhook into +GitOps Reverser replaces it. A sync in progress also holds GitOps Reverser's next request, for up to +5 minutes. + +`argocd app rollback` needs automated sync off, so the standard makes it possible. A rollback applies +an older commit over everything since, API edits included, and GitOps Reverser then publishes the +result as a new commit. To undo a change, revert it in Git instead, and GitOps Reverser asks for the +sync. + +A manual sync is the right tool only [while GitOps Reverser is down](#while-gitops-reverser-is-down). + +### Sync windows that block automated syncs + +GitOps Reverser's syncs carry `automated: true`, so an `AppProject` sync window treats them as +automated syncs. An active `deny` window blocks them, and so does an `allow` window outside its +schedule; `manualSync: true` on the window does not let them through. + +A blocked sync is not refused. Argo CD keeps the operation `Running`, with the message `Sync +operation blocked by sync window`, and runs it when the window opens, still pinned to the commit it +was asked for. By then the branch may hold newer commits from API edits, and the late sync reverts +them on the cluster, outside any apply window, so GitOps Reverser publishes the revert. Meanwhile +every new request waits behind the blocked operation and fails after 5 minutes. Keep sync windows +off the `AppProject`s of `Application`s that a `GitTarget` names. + +### Watching the objects your hooks create + +[Hooks](https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/) run on every sync, so on +every `ForeignCommit` and `Refusal`, whether or not the push touched the folder. Argo CD's default +delete policy, `BeforeHookCreation`, deletes a hook's object and creates it again on each sync, and a +hook named with `generateName` is a new object every time. GitOps Reverser mirrors objects of the +kinds a `WatchRule` covers, hooks included, and with the default `spec.prune.mode` (`OnEvent`) it +mirrors their deletion too. A `WatchRule` covering a hook's kind (usually `Job`) in the namespace +it runs in therefore commits on every sync, and can remove the hook's own manifest from Git. +Keep hook kinds out of the folder's `WatchRule`s. + +### Switching automated sync back on + +There is no temporary automated sync. The moment `enabled` turns true, Argo CD syncs every new commit +on its own again, GitOps Reverser's included, and the next one can land over an unpublished edit. +GitOps Reverser stops syncing the `Application` and reports `AutomatedSyncEnabled`, but it cannot stop +Argo CD. Name another applier, or none, before switching it on. + +### Generating the `Application` from an `ApplicationSet` + +The `ApplicationSet` controller owns the spec of the `Application`s it generates, and by default +writes its template back over any change. Its template must carry `automated: {enabled: false}`; a hand edit to +the generated `Application` does not last. This setup is not covered by tests. + +### Expecting `selfHeal` or `ignoreDifferences` to protect an edit + +`selfHeal` does nothing with automated sync off; live drift is published to Git instead of +reverted. `ignoreDifferences` with `RespectIgnoreDifferences=true` still works on the syncs GitOps +Reverser starts, and hands the field to the API side in both directions: a later Git edit to it does +not reach an existing object. That is [split ownership](bi-directional.md#the-split-ownership-alternative), +not bi-directional editing. + +### Relying on `Replace=true` + +A `Replace=true` sync, by sync option or resource annotation, creates and replaces objects without a +field manager, so their writes carry `argocd-application-controller`. GitOps Reverser counts that as +Argo CD's in the apply window, but no test measures it yet. The default apply, client-side or +`ServerSideApply=true`, writes as `argocd-controller` and is what the end-to-end tests cover. + +### Expecting a retry to pick up a newer commit + +A `syncPolicy.retry` with `refresh: true` makes each retry sync the newest revision. GitOps Reverser +copies the retry without `refresh`, so a failed sync retries the commit it was asked for. A newer +commit gets its own request once the branch is quiet. + ## Reverting refused edits With `onRefusal: Reapply`, an edit GitOps Reverser cannot write (one that has no home in the folder, @@ -317,7 +414,8 @@ it. | `ApplierConfigured=False`, `Forbidden` | The identity lacks one of the [permissions](#permissions) | Grant it, or set `applierAccess.argocd: true` | | A Warning `ApplyRequestFailed` Event | A request could not be delivered; the message says why, such as a sync somebody else started that did not end within 5 minutes | GitOps Reverser keeps asking on a quiet branch, at most every five minutes | | A file deleted in Git leaves its object running | The `Application` has no `automated.prune: true` | Add `automated: {enabled: false, prune: true}` | -| A sync is skipped | A sync window denies automated syncs now | Wait for the window, or allow automated syncs in it | +| The last sync stays `Running`: `Sync operation blocked by sync window` | A sync window blocks automated syncs, GitOps Reverser's included | Remove the window from the `AppProject`; see [sync windows](#sync-windows-that-block-automated-syncs) | +| A commit appears on every sync, adding or removing a `Job` | A `WatchRule` covers the objects a hook creates | Keep hook kinds out of the `WatchRule`s; see [hooks](#watching-the-objects-your-hooks-create) | ## While GitOps Reverser is down diff --git a/docs/design/apply-trigger.md b/docs/design/apply-trigger.md index 462240d5..764db515 100644 --- a/docs/design/apply-trigger.md +++ b/docs/design/apply-trigger.md @@ -201,7 +201,8 @@ Every rule the client relies on is Argo CD's own, read from its source (3.5.0): | The controller clears `operation` when the operation completes and leaves the result in `status.operationState`; a failed operation that will be retried is `Running` again, with a `finishedAt` | An attempt's window closes when the operation state carrying its token has completed (`Succeeded`, `Failed`, `Error`); a pending retry is still running | | A retry with `retry.refresh: true` drops the operation's revision and syncs the newest one | That one field is dropped from the copied retry: a retry stays pinned | | Every apply writes as field manager `argocd-controller`, client-side and server-side alike; a `Replace=true` sync creates and replaces with no field manager, so the API server takes it from the user agent, `argocd-application-controller` | The apply window knows both as Argo CD's managers | -| Sync windows are checked for every operation, by `InitiatedBy.Automated` | A sync GitOps Reverser starts obeys the project's windows for automated syncs | +| Sync windows are checked for every operation, by `InitiatedBy.Automated`, and a blocked operation stays `Running` until the window opens (`syncWindowPreventsSync`, `CanSync`) | A sync GitOps Reverser starts obeys the project's windows for automated syncs, and a blocked one runs later, still pinned: by then it can be older than the tip. The guide forbids sync windows on a named `Application`; `ApplierConfigured` does not check them yet | +| Hooks default to `BeforeHookCreation`: each sync deletes and recreates the hook's object (`DeletePolicies`) | A `WatchRule` covering a hook's kind mirrors that delete and create on every sync; the guide keeps hook kinds out of them | Three choices were decided (2026-10-17):