Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions DO-THIS-LOCAL.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,11 +154,10 @@ CRD-primary path (see [milestones/milestone-14.md](milestones/milestone-14.md)):
# Newman against orchestrator creating StackRuns (skips isolation/phase2 if already green)
./scripts/run-cluster-ci.sh --skip-isolation --skip-phase2 --with-operator

# Or Helm (also sets STACKRUN_VIA_CRD when operator.enabled)
# Or Helm (Flask always creates StackRuns; operator.enabled still controls the manager)
cd helm/tekton-dag && ./package.sh
helm upgrade --install tekton-dag . -n tekton-pipelines \
--set operator.enabled=true \
--set orchestrationService.stackrunViaCrd=true
--set operator.enabled=true

# Expect Stack status.valid=true and StackRun to create a PipelineRun
kubectl get stacks,stackruns -n tekton-pipelines
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Each row links to the **in-browser player** on Pages (`#seg-…`) and to the **c
| [M13](milestones/milestone-13.md) | **Partial** | Production hardening foundations shipped: webhook HMAC, stack secrets/config + deploy wiring + injection-status APIs, PipelineRun timeouts / task retries / failure classifier / resource profiles, `stack-promote` + registries + approval gate. Open: intercept secret/config wiring, Helm `appConfig` / ESO, GUI panels, observability, cross-cluster deploy. Roadmap video: [segment 18](https://jmjava.github.io/tekton-dag/#seg-18). Local cluster checklist: [DO-THIS-LOCAL.md](DO-THIS-LOCAL.md). |
| [M14](milestones/milestone-14.md) | **Partial** | **Kubernetes operator (CRD-primary, default-on):** `Stack` + `StackRun` + `Team` (`tektondag.io/v1alpha1`). Helm `operator.enabled` defaults **true**; Flask/GUI/Triggers/`generate-run.sh` create StackRuns. Kind soak path: `scripts/install-operator-kind.sh`, `run-cluster-ci.sh` (operator on unless `--skip-operator`). Hygiene: [M15](milestones/milestone-15.md). Follow-ons: [M16](milestones/milestone-16.md). |
| [M15](milestones/milestone-15.md) | **Completed** | Control-plane hygiene: idempotent StackRun→PipelineRun, GHA `--skip-operator`, Triggers `prNumber`, Flask/GUI promote approval, soak `ready==total`, dead PipelineRun builders. Squash-merged [#19](https://github.com/jmjava/tekton-dag/pull/19). |
| [M16](milestones/milestone-16.md) | **In progress** | Team CR overlay for Flask/GUI, `spec.continueFrom` → `stack-pr-continue`, Kind Stack admission webhook installer. Remaining: retire PipelineRun escape hatches, demo narration/MP4 rebuild, S34. |
| [M16](milestones/milestone-16.md) | **In progress** | Team CR overlay for Flask/GUI, `spec.continueFrom` → `stack-pr-continue`, Kind Stack admission webhook installer, PipelineRun escape hatches retired. Remaining: demo narration/MP4 rebuild, S34. |

Older milestones (M2, M3) are in [milestones/completed/](milestones/completed/).

Expand Down
2 changes: 1 addition & 1 deletion docs/SCRIPTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ Shared helpers live in [`scripts/common.sh`](../scripts/common.sh) (sourced by m
|--------|---------|
| [`publish-build-images.sh`](../scripts/publish-build-images.sh) | Build/push **compile** images (polyglot builders). |
| [`publish-orchestrator-image.sh`](../scripts/publish-orchestrator-image.sh) | Build/push **orchestrator** image. |
| [`generate-run.sh`](../scripts/generate-run.sh) | Emit/apply a **StackRun** (operator). Pass `--pipeline-run` for a raw Tekton PipelineRun (`pr` / `merge` only). |
| [`generate-run.sh`](../scripts/generate-run.sh) | Emit/apply a **StackRun** (operator). `--pipeline-run` was removed in M16. |
| [`promote-pipelines.sh`](../scripts/promote-pipelines.sh) | Promote pipeline definitions across environments/namespaces. |
| [`create-and-push-sample-repos.sh`](../scripts/create-and-push-sample-repos.sh) | Sample app repos for demos/regression. |
| [`ensure-git-ssh-secret.sh`](../scripts/ensure-git-ssh-secret.sh) | Git SSH secret for cluster git operations. |
Expand Down
15 changes: 7 additions & 8 deletions docs/demos/docgen.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ narration_from_source:
operator default-on (Stack/StackRun/Team CRs).'
- Do not frame milestone 13 as entirely future work. Say foundations are shipped and name only remaining open items when
discussing production hardening.
- The Kubernetes operator under operator/ is CRD-primary (Stack + StackRun + Team, tektondag.io/v1alpha1). Helm operator.enabled defaults on. Flask STACKRUN_VIA_CRD is true when the operator is installed.
- The Kubernetes operator under operator/ is CRD-primary (Stack + StackRun + Team, tektondag.io/v1alpha1). Helm operator.enabled defaults on. Flask always creates StackRun CRs.
- Demo video for this bundle is Manim + TTS + ffmpeg compose only (current docgen). Do not narrate VHS, ttyd, or terminal-tape
capture as a pipeline stage.
- 'Timestamps for this bundle must use Whisper (timestamps.engine: whisper). Do not use the local silencedetect aligner
Expand Down Expand Up @@ -368,9 +368,8 @@ narration_from_source:
08:
hints:
- 'Flask orchestrator on 8080: healthz/readyz, stacks API, POST /api/run modes including promote.'
- Orchestrator still owns webhook HMAC (fail-closed when named Secret missing) and stack resolve. When STACKRUN_VIA_CRD
is true it creates StackRun CRs instead of PipelineRuns — operator reconciles (see segment 19). Default remains direct
PipelineRun creation.
- Orchestrator still owns webhook HMAC (fail-closed when named Secret missing) and stack resolve. Flask always
creates StackRun CRs — operator reconciles (see segment 19).
context:
paths:
- README.md
Expand Down Expand Up @@ -461,8 +460,8 @@ narration_from_source:
timeouts/retries/classifier/profiles, promote+registries).'
- 'Remaining M13 open items only: intercept secret wiring, Helm appConfig/ESO templates, GUI panels, Prometheus/cost
labels, cross-cluster deploy, Results/Neo4j backup polish.'
- 'Post-M14 follow-ons: GUI native StackRun views / promote approve via StackRun.spec.approvedBy, enable Stack
admission webhook with certs, retire --pipeline-run / STACKRUN_VIA_CRD=false escape hatches.'
- 'Post-M14 follow-ons: M15 hygiene and M16 Team overlay / continueFrom / webhook installer / escape-hatch
retirement are landed. Remaining: spoken demo rebuild; S34 intercept E2E is parked.'
context:
paths:
- milestones/milestone-13.md
Expand All @@ -475,8 +474,8 @@ narration_from_source:
- 'CRDs: Stack (validate/topo status) vs StackRun (mode pr|bootstrap|merge|promote -> PipelineRun).'
- 'Flow: GitHub/API -> Flask (HMAC+resolve) -> StackRun -> tekton-dag-operator -> PipelineRun (orphaned on delete for
Results history).'
- 'Enablement: Helm operator.enabled and Kind STACKRUN_VIA_CRD default on. Golden Python↔Go PipelineRun builders.'
- 'Status: CRD-primary is the default runtime; webhook certs and GUI StackRun views remain follow-ons.'
- 'Enablement: Helm operator.enabled default on. Flask always creates StackRuns. Golden Python↔Go PipelineRun builders.'
- 'Status: CRD-primary is the default runtime; M15/M16 control-plane follow-ons except spoken demo rebuild are landed.'
context:
paths:
- operator/README.md
Expand Down
2 changes: 1 addition & 1 deletion docs/demos/hints/project-context.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ docgen:
only remaining open items when discussing production hardening.
- >-
The Kubernetes operator under operator/ is CRD-primary (Stack + StackRun + Team,
tektondag.io/v1alpha1). Flask STACKRUN_VIA_CRD and Helm operator.enabled default on.
tektondag.io/v1alpha1). Flask always creates StackRun CRs. Helm operator.enabled defaults on.
- >-
Demo video for this bundle is Manim + TTS + ffmpeg compose only (current docgen). Do not
narrate VHS, ttyd, or terminal-tape capture as a pipeline stage.
Expand Down
5 changes: 2 additions & 3 deletions docs/demos/hints/segment-08-orchestrator.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,8 @@ docgen:
including promote.
- >-
Orchestrator still owns webhook HMAC (fail-closed when named Secret missing) and
stack resolve. When STACKRUN_VIA_CRD is true it creates StackRun CRs instead of
PipelineRuns — operator reconciles (see segment 19). Default remains direct
PipelineRun creation.
stack resolve. Flask always creates StackRun CRs; the operator reconciles
them to PipelineRuns (see segment 19).
context:
paths:
- README.md
Expand Down
3 changes: 1 addition & 2 deletions docs/demos/hints/segment-18-roadmap-forward.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,7 @@ docgen:
Results/Neo4j backup polish.
- >-
Post-M14: M15 control-plane hygiene (idempotent PipelineRun create).
Remaining M16: Team CR as Flask/GUI source, Stack admission webhook with
certs, retire --pipeline-run / STACKRUN_VIA_CRD=false escape hatches.
Remaining M16: demo spoken narration/MP4 rebuild. S34 intercept E2E is parked.
context:
paths:
- milestones/milestone-13.md
Expand Down
9 changes: 5 additions & 4 deletions docs/demos/hints/segment-19-kubernetes-operator.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,12 @@ docgen:
Flow: GitHub/API -> Flask (HMAC+resolve) -> StackRun -> tekton-dag-operator ->
PipelineRun (orphaned on delete for Results history).
- >-
Enablement: Helm operator.enabled and Kind STACKRUN_VIA_CRD default on. Golden
Python↔Go PipelineRun builders.
Enablement: Helm operator.enabled default on. Flask always creates StackRuns.
Golden Python↔Go PipelineRun builders.
- >-
Status: CRD-primary is the default runtime; M15 hygiene (idempotent
create) in progress. Webhook certs and Team CR as API source remain M16.
Status: CRD-primary is the default runtime; M15 hygiene and M16 Team overlay /
continueFrom / webhook installer / escape-hatch retirement are landed. Spoken
demo rebuild remains M16.
context:
paths:
- operator/README.md
Expand Down
3 changes: 2 additions & 1 deletion helm/tekton-dag/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,6 @@ Packaged content under `raw/` is **not** committed by default; `package.sh` copi
| `orchestrationService.replicas` | int | `1` | Replica count |
| `orchestrationService.port` | int | `8080` | Container and Service port |
| `orchestrationService.resources` | object | requests/limits | Pod resources |
| `orchestrationService.stackrunViaCrd` | bool | `true` | When `operator.enabled`, set `STACKRUN_VIA_CRD` (set `false` to keep direct PipelineRun creates) |
| `operator.enabled` | bool | `true` | Deploy M14 Go operator + use CRDs from chart `crds/` |
| `operator.image` | string | `localhost:5000/tekton-dag-operator:latest` | Operator manager image |
| `operator.imagePullPolicy` | string | `Always` | Pull policy (`Always` for Kind local registry) |
Expand All @@ -90,6 +89,8 @@ Packaged content under `raw/` is **not** committed by default; `package.sh` copi
| `triggers.githubTokenSecretName` | string | `"github-token"` | Secret name for PR comment token (pipelines) |
| `dashboard.url` | string | `""` | Optional Tekton Dashboard base URL for PR links |

The orchestrator always creates **StackRun** CRs (`STACKRUN_VIA_CRD` is hard-coded `"true"`). `orchestrationService.stackrunViaCrd` was removed in M16; `--pipeline-run` on `generate-run.sh` is unsupported.

`compileImages` / `compileImageVariants` are the **source of truth** for image URLs you pass into PipelineRuns as `compile-image-*` parameters (see `scripts/generate-run.sh`). Wiring those params from Helm into every trigger is cluster-specific; keep values aligned with the images you push.

## Multi-team setup
Expand Down
4 changes: 2 additions & 2 deletions helm/tekton-dag/templates/orchestration-deployment.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ spec:
- name: REGISTRIES_FILE
value: {{ .Values.orchestrationService.registriesFile | quote }}
- name: STACKRUN_VIA_CRD
# On when operator is enabled (unless stackrunViaCrd is explicitly false).
value: {{ if and .Values.operator.enabled (ne .Values.orchestrationService.stackrunViaCrd false) }}"true"{{ else }}"false"{{ end }}
# Retired M16: Flask always creates StackRuns. Kept true so mixed-image rollouts stay on the CRD path.
value: "true"
- name: STACKS_DIR
value: /stacks
- name: TEAMS_DIR
Expand Down
3 changes: 0 additions & 3 deletions helm/tekton-dag/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -67,9 +67,6 @@ orchestrationService:
# Verify GitHub webhooks when the named Secret exists (key: secret|value|webhook-secret)
webhookVerifySignature: true
registriesFile: "/stacks/registries.yaml"
# M14: when operator.enabled, defaults to creating StackRun CRs.
# Set explicitly to false to keep direct PipelineRun creation while operator is installed.
stackrunViaCrd: true
resources:
requests:
cpu: "100m"
Expand Down
6 changes: 3 additions & 3 deletions milestones/milestone-16.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
# Milestone 16 — Control-plane follow-ons

**Status:** In progress (M15 landed as #19).
**Status:** In progress (M15 landed as #19; Team overlay / continueFrom / webhook installer as #20; escape hatches retired on this follow-on).

**Goal:** Finish the remaining dual sources of truth after M15 hygiene. Still **no new CRD kinds**.

## Checklist

- [x] Flask `StackResolver` and GUI `TeamRegistry` overlay **Team CRs** when a loader is wired; Git `teams/*/team.yaml` remains the GitOps source that `apply-stack-crs.sh` / `package.sh` mirror
- [x] Kind webhook installer (`scripts/install-operator-webhook-kind.sh`): certs + Service + ValidatingWebhookConfiguration; `ENABLE_WEBHOOKS=true` only with those certs. Helm still does not apply Fail-closed admission by default.
- [ ] Retire `--pipeline-run` / Flask `STACKRUN_VIA_CRD=false` once every documented cluster path runs the operator
- [x] Retire `--pipeline-run` / Flask `STACKRUN_VIA_CRD=false` (Flask always creates StackRuns; Helm env stays `"true"` for mixed-image rollouts; `--skip-operator` no longer flips the flag)
- [x] `spec.continueFrom` on StackRun + operator `stack-pr-continue` builder; `rerun-pr-from.sh` creates a StackRun
- [ ] Demo **hints** already describe default-on; rebuild **spoken** `docs/demos/narration/*.md` + MP4s via `cd docs/demos && docgen rebuild-after-audio` (do not `compose` alone)
- [ ] S34 intercept E2E remains a separate stop — do not start unless explicitly requested
Expand All @@ -17,7 +17,7 @@

1. GUI/Flask team list matches Team CRs applied from Git YAML (overlay).
2. Webhook installer exists; Kind without the installer still does not apply Fail-closed admission.
3. Escape hatches removed or documented as unsupported.
3. Escape hatches removed: `--pipeline-run` and `STACKRUN_VIA_CRD=false` are unsupported.
4. Narration lint + `docgen validate --pre-push` green after the audio rebuild.

## Do not
Expand Down
7 changes: 3 additions & 4 deletions orchestrator/app.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
"""
tekton-dag orchestration service.

Receives GitHub webhooks, resolves stacks, creates Tekton PipelineRuns.
Receives GitHub webhooks, resolves stacks, creates StackRun CRs.
The operator reconciles those into Tekton PipelineRuns.
Runs as an in-cluster pod alongside Tekton.
"""

Expand Down Expand Up @@ -46,9 +47,7 @@ def create_app():
"REGISTRIES_FILE",
os.path.join(os.environ.get("STACKS_DIR", "/stacks"), "registries.yaml"),
),
# M14: when true, create StackRun CRs instead of PipelineRuns directly.
STACKRUN_VIA_CRD=os.environ.get("STACKRUN_VIA_CRD", "false").lower()
in ("1", "true", "yes"),
# Retired M16: Flask always creates StackRun CRs. STACKRUN_VIA_CRD env is ignored.
)

stacks_dir = os.environ.get("STACKS_DIR", "/stacks")
Expand Down
Loading
Loading