service: schedule-based alerting (alerting windows)

[JRemitz]

What

Adds a third service notification urgency, scheduled: the service notifies only during configured weekly windows (day-of-week + time-of-day + IANA timezone). Outside them the alert is still created, deduped and visible in the UI — the escalation policy simply never runs.

Motivating case: our rotations are staffed 24x5. We want to keep paging through the week and go quiet at the weekend, without losing the alerts.

Behavior

An alert captured outside a window notifies when the window next opens — its escalation policy never started, so it runs as if the alert had just arrived. This is exactly how maintenance mode and low urgency already behave in this fork. Operationally it means a quiet weekend's backlog pages at once on Monday; that was a deliberate choice, and it's pinned by a smoke test.

Manual escalation is refused outside the window, so the Escalate button isn't a way around it.

Design

The window predicate is evaluated in Go, reusing schedule/rule.Rule.IsActive, and materialized into services.notification_suppressed by the escalation manager each tick. The three escalation joins gain and not s.notification_suppressed.

Evaluating it in SQL was considered and rejected:

• It would be a second implementation of subtle semantics. Postgres ::time discards the DST fall-back fold, so a window spanning the repeated hour would flap off and back on — Clock.FirstOfDay/LastOfDay deliberately pick the first occurrence for a start and the last for an end.
• One service with an unresolvable timezone would raise inside the to_escalate CTE and abort the entire escalation batch. In Go it fails closed for that service alone.
• It puts timezone math on the O(alerts) read path, three times per pass, with no useful index.

This also matches how the engine already works — schedulemanager computes rule activity in Go and materializes it.

The flag is tick-consistent, not instantaneously consistent (5s cycle). That's the same contract every other materialized-state module operates under.

Rollout

Behind the svc-alert-schedule experimental flag. Gating is in the resolver rather than @experimental directives, because the directive can't be applied to a single enum value, and putting it on the input fields would leave already-scheduled services uneditable once the flag is turned off.

Bumps the escalation processing version 5 → 6. Old and new engines will not both process during a rolling deploy — expect a brief escalation pause between the migration and the new binaries. Same handoff the two prior service migrations performed.

Rollback converts scheduled → high, deliberately: over-paging is recoverable, silently never paging again is not.

Testing

• Smoke (7) — notifies inside the window; records silently outside; backlog flushes when the window opens; manual escalation refused; flag gates the mutation; an unrelated edit doesn't re-suppress an open service; a scheduled service stays editable with the flag off.
• Unit (Go) — weekday indexing, overnight and 24h windows, America/Chicago spring-forward and fall-back, a half-hour-offset zone.
• Jest — ClockTime conversion and zone re-anchoring.
• Playwright — configure a window end to end, including a 09:00–17:00 round-trip through the API and a weekday-filter round-trip.

Full smoke suite green; Playwright green across three consecutive full runs.

Reviewer notes

• service/store.go and service/search.go scan services independently — both were updated.
• In-flight notification policy cycles are not stopped when a window closes, matching existing maintenance-mode behavior. Only escalation is gated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AjGEDUyqZq824xB6bzYX8E