Skip to content

fix(core): fail deploy when controllers stay unhealthy - #173

Open
nickhuo wants to merge 1 commit into
nickhuo/fix-controller-readinessfrom
nickhuo/fail-unhealthy-deploy
Open

nickhuo wants to merge 1 commit into
nickhuo/fix-controller-readinessfrom
nickhuo/fail-unhealthy-deploy

Conversation

@nickhuo

@nickhuo nickhuo commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • fail deployment when any controller is still non-healthy at the readiness deadline
  • perform one final status read at the deadline so a controller that became healthy on the boundary is accepted
  • report every unresolved controller and its final status in one fatal diagnostic
  • make verbose deploy output stop and propagate fatal log lines instead of waiting indefinitely

Behavior

This PR is stacked on #170, which makes starting, healthy, and failed trustworthy. It deliberately preserves the existing startup window rather than aborting on the first observed failed: Redis can be reused between deploys, so an early read can see a previous run's status before the new controller publishes its own state.

After the timeout, the Global Controller now performs a final read. If any configured replica is still not healthy, it logs CRITICAL, raises, exits the in-container deploy process non-zero, and never reaches the Global controller started marker that causes the CLI to print Deploy is live.

Verification

  • uvx --from uv==0.12.15 uv run --package canyonos-core pytest -q packages/core/tests — 268 passed, 1 skipped, 29 subtests passed
  • uvx --from uv==0.12.15 uv run --package canyonos pytest -q packages/cli/tests — 156 passed
  • uvx --from uv==0.12.15 uv run ruff check ... — passed
  • uvx --from uv==0.12.15 uv run ruff format --check ... — passed
  • uvx --from uv==0.12.15 uv run ty check — passed

Depends on #170.

@coderabbitai

coderabbitai Bot commented Sep 22, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)
  • main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: a7abbeb4-7a9d-4f18-b779-03064d70e42e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant