Skip to content

Add per-state duration metric to reconciliation - #2840

Open
rajathagasthya wants to merge 1 commit into
NVIDIA:mainfrom
rajathagasthya:state-duration-metric
Open

Add per-state duration metric to reconciliation#2840
rajathagasthya wants to merge 1 commit into
NVIDIA:mainfrom
rajathagasthya:state-duration-metric

Conversation

@rajathagasthya

@rajathagasthya rajathagasthya commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Description

A reconcile processes many component states, but existing metrics only
expose whole-reconcile totals, so a slow state cannot be identified
from metrics alone. Add a gpu_operator_state_seconds histogram
labeled by controller and state, covering the ClusterPolicy,
GPUCluster, and NVIDIADriver paths.

Checklist

  • No secrets, sensitive information, or unrelated changes
  • Lint checks passing (make lint)
  • Generated assets in-sync (make validate-generated-assets)
  • Go mod artifacts in-sync (make validate-modules)
  • Test cases are added for new code paths

Testing

go test ./controllers/ ./internal/state/; go build; gofmt.

@rajathagasthya rajathagasthya changed the title Add per-state duration metric to ClusterPolicy reconciliation Add per-state duration metric to reconciliation Aug 31, 2026
@rajathagasthya
rajathagasthya force-pushed the state-duration-metric branch 3 times, most recently from baff49d to 61a1c00 Compare August 31, 2026 20:30
A reconcile processes many component states, but existing metrics only
expose whole-reconcile totals, so a slow state cannot be identified
from metrics alone. Add a gpu_operator_state_seconds histogram labeled
by controller and state, covering the ClusterPolicy, GPUCluster, and
NVIDIADriver paths.

Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
@coderabbitai

coderabbitai Bot commented Aug 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Enterprise

Run ID: a1a60954-2eac-4c2c-956e-aeada06b6822

📥 Commits

Reviewing files that changed from the base of the PR and between f12acb8 and 0b0d474.

📒 Files selected for processing (3)
  • controllers/clusterpolicy_controller.go
  • internal/state/manager.go
  • internal/state/metrics.go

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The change adds the StateDurationSeconds Prometheus histogram. The metric uses controller and state labels and is registered with the controller-runtime metrics registry. stateManager stores the CRD kind and records synchronization duration. ClusterPolicy reconciliation records duration for each reconciliation state.

Merge Risk: ⚪ Minimal · up to 0b0d4

The change adds per-state reconciliation duration metrics without any identified merge-blocking risk; it is merge-ready after normal checks and review.


Comment @coderabbitai help to get the list of available commands.

@rajathagasthya rajathagasthya self-assigned this Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant