Skip to content

Add update-target-stage gate annotation - #1913

Open
ciecierski wants to merge 1 commit into
openstack-k8s-operators:mainfrom
ciecierski:staged-update
Open

Add update-target-stage gate annotation#1913
ciecierski wants to merge 1 commit into
openstack-k8s-operators:mainfrom
ciecierski:staged-update

Conversation

@ciecierski

@ciecierski ciecierski commented May 5, 2026

Copy link
Copy Markdown
Contributor

Introduce a core.openstack.org/update-target-stage annotation on
OpenStackVersion. When set, the update controller completes all
stages up to and including the named stage, marks the next stage as
blocked (FalseCondition/Gated), and pauses reconciliation.
Removing the annotation or advancing it to a later stage resumes the
update.

AI-assisted: Cursor (Claude Sonnet 4.6 by Anthropic)

@ciecierski
ciecierski requested a review from stuggi May 5, 2026 06:48
@openshift-ci
openshift-ci Bot requested review from rabi and rebtoor May 5, 2026 06:48
@openshift-ci

openshift-ci Bot commented May 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ciecierski
Once this PR has been reviewed and has the lgtm label, please assign stuggi for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@github-actions

github-actions Bot commented May 5, 2026

Copy link
Copy Markdown

OpenStackControlPlane CRD Size Report

Metric Value
CRD JSON size 337535 bytes (330KB)
Base branch size 337535 bytes
Change +0.00%
Status yellow — growing
Threshold reference
Color Range Meaning
🟢 green < 300KB Comfortable
🟡 yellow 300–400KB Growing
🟠 orange 400–750KB Concerning
🔴 red > 750KB Approaching 1.5MB etcd limit (cut in half to allow space for update)

Comment thread api/core/v1beta1/conditions.go Outdated
Comment thread test/functional/ctlplane/openstackversion_controller_test.go Outdated
Comment thread test/functional/ctlplane/openstackversion_controller_test.go Outdated

@stuggi stuggi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we might want to add webhook validation for the annotation values, like validating the annotation value in the ValidateUpdate webhook on OpenStackVersion . If someone sets target-stage=tyop, the webhook rejects the update immediately with a clear error, rather than silently ignoring it and not would run a full update.

The OpenStackVersion already has a webhook. Adding a check like:

  if stage, ok := r.Annotations[MinorUpdateTargetStageAnnotation]; ok {
      validStages := map[string]bool{...}
      if !validStages[stage] {
          return Forbidden("invalid target stage")
      }
  }

Comment thread api/core/v1beta1/conditions.go Outdated
Comment thread api/core/v1beta1/openstackversion_types.go Outdated
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/f690a18c992143429f1e0c967f1684a0

openstack-k8s-operators-content-provider FAILURE in 7m 30s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ openstack-operator-tempest-multinode SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ openstack-operator-docs-preview SUCCESS in 2m 58s
⚠️ openstack-operator-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider

@ciecierski
ciecierski force-pushed the staged-update branch 2 times, most recently from 49cecea to 1ed02a9 Compare May 12, 2026 16:19
@ciecierski ciecierski changed the title Add minor update target-stage gate annotation Add update-target-stage gate annotation May 12, 2026
@ciecierski

Copy link
Copy Markdown
Contributor Author

/test functional

1 similar comment
@ciecierski

Copy link
Copy Markdown
Contributor Author

/test functional

@ciecierski
ciecierski requested review from abays and stuggi May 18, 2026 13:39
@abays

abays commented May 18, 2026

Copy link
Copy Markdown
Contributor

What happens if...

  1. A previous annotation is present for, say, stage D, which the controller reached, paused and set the associated condition to True
  2. But then the user changes the annotation to target, say, stage C?

Stage D's condition would be reset to False, right? Is that okay? Even if it is, do we want to block that via the webhook anyhow?

Comment thread api/core/v1beta1/openstackversion_types.go
@ciecierski

Copy link
Copy Markdown
Contributor Author

What happens if...

  1. A previous annotation is present for, say, stage D, which the controller reached, paused and set the associated condition to True
  2. But then the user changes the annotation to target, say, stage C?

Stage D's condition would be reset to False, right? Is that okay? Even if it is, do we want to block that via the webhook anyhow?

We have two alternatives either let user set annotation for stage C and let user fix it with fixed oc patch. Or as you mentioned set webook to block user from making this kind of harmless mistake(setting annotations with stage C won't rollback updated containers in stage D) .

@ciecierski

Copy link
Copy Markdown
Contributor Author

/retest-required

@ciecierski
ciecierski force-pushed the staged-update branch 3 times, most recently from fae4854 to 6bda39d Compare May 28, 2026 13:44
@ciecierski

Copy link
Copy Markdown
Contributor Author

/retest

@ciecierski
ciecierski requested a review from stuggi July 21, 2026 07:43

@stuggi stuggi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general looks good, just some corner cases commented inline.

Comment thread internal/controller/core/openstackversion_controller.go Outdated
Comment thread internal/controller/core/openstackversion_controller.go Outdated
Comment thread api/core/v1beta1/openstackversion_webhook.go Outdated
Comment thread api/core/v1beta1/openstackversion_types.go Outdated
Comment thread internal/controller/core/openstackcontrolplane_controller.go Outdated
Comment thread docs/assemblies/proc_minor-update-staged-rollout.adoc
@ciecierski
ciecierski force-pushed the staged-update branch 2 times, most recently from 4cbffb2 to 2fab0eb Compare July 27, 2026 14:14
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/d1db42c981ff418ea795baf2d6c87502

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 28m 18s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 12s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000154753 failed in 0s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 51m 52s
✔️ openstack-operator-docs-preview SUCCESS in 3m 03s
openstack-operator-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000154755 failed in 0s

@stuggi

stuggi commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

/retest

@stuggi

stuggi commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/00d1394a467042e68ae9f98da2bba627

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 30m 23s
podified-multinode-edpm-deployment-crc RETRY_LIMIT in 3m 44s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 45m 05s
openstack-operator-tempest-multinode RETRY_LIMIT in 4m 04s
✔️ openstack-operator-docs-preview SUCCESS in 3m 50s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 14m 45s

@stuggi

stuggi commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

recheck

Log.Info("Returning for ControlPlane minor update reconcile")
return ctrlResult, nil
}
return ctrl.Result{}, nil

@stuggi stuggi Jul 29, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should return return ctrlResult, nil if ctrlResult != ctrl.Result{} ?

@stuggi stuggi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on comment inline and if I have not missed it, #1913 (comment) is still open?

Comment thread docs/assemblies/proc_minor-update-staged-rollout.adoc
Comment thread internal/controller/core/openstackversion_controller.go Outdated
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/0eb71472bd2d4311937f0d010f079eb1

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 31m 54s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 39m 50s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000156435 failed in 0s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 49m 22s
✔️ openstack-operator-docs-preview SUCCESS in 4m 19s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 11m 52s

@ciecierski
ciecierski force-pushed the staged-update branch 3 times, most recently from 9b1280e to 4e565a2 Compare August 18, 2026 17:02
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 078779e3-f5d6-418f-9db4-6a6c72b4372b

📥 Commits

Reviewing files that changed from the base of the PR and between a6d8851 and 4e565a2.

📒 Files selected for processing (9)
  • api/core/v1beta1/conditions.go
  • api/core/v1beta1/openstackversion_types.go
  • api/core/v1beta1/openstackversion_webhook.go
  • api/core/v1beta1/openstackversion_webhook_test.go
  • docs/assemblies/proc_minor-update-staged-rollout.adoc
  • docs/assemblies/updating-the-data-plane.adoc
  • internal/controller/core/openstackcontrolplane_controller.go
  • internal/controller/core/openstackversion_controller.go
  • test/functional/ctlplane/openstackversion_controller_test.go

Included review availability: Your plan includes up to 1 review per rolling hour; 0 remain after this review.


📝 Walkthrough

Summary by CodeRabbit

  • New Features

    • Added staged rollout controls for OpenStack minor updates.
    • Administrators can pause updates after defined infrastructure stages and resume or advance them safely.
    • Status now clearly indicates when the next stage is gated.
  • Bug Fixes

    • Prevented invalid, backward, or retroactive stage changes during updates.
    • Improved progression tracking so completed stages are not repeated.
  • Documentation

    • Added procedures for staged minor updates, pausing and resuming rollouts, validation, and troubleshooting.
    • Updated data plane update guidance with staged rollout instructions.

Walkthrough

This change adds annotation-controlled minor-update stages. It validates stage changes, gates controller reconciliation, records gated conditions, adds functional coverage, and documents staged control-plane and dataplane rollout procedures.

Changes

Staged minor-update rollout

Layer / File(s) Summary
Stage contract and validation
api/core/v1beta1/conditions.go, api/core/v1beta1/openstackversion_types.go, api/core/v1beta1/openstackversion_webhook.go, api/core/v1beta1/openstackversion_webhook_test.go
Defines ordered stages and gated conditions. Validates annotation values and prevents backward or retroactive stage changes during updates.
Controller stage gating
internal/controller/core/openstackversion_controller.go, internal/controller/core/openstackcontrolplane_controller.go
Pauses reconciliation at configured stages. Waits for completed conditions before continuing through control-plane components and dataplane processing.
Functional staged-rollout coverage
test/functional/ctlplane/openstackversion_controller_test.go
Tests pauses, annotation removal, stage advancement, and rejection of gates behind completed stages.
Staged rollout procedures
docs/assemblies/proc_minor-update-staged-rollout.adoc, docs/assemblies/updating-the-data-plane.adoc
Documents staged update commands, status checks, dataplane deployments, completion validation, and troubleshooting.

Estimated code review effort: 4 (Complex) | ~60 minutes

Merge Risk: 🔵 Low · up to 4e565

The staged-update annotation workflow is implemented, but the operator documentation could cause users to misunderstand which completed stages are valid targets. This bounded documentation risk is mergeable with explicit owner follow-up to correct the wording.

Sequence Diagram(s)

sequenceDiagram
  participant Operator
  participant OpenStackVersionWebhook
  participant OpenStackVersionController
  participant OpenStackControlPlaneController
  Operator->>OpenStackVersionWebhook: set update-target-stage annotation
  OpenStackVersionWebhook-->>Operator: accept or reject stage change
  OpenStackVersionController->>OpenStackControlPlaneController: allow configured rollout stage
  OpenStackControlPlaneController->>OpenStackVersionController: report stage completion
  OpenStackVersionController-->>Operator: mark next stage Gated or continue update
Loading

Suggested reviewers: rabi

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the new update-target-stage gate annotation, which is the main change.
Description check ✅ Passed The description accurately explains the annotation behavior, staged completion, gating, pausing, and resumption.
✨ Finishing Touches 💡 1
🛠️ Fix failing CI checks 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ciecierski

Copy link
Copy Markdown
Contributor Author

/test functional

@ciecierski
ciecierski requested a review from stuggi August 18, 2026 18:29
@ciecierski

Copy link
Copy Markdown
Contributor Author

on comment inline and if I have not missed it, #1913 (comment) is still open?

all done

Introduce a core.openstack.org/update-target-stage annotation on
OpenStackVersion. When set, the update controller completes all
stages up to and including the named stage, marks the next stage as
blocked (FalseCondition/Gated), and pauses reconciliation.

Removing the annotation or advancing it to a later stage resumes the
update.

Includes stage-name constants, the gated-message format string,
controller logic for all seven stages, functional tests for block/resume/
advance scenarios, webhooks and updated operator documentation.

AI-assisted: Cursor (Claude Sonnet 4.6 by Anthropic)
@openshift-ci

openshift-ci Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

@ciecierski: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/openstack-operator-build-deploy-kuttl-4-18 e5ffe26 link true /test openstack-operator-build-deploy-kuttl-4-18

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@ciecierski

ciecierski commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

@dprince CI failure for functional is:
• [FAILED] [42.053 seconds]
DataplaneDeployment Webhook when a running deployment is deleted without the confirm annotation [It] should reject the delete request
/go/src/github.com/openstack-k8s-operators/operator/test/functional/dataplane/openstackdataplanedeployment_webhook_test.go:75

When I run locally the same test:
make ginkgo-run GINKGO_TESTS=./test/functional/dataplane GINKGO_ARGS='--focus="DataplaneDeployment Webhook when a running deployment is deleted without the confirm annotation" it is pass.

Running Suite: DataPlane Controller Suite - /openstack-operator/test/functional/dataplane
======================================================================================================================
Random Seed: 1787079305

Will run 1 of 98 specs
Running in parallel across 3 processes
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•

Ran 1 of 98 Specs in 126.079 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 97 Skipped

coverage: 5.0% of statements
composite coverage: 5.0% of statements

Ginkgo ran 1 suite in 2m12.05405s
Test Suite Passed

@ciecierski

Copy link
Copy Markdown
Contributor Author

/test functional

@ciecierski

Copy link
Copy Markdown
Contributor Author

re kuttl - failure: WARN[2026-07-15T09:42:42Z] Found 1 conditions for ClusterClaim:
*[2026-07-15T08:42:43Z]NoClusters: No clusters in pool are ready to be claimed

@ciecierski

Copy link
Copy Markdown
Contributor Author

/test openstack-operator-build-deploy-kuttl-4-18

@ciecierski

Copy link
Copy Markdown
Contributor Author

/test openstack-operator-build-deploy-kuttl-4-20

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/cc1a2bf4a73e4b029c6ec1f4896a9465

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 15m 33s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 34m 36s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 53m 07s
openstack-operator-tempest-multinode NODE_FAILURE Node(set) request 099-0000173815 failed in 0s
✔️ openstack-operator-docs-preview SUCCESS in 3m 38s
openstack-operator-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000173816 failed in 0s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants