fix(etcd): pin tenant etcd to v3.5.31 to restore consistent watches - #3122
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a compatibility issue where tenant Kubernetes control planes running on newer versions (1.31+) were failing to perform consistent watches due to an outdated etcd version (v3.5.12). By strategically overriding the etcd image in the EtcdCluster pod template, the change ensures that the required RequestWatchProgress storage feature is available, restoring stability for watch-based clients without requiring immediate changes to the underlying etcd-operator. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
📝 WalkthroughWalkthroughThe etcd cluster template pins the etcd container image to Changesetcd image pin and tracking
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related issues
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request overrides the default etcd image in the etcd-cluster template to quay.io/coreos/etcd:v3.5.31 to support consistent watches (which require etcd >= 3.5.13) and adds corresponding unit tests. However, the review feedback correctly points out that v3.5.31 is a non-existent image tag (likely a typo for v3.5.13 or v3.4.31) that will cause image pull failures. The reviewer suggests correcting the image tag to a valid version like v3.5.13 in both the template and the test suite.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| # in 1.35), so on etcd 3.5.12 every consistent watch fails inside the | ||
| # stream with "the required storage feature RequestWatchProgress is | ||
| # disabled" (HTTP 500). Pinning a >= 3.5.13 patch release restores it. | ||
| image: quay.io/coreos/etcd:v3.5.31 |
There was a problem hiding this comment.
The etcd image version v3.5.31 does not exist. The etcd 3.5 release branch has not reached .31 (the latest release is around v3.5.17). This is likely a transposition typo of v3.5.13 (the minimum version required for RequestWatchProgress) or a mix-up with v3.4.31. Using a non-existent image tag will cause ErrImagePull / ImagePullBackOff and completely break the tenant etcd cluster. Please use a valid existing version, such as v3.5.13 or a newer stable patch release like v3.5.17.
image: quay.io/coreos/etcd:v3.5.13| value: etcd | ||
| - equal: | ||
| path: spec.podTemplate.spec.containers[0].image | ||
| value: quay.io/coreos/etcd:v3.5.31 |
There was a problem hiding this comment.
Update the expected image value in the test to match the corrected etcd version (e.g., v3.5.13 or v3.5.17) to ensure the test suite passes.
value: quay.io/coreos/etcd:v3.5.13147fe85 to
78cf9e1
Compare
etcd-operator v0.4.5 hardcodes quay.io/coreos/etcd:v3.5.12 and the
v1alpha1 EtcdCluster CRD exposes no version field, so the tenant
control-plane etcd runs 3.5.12. kube-apiserver only enables the
RequestWatchProgress storage feature on etcd >= 3.5.13 / >= 3.4.31, and
Kubernetes >= 1.31 enables ConsistentListFromCache (locked-to-true in
1.35), which depends on it. On etcd 3.5.12 every consistent watch fails
inside the stream with HTTP 500 ("the required storage feature
RequestWatchProgress is disabled"), breaking watch-based clients of
aggregated APIs in modern tenants.
Override the etcd container image through spec.podTemplate.spec, which
the operator strategically merges over its generated pod spec (matched
by container name), pinning a >= 3.5.13 patch release. This restores
consistent watches without waiting on the etcd-operator v1alpha2
migration.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The etcd image in the extra/etcd EtcdCluster podTemplate is a hardcoded string that overrides etcd-operator v0.4.5's baked-in default. None of the enabled managers (gomod, dockerfile, github-actions) can see a literal image string in a Helm template, so the pin would silently age - and etcd 3.5.x releases are overwhelmingly security fixes. Add a custom.regex manager that matches the image line and resolves updates from the docker datasource, so future etcd patch releases surface as Renovate PRs like any other tracked dependency. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la>
78cf9e1 to
d1a2c1b
Compare
myasnikovdaniil
left a comment
There was a problem hiding this comment.
LGTM.
Two notes for anyone weighing the existing bot comments:
quay.io/coreos/etcd:v3.5.31does exist — it is a multi-arch manifest list on quay.io (amd64/arm64/ppc64le/s390x), published 2026-06-01, verified against the registry API. The "version does not exist / latest is ~v3.5.17" comments are out of date and should not block this (see inline).- The
podTemplateimage override actually takes effect with etcd-operator v0.4.5 — it strategic-mergespodTemplate.specover the operator-generated pod spec, matched by container name, so thisimagewins over the hardcoded default (see inline).
The pinned-floor test (>= 3.5.13, excludes v3.5.12) and the Renovate custom manager are the right shape. The rolling restart of every tenant etcd on merge is documented in the description (safe in-minor, quorum-preserving). No blockers; E2E green.
| # in 1.35), so on etcd 3.5.12 every consistent watch fails inside the | ||
| # stream with "the required storage feature RequestWatchProgress is | ||
| # disabled" (HTTP 500). Pinning a >= 3.5.13 patch release restores it. | ||
| image: quay.io/coreos/etcd:v3.5.31 |
There was a problem hiding this comment.
Confirmed valid — this does not need changing. quay.io/coreos/etcd:v3.5.31 exists on quay.io as a multi-arch manifest list (amd64/arm64/ppc64le/s390x), published 2026-06-01, verified against the registry API. The "tag does not exist / latest is ~v3.5.17" comments predate the release.
The override also takes effect: the v1alpha1 EtcdCluster CRD exposes no spec.image/spec.version, so podTemplate is the only lever. etcd-operator v0.4.5 builds the etcd container with its hardcoded default and then strategic-merges podTemplate.spec over it (matched by container name etcd), so this image wins and the operator does not re-assert its default — the same mechanism already used here for the metrics port, probes, and resources.
…rsion Signed-off-by: Andrey Kolkov <androndo@gmail.com>
…rsion Signed-off-by: Andrey Kolkov <androndo@gmail.com>
…rsion Signed-off-by: Andrey Kolkov <androndo@gmail.com>
…rsion Signed-off-by: Andrey Kolkov <androndo@gmail.com>
What this PR does
Tenant Kubernetes control planes get their etcd from the
extra/etcdchart, which creates anEtcdCluster(etcd.aenix.io/v1alpha1) reconciled by etcd-operator v0.4.5. That operator hardcodes the etcd image toquay.io/coreos/etcd:v3.5.12, and the v1alpha1 CRD exposes no version field, so every tenant etcd runs 3.5.12.kube-apiserver only enables the
RequestWatchProgressstorage feature on etcd >= 3.5.13 (or >= 3.4.31). Kubernetes >= 1.31 enablesConsistentListFromCache— locked-to-true in 1.35 — which depends on it. On etcd 3.5.12 every consistent watch therefore fails inside the stream with HTTP 500 (the required storage feature RequestWatchProgress is disabled), which breaks watch-based clients of aggregated APIs in modern (k8s 1.35) tenants.This pins the etcd container image to a >= 3.5.13 patch release (v3.5.31) by setting
imageon theetcdcontainer in the EtcdCluster'sspec.podTemplate.spec. etcd-operator v0.4.5 strategically mergespodTemplate.specover its generated pod spec (matched by container name), so the override takes effect with no operator change — the same mechanism the chart already uses for the metrics port, probes, and resources. It restores consistent watches without waiting on the etcd-operator v1alpha2 migration (#2859), which is the longer-term fix.A helm-unittest suite asserts the etcd container carries the pinned image and is no longer left on the operator default v3.5.12.
Addresses #3080.
Upgrade impact
On merge, etcd-operator reconciles every existing tenant
EtcdClusterand rolls its StatefulSet from 3.5.12 to 3.5.31. This is a safe in-minor rolling patch upgrade (no etcd storage-format change, quorum preserved by the operator's rolling strategy), so expect a brief, sequential etcd pod restart per tenant and no data migration.Release note
Summary by CodeRabbit
quay.io/coreos/etcd:v3.5.31to improve compatibility with Kubernetes watch-related storage behavior.