Skip to content

fix(platform): make etcd v1alpha2 adoption (migration 50) robust in-cluster - #3258

Closed
Andrey Kolkov (androndo) wants to merge 1 commit into
cozystack:mainfrom
androndo:fix/etcd-migration50-adoption-robustness
Closed

fix(platform): make etcd v1alpha2 adoption (migration 50) robust in-cluster#3258
Andrey Kolkov (androndo) wants to merge 1 commit into
cozystack:mainfrom
androndo:fix/etcd-migration50-adoption-robustness

Conversation

@androndo

Copy link
Copy Markdown
Contributor

What this PR does

Fixes two defects in platform migration 50 (etcd etcd.aenix.ioetcd-operator.cozystack.io/v1alpha2 adoption, #2859) that block every in-cluster 1.5 → 1.6 upgrade on a cluster with an existing etcd:

  1. Cert-SAN wait treated transient kubectl failures as "SAN absent." ensure_wildcard_sans checked/awaited the wildcard SAN with kubectl get … 2>/dev/null | grep; any transient GET failure (API discovery refresh, apiserver blip, throttling) yields an empty string indistinguishable from a genuine absence → false miss, and the 120s wait never recovers. Replaced the two checks with a _san_present helper that retries on an empty read (a real Certificate/Secret never has empty dnsNames/alt-names) and accepts the native wildcard from either the issued Secret's cert-manager.io/alt-names annotation or the Certificate spec.dnsNames. This is the deeper root cause behind etcd v1alpha2 adoption migration hard-fails 1.5→1.6 upgrade on clusters with an existing etcd #3243's "cert re-issue race (times out a few seconds too early)".

  2. etcd-migrate had no kubeconfig in-cluster (etcd v1alpha2 adoption (migration 50): etcd-migrate has no kubeconfig in-cluster → adoption aborts #3255). etcd-migrate only reads a kubeconfig file (-k/--kubeconfig, default /root/.kube/config) and, unlike kubectl, does not fall back to the mounted in-cluster ServiceAccount. Neither the hook Job nor the script provided one, so both the dry-run and --apply aborted with error building kubeconfig: stat /root/.kube/config: no such file. Now the script synthesizes an in-cluster kubeconfig from the mounted ServiceAccount and passes --kubeconfig to both etcd-migrate calls.

Scope is limited to packages/core/platform/images/migrations/migrations/50 (a shell script — no make generate artifacts affected).

Verification

Verified end-to-end on a real 1.5.2 → 1.6.0-rc.1 upgrade (3-node cluster): with these fixes the pre-upgrade hook passes the cert step reliably and etcd-migrate runs, so the adoption completes in-place (etcd pods never restarted, data intact) and the adopted EtcdCluster reaches readyMembers=3 / Available=True.

Note: a couple of adjacent adoption-completion blockers were found in the same run and reported separately — see #3255, #3256, and the follow-up comment on #3243 (etcd-operator memory floor / OOM, and the legacy etcd-headless Service being pruned mid-transition so per-pod DNS breaks). They live outside migrations/50 and are not part of this PR.

fix(platform): etcd v1alpha2 adoption migration is now robust in-cluster — the cert-SAN wait no longer mis-reads a transient kubectl failure as a missing SAN, and `etcd-migrate` is given an in-cluster kubeconfig so the adoption runs to completion.

…luster

Migration 50 (etcd.aenix.io -> etcd-operator.cozystack.io/v1alpha2 adoption)
had two defects that blocked every in-cluster 1.5 -> 1.6 upgrade on a cluster
with an existing etcd:

1. Cert-SAN wait treated transient kubectl failures as "SAN absent".
   ensure_wildcard_sans checked/awaited the wildcard SAN with
   `kubectl get ... 2>/dev/null | grep`, so any transient GET failure (API
   discovery refresh, apiserver blip, throttling) produced an empty string
   indistinguishable from a genuine absence -> false miss, and the 120s wait
   never recovered. Replace the two ad-hoc checks with a _san_present helper
   that retries on an empty read (a real Certificate/Secret never has empty
   dnsNames/alt-names) and accepts the native wildcard from EITHER the issued
   Secret's cert-manager.io/alt-names annotation OR the Certificate
   spec.dnsNames (the source of truth for what cert-manager will issue).

2. etcd-migrate had no kubeconfig in-cluster.
   etcd-migrate only reads a kubeconfig file (-k/--kubeconfig, default
   /root/.kube/config) and, unlike kubectl, does not fall back to the mounted
   in-cluster ServiceAccount. The hook Job set no KUBECONFIG and passed no
   --kubeconfig, so both the dry-run and --apply aborted with
   "error building kubeconfig: stat /root/.kube/config: no such file".
   Synthesize an in-cluster kubeconfig from the mounted ServiceAccount and pass
   --kubeconfig to both etcd-migrate invocations.

Verified end-to-end on a 1.5.2 -> 1.6.0-rc.1 upgrade: the adoption now
completes in-place (pods never restarted, data intact) and the cluster reaches
readyMembers=3 / Available=True.

Refs: cozystack#3243, cozystack#3255

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 80a1e5e0-79b8-4d24-ac55-f31e44216c32

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/M This PR changes 30-99 lines, ignoring generated files area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) kind/bug Categorizes issue or PR as related to a bug labels Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) kind/bug Categorizes issue or PR as related to a bug size/M This PR changes 30-99 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant