fix(seaweedfs): fix s3 probe scheme and cosi ServiceAccount naming broken by 4.31 - #2904
fix(seaweedfs): fix s3 probe scheme and cosi ServiceAccount naming broken by 4.31#2904myasnikovdaniil wants to merge 5 commits into
Conversation
The 4.31 chart re-vendor (#2834) gates s3 TLS behind s3.httpsPort, which defaults to 0, so `weed s3 -port=8333` now serves plain HTTP. cozystack's overrides still set the s3 readiness/liveness probe scheme to HTTPS and the ingress backend-protocol to HTTPS, both pointing at that plain-HTTP port. The HTTPS probe fails ("server gave HTTP response to HTTPS client"), the s3 Deployment never goes Ready, the seaweedfs-system HelmRelease stays InProgress, and the seaweedfs umbrella HR times out — failing the tenant install. Set the s3 probe scheme and the ingress backend-protocol to HTTP. TLS still terminates at the ingress edge (letsencrypt); in-cluster gRPC mTLS (global.seaweedfs.enableSecurity) is unaffected. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…s Deployment
The cosi objectstorage-provisioner Deployment requests serviceAccountName via the
seaweedfs.componentName helper (release-name derived, e.g.
seaweedfs-system-objectstorage-provisioner), but the ServiceAccount and the
ClusterRoleBinding subject were named from
{{ .Values.global.seaweedfs.serviceAccountName }}-objectstorage-provisioner. With
cozystack's global.seaweedfs.serviceAccountName ("tenant-foo-seaweedfs") differing
from the release name, the SA is created under a name the Deployment never
references, so the ReplicaSet cannot create pods:
serviceaccount "seaweedfs-system-objectstorage-provisioner" not found
and the provisioner never starts, leaving the seaweedfs-system HelmRelease stuck.
Name the SA and the RoleBinding subject with the same componentName helper the
Deployment uses, so all three always agree regardless of the configured
serviceAccountName. Carried as patches/cosi-sa-componentname.patch so it survives
`make update` re-vendoring; this should also be sent upstream to
seaweedfs/seaweedfs-helm (the bug is latent there, hidden only when
serviceAccountName == fullname).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add two helm-unittest suites guarding the install-blocking regressions: - cosi_serviceaccount_test.yaml: with global.seaweedfs.serviceAccountName set to a value other than the release name, asserts the cosi ServiceAccount, the ClusterRoleBinding subject, and the Deployment serviceAccountName all resolve to the same componentName-derived name. Mutation-verified: restoring the serviceAccountName-based SA name fails the suite. - s3_probe_scheme_test.yaml: asserts the s3 readiness/liveness probes use scheme HTTP, matching the plain-HTTP s3 port. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The 'Configure Tenant and wait for applications' test waited 10m for hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root, but seaweedfs installs as a serial chain seaweedfs-db (CNPG bootstrap, HR timeout 10m) -> seaweedfs-system (HR timeout 10m) -> seaweedfs wrapper. On a loaded CI runner each stage can run close to its own HR timeout, so a 10m wait is smaller than the chain it waits on and times out with the chain still converging (observed in runs 27293490370, 27293636512, 27331238229: master/filer/volume Running, s3 pods 2-4m old still settling, seaweedfs-db remediated mid-flight). 20m covers the serial worst case while a real regression still fails deterministically. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… chain The tenant-root/seaweedfs umbrella HelmRelease had timeout: 10m, but its helm action waits for the serial child chain seaweedfs-db (CNPG bootstrap, own HR timeout 10m) -> seaweedfs-system (own HR timeout 10m). The umbrella timeout was therefore structurally smaller than its children's legitimate worst case. On a loaded cluster the umbrella hit 10m mid-convergence, and because install/upgrade remediation is retries: -1, uninstall remediation then deleted both child HRs -- including the half-bootstrapped seaweedfs-db database -- and reinstalled from zero, looping forever (observed in e2e runs 27293490370, 27293636512, 27331238229: UninstallSucceeded on seaweedfs-db, seaweedfs-db-1 Terminating, fresh s3 pods every cycle). Raising the umbrella timeout to 20m covers the serial child chain. This is a failure backstop, not a fixed wait: Ready still flips the moment the chain converges. It also removes a destructive reset that would delete a tenant's seaweedfs database mid-bootstrap on any slow real-world cluster, not just in CI. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses two critical regressions introduced by the SeaweedFS 4.31 update that were preventing successful tenant installations. By correcting the S3 health-probe protocol and aligning the COSI ServiceAccount naming with the deployment expectations, the system is now able to reach a Ready state. Additionally, the umbrella HelmRelease timeout has been extended to provide more robust convergence during installation on resource-constrained environments. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
📝 WalkthroughWalkthroughThe PR adjusts SeaweedFS installation timeouts from 10m to 20m in both HelmRelease manifests and e2e tests to account for serial CNPG/HR bootstrap chain duration, standardizes COSI ServiceAccount naming using a Helm helper instead of static values, and corrects S3 health probe and ingress protocol configurations from HTTPS to HTTP. ChangesSeaweedFS Installation and Configuration Fixes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Trivy (0.69.3)Trivy execution failed: 2026-06-11T14:44:59Z FATAL Fatal error run error: fs scan error: scan error: scan failed: failed analysis: post analysis error: post analysis error: kubernetes scan error: scan config error: context deadline exceeded Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request increases the installation and wait timeouts for SeaweedFS to 20 minutes to prevent timeouts on loaded runners. It also updates the S3 health probe scheme and ingress backend protocol from HTTPS to HTTP, and adds regression tests for COSI ServiceAccount naming and S3 probe schemes. The reviewer pointed out that the patch file packages/system/seaweedfs/patches/cosi-sa-componentname.patch referenced in the Makefile is missing from the pull request, which will cause the build to fail.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| tar xzvf - --strip 3 -C charts seaweedfs-$${version}/k8s/charts/seaweedfs | ||
| patch --no-backup-if-mismatch -p4 < patches/resize-api-server-annotation.diff | ||
| patch --no-backup-if-mismatch -p4 < patches/disable-ca-key-rotation.patch | ||
| patch --no-backup-if-mismatch -p4 < patches/cosi-sa-componentname.patch |
There was a problem hiding this comment.
The patch file packages/system/seaweedfs/patches/cosi-sa-componentname.patch referenced here is missing from this pull request. Please make sure to add and commit this file so that make update can run successfully and the build does not fail.
References
- Structural changes to vendored charts should be applied via a patch file in packages//patches/ applied by the Makefile. (link)
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml (1)
16-24: ⚡ Quick winEffective regression guard for HTTP probe scheme.
The assertions correctly validate that both probes render with HTTP scheme, catching the 4.31 regression if defaults change.
Optional enhancement: Consider adding a mutation test case that overrides
s3.readinessProbe.httpGet.scheme: HTTPSvia theset:block and documents the expected runtime failure in a comment, demonstrating that the test would catch incorrect configuration. This provides defense-in-depth verification that the template correctly wires the scheme value.Example structure:
- it: documents that HTTPS scheme breaks s3 (mutation guard) set: s3.readinessProbe.httpGet.scheme: HTTPS asserts: - equal: path: spec.template.spec.containers[0].readinessProbe.httpGet.scheme value: HTTPS # Would render but fail at runtime with "HTTP response to HTTPS client"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml` around lines 16 - 24, Add a mutation-style test case to the existing s3_probe_scheme_test.yaml that sets s3.readinessProbe.httpGet.scheme to HTTPS (using the set: block) and asserts the rendered value at spec.template.spec.containers[0].readinessProbe.httpGet.scheme equals HTTPS; include an explanatory comment in the test describing that this mutation would render but fail at runtime with "HTTP response to HTTPS client" to document the regression-guard intent and ensure the template wiring is validated as a defense-in-depth check.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml`:
- Around line 16-24: Add a mutation-style test case to the existing
s3_probe_scheme_test.yaml that sets s3.readinessProbe.httpGet.scheme to HTTPS
(using the set: block) and asserts the rendered value at
spec.template.spec.containers[0].readinessProbe.httpGet.scheme equals HTTPS;
include an explanatory comment in the test describing that this mutation would
render but fail at runtime with "HTTP response to HTTPS client" to document the
regression-guard intent and ensure the template wiring is validated as a
defense-in-depth check.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 444f9929-0fcf-4106-b036-e38299a4b6f6
📒 Files selected for processing (9)
hack/e2e-install-cozystack.batspackages/apps/tenant/templates/seaweedfs.yamlpackages/system/seaweedfs/Makefilepackages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-cluster-role.yamlpackages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-service-account.yamlpackages/system/seaweedfs/patches/cosi-sa-componentname.patchpackages/system/seaweedfs/tests/cosi_serviceaccount_test.yamlpackages/system/seaweedfs/tests/s3_probe_scheme_test.yamlpackages/system/seaweedfs/values.yaml
|
Superseded by #2916, which fixes the same two root causes (S3 TLS on the main port + COSI provisioner ServiceAccount name) and is merged to main. The timeout backstop from this PR was not carried over; if it's still wanted on loaded runners it can come back as a focused change. Closing in favor of #2916. |
What this PR does
The SeaweedFS 4.31 re-vendor (#2834) introduced two deterministic regressions that fail a fresh tenant install at the
seaweedfsumbrella HelmRelease (surfaced now that the install retry was removed):s3 health-probe scheme. 4.31 gates s3 TLS behind
s3.httpsPort(default0), soweed s3 -port=8333serves plain HTTP — but the cozystack overrides still set the s3 readiness/liveness probescheme: HTTPSand the ingressbackend-protocol: HTTPS. The probe fails (http: server gave HTTP response to HTTPS client), s3 never goes Ready, andseaweedfs-systemstaysInProgress. Set the probe + ingress backend to HTTP; TLS still terminates at the ingress edge and gRPCenableSecurityis untouched. (Verified against a real 4.31 s3 pod:http://:8333/status→200,https://→ TLSwrong version number.)cosi ServiceAccount name mismatch. The objectstorage-provisioner Deployment requests its
serviceAccountNamevia theseaweedfs.componentNamehelper (release-derived), but the ServiceAccount and the ClusterRoleBinding subject were named fromglobal.seaweedfs.serviceAccountName, which differs from the release name. The SA is created under a name the Deployment never references →serviceaccount "…-objectstorage-provisioner" not found, and the provisioner never starts. Name the SA + RoleBinding subject with the samecomponentNamehelper, carried aspatches/cosi-sa-componentname.patchso it survivesmake update(this should also be sent upstream to seaweedfs/seaweedfs-helm — the bug is latent there, hidden only whenserviceAccountName == fullname).Also keeps the
seaweedfsumbrella HRtimeoutand the e2e convergence wait at 20m as belt-and-suspenders for the serialseaweedfs-db → seaweedfs-system → seaweedfschain on loaded runners.Regression coverage: two helm-unittest suites pin the s3 probe scheme and the cosi SA ↔ Deployment ↔ RoleBinding naming (mutation-verified). Validated at runtime on a dev cluster (k8s 1.35.4) in isolation; the existing tenant-root seaweedfs install was left untouched.
May need a
backportto whichever release shipped SeaweedFS 4.31.Release note
Summary by CodeRabbit
Bug Fixes
Tests