Skip to content

fix(seaweedfs): fix s3 probe scheme and cosi ServiceAccount naming broken by 4.31 - #2904

Closed
myasnikovdaniil wants to merge 5 commits into
mainfrom
fix/seaweedfs-s3-http-cosi-sa
Closed

fix(seaweedfs): fix s3 probe scheme and cosi ServiceAccount naming broken by 4.31#2904
myasnikovdaniil wants to merge 5 commits into
mainfrom
fix/seaweedfs-s3-http-cosi-sa

Conversation

@myasnikovdaniil

@myasnikovdaniil myasnikovdaniil commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

What this PR does

The SeaweedFS 4.31 re-vendor (#2834) introduced two deterministic regressions that fail a fresh tenant install at the seaweedfs umbrella HelmRelease (surfaced now that the install retry was removed):

  1. s3 health-probe scheme. 4.31 gates s3 TLS behind s3.httpsPort (default 0), so weed s3 -port=8333 serves plain HTTP — but the cozystack overrides still set the s3 readiness/liveness probe scheme: HTTPS and the ingress backend-protocol: HTTPS. The probe fails (http: server gave HTTP response to HTTPS client), s3 never goes Ready, and seaweedfs-system stays InProgress. Set the probe + ingress backend to HTTP; TLS still terminates at the ingress edge and gRPC enableSecurity is untouched. (Verified against a real 4.31 s3 pod: http://:8333/status200, https:// → TLS wrong version number.)

  2. cosi ServiceAccount name mismatch. The objectstorage-provisioner Deployment requests its serviceAccountName via the seaweedfs.componentName helper (release-derived), but the ServiceAccount and the ClusterRoleBinding subject were named from global.seaweedfs.serviceAccountName, which differs from the release name. The SA is created under a name the Deployment never references → serviceaccount "…-objectstorage-provisioner" not found, and the provisioner never starts. Name the SA + RoleBinding subject with the same componentName helper, carried as patches/cosi-sa-componentname.patch so it survives make update (this should also be sent upstream to seaweedfs/seaweedfs-helm — the bug is latent there, hidden only when serviceAccountName == fullname).

Also keeps the seaweedfs umbrella HR timeout and the e2e convergence wait at 20m as belt-and-suspenders for the serial seaweedfs-db → seaweedfs-system → seaweedfs chain on loaded runners.

Regression coverage: two helm-unittest suites pin the s3 probe scheme and the cosi SA ↔ Deployment ↔ RoleBinding naming (mutation-verified). Validated at runtime on a dev cluster (k8s 1.35.4) in isolation; the existing tenant-root seaweedfs install was left untouched.

May need a backport to whichever release shipped SeaweedFS 4.31.

Release note

fix(seaweedfs): fix the s3 health-probe scheme and cosi ServiceAccount naming that broke tenant installs after the SeaweedFS 4.31 update

Summary by CodeRabbit

  • Bug Fixes

    • Increased installation timeouts from 10 to 20 minutes for tenant configuration and SeaweedFS to prevent mid-convergence timeout failures and remediation loops
    • Fixed S3 connectivity by updating health check probes and ingress controller to use HTTP instead of HTTPS
  • Tests

    • Added regression tests for ServiceAccount naming consistency across components
    • Added validation tests for S3 probe HTTP scheme configuration

The 4.31 chart re-vendor (#2834) gates s3 TLS behind s3.httpsPort, which
defaults to 0, so `weed s3 -port=8333` now serves plain HTTP. cozystack's
overrides still set the s3 readiness/liveness probe scheme to HTTPS and the
ingress backend-protocol to HTTPS, both pointing at that plain-HTTP port. The
HTTPS probe fails ("server gave HTTP response to HTTPS client"), the s3
Deployment never goes Ready, the seaweedfs-system HelmRelease stays InProgress,
and the seaweedfs umbrella HR times out — failing the tenant install.

Set the s3 probe scheme and the ingress backend-protocol to HTTP. TLS still
terminates at the ingress edge (letsencrypt); in-cluster gRPC mTLS
(global.seaweedfs.enableSecurity) is unaffected.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…s Deployment

The cosi objectstorage-provisioner Deployment requests serviceAccountName via the
seaweedfs.componentName helper (release-name derived, e.g.
seaweedfs-system-objectstorage-provisioner), but the ServiceAccount and the
ClusterRoleBinding subject were named from
{{ .Values.global.seaweedfs.serviceAccountName }}-objectstorage-provisioner. With
cozystack's global.seaweedfs.serviceAccountName ("tenant-foo-seaweedfs") differing
from the release name, the SA is created under a name the Deployment never
references, so the ReplicaSet cannot create pods:

  serviceaccount "seaweedfs-system-objectstorage-provisioner" not found

and the provisioner never starts, leaving the seaweedfs-system HelmRelease stuck.

Name the SA and the RoleBinding subject with the same componentName helper the
Deployment uses, so all three always agree regardless of the configured
serviceAccountName. Carried as patches/cosi-sa-componentname.patch so it survives
`make update` re-vendoring; this should also be sent upstream to
seaweedfs/seaweedfs-helm (the bug is latent there, hidden only when
serviceAccountName == fullname).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add two helm-unittest suites guarding the install-blocking regressions:

- cosi_serviceaccount_test.yaml: with global.seaweedfs.serviceAccountName set to
  a value other than the release name, asserts the cosi ServiceAccount, the
  ClusterRoleBinding subject, and the Deployment serviceAccountName all resolve
  to the same componentName-derived name. Mutation-verified: restoring the
  serviceAccountName-based SA name fails the suite.

- s3_probe_scheme_test.yaml: asserts the s3 readiness/liveness probes use scheme
  HTTP, matching the plain-HTTP s3 port.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The 'Configure Tenant and wait for applications' test waited 10m for
hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root, but
seaweedfs installs as a serial chain seaweedfs-db (CNPG bootstrap,
HR timeout 10m) -> seaweedfs-system (HR timeout 10m) -> seaweedfs
wrapper. On a loaded CI runner each stage can run close to its own
HR timeout, so a 10m wait is smaller than the chain it waits on and
times out with the chain still converging (observed in runs
27293490370, 27293636512, 27331238229: master/filer/volume Running,
s3 pods 2-4m old still settling, seaweedfs-db remediated mid-flight).
20m covers the serial worst case while a real regression still fails
deterministically.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… chain

The tenant-root/seaweedfs umbrella HelmRelease had timeout: 10m, but its
helm action waits for the serial child chain seaweedfs-db (CNPG bootstrap,
own HR timeout 10m) -> seaweedfs-system (own HR timeout 10m). The umbrella
timeout was therefore structurally smaller than its children's legitimate
worst case. On a loaded cluster the umbrella hit 10m mid-convergence, and
because install/upgrade remediation is retries: -1, uninstall remediation
then deleted both child HRs -- including the half-bootstrapped seaweedfs-db
database -- and reinstalled from zero, looping forever (observed in e2e
runs 27293490370, 27293636512, 27331238229: UninstallSucceeded on
seaweedfs-db, seaweedfs-db-1 Terminating, fresh s3 pods every cycle).

Raising the umbrella timeout to 20m covers the serial child chain. This is
a failure backstop, not a fixed wait: Ready still flips the moment the
chain converges. It also removes a destructive reset that would delete a
tenant's seaweedfs database mid-bootstrap on any slow real-world cluster,
not just in CI.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
@github-actions github-actions Bot added area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) kind/bug Categorizes issue or PR as related to a bug size/L This PR changes 100-499 lines, ignoring generated files labels Jun 11, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses two critical regressions introduced by the SeaweedFS 4.31 update that were preventing successful tenant installations. By correcting the S3 health-probe protocol and aligning the COSI ServiceAccount naming with the deployment expectations, the system is now able to reach a Ready state. Additionally, the umbrella HelmRelease timeout has been extended to provide more robust convergence during installation on resource-constrained environments.

Highlights

  • S3 Health Probe Fix: Updated the S3 readiness and liveness probe schemes from HTTPS to HTTP to match the SeaweedFS 4.31 default behavior where S3 serves plain HTTP.
  • COSI ServiceAccount Naming: Fixed a ServiceAccount naming mismatch in the COSI provisioner by using the componentName helper, ensuring the Deployment correctly references the created ServiceAccount.
  • Deployment Stability: Increased the HelmRelease timeout for the SeaweedFS umbrella from 10m to 20m to accommodate the serial installation chain on loaded runners.
  • Regression Testing: Added two new helm-unittest suites to pin the S3 probe scheme and the COSI ServiceAccount naming, preventing future regressions.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files
  • Ignored by pattern: **/*.patch (1)
    • packages/system/seaweedfs/patches/cosi-sa-componentname.patch
  • Ignored by pattern: **/charts/** (2)
    • packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-cluster-role.yaml
    • packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-service-account.yaml
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@dosubot dosubot Bot added area/testing Issues or PRs related to testing (e2e, bats, unit tests) backport Should change be backported on previous release kind/regression Categorizes issue or PR as related to a regression from a prior release labels Jun 11, 2026
@coderabbitai

coderabbitai Bot commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The PR adjusts SeaweedFS installation timeouts from 10m to 20m in both HelmRelease manifests and e2e tests to account for serial CNPG/HR bootstrap chain duration, standardizes COSI ServiceAccount naming using a Helm helper instead of static values, and corrects S3 health probe and ingress protocol configurations from HTTPS to HTTP.

Changes

SeaweedFS Installation and Configuration Fixes

Layer / File(s) Summary
SeaweedFS Serial Chain Timeout Adjustment
packages/apps/tenant/templates/seaweedfs.yaml, hack/e2e-install-cozystack.bats
HelmRelease spec and e2e test kubectl wait timeout increased from 10m to 20m with updated comments explaining the serial umbrella/child chain timing requirements and risk of mid-convergence timeout-triggered remediation loops.
COSI ServiceAccount Naming via Helm Helper
packages/system/seaweedfs/Makefile, packages/system/seaweedfs/patches/cosi-sa-componentname.patch, packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-cluster-role.yaml, packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-service-account.yaml, packages/system/seaweedfs/tests/cosi_serviceaccount_test.yaml
Makefile applies new patch that migrates objectstorage-provisioner ServiceAccount naming from .Values.global.seaweedfs.serviceAccountName-based to seaweedfs.componentName Helm helper across ClusterRole subject and ServiceAccount metadata templates, with regression test ensuring ServiceAccount, ClusterRoleBinding subject, and Deployment all match the computed name while global value is set to a mismatching sentinel.
S3 Health Probe and Ingress Protocol Correction
packages/system/seaweedfs/values.yaml, packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml
S3 readiness and liveness probe httpGet.scheme and ingress backend-protocol annotation updated from HTTPS to HTTP to match the plain HTTP S3 backend service, with test validating probe scheme usage within deployment template.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

  • cozystack/cozystack#2724: Both PRs modify hack/e2e-install-cozystack.bats's "Configure Tenant and wait for applications" HelmRelease readiness gating, including hr/seaweedfs and hr/tenant-root kubectl wait timeout values.
  • cozystack/cozystack#2601: Both PRs address SeaweedFS HelmRelease installation and wait sequencing for CNPG/DB bootstrap; the main PR's timeout adjustment directly accommodates the changed SeaweedFS rollout behavior from that prior refactoring.

Suggested labels

kind/bug, size/M, area/storage, area/kubernetes, area/testing

Suggested reviewers

  • lexfrei
  • lllamnyp
  • androndo
  • IvanHunters
  • kvaps

Poem

🐰 Timeouts stretched and schemes corrected,
COSI names now well-directed,
S3 speaks plain HTTP with glee,
SeaweedFS chains now flow free!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately and concisely summarizes the two main bug fixes: s3 probe scheme correction and COSI ServiceAccount naming, both related to SeaweedFS 4.31 regressions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/seaweedfs-s3-http-cosi-sa

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Trivy (0.69.3)

Trivy execution failed: 2026-06-11T14:44:59Z FATAL Fatal error run error: fs scan error: scan error: scan failed: failed analysis: post analysis error: post analysis error: kubernetes scan error: scan config error: context deadline exceeded


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request increases the installation and wait timeouts for SeaweedFS to 20 minutes to prevent timeouts on loaded runners. It also updates the S3 health probe scheme and ingress backend protocol from HTTPS to HTTP, and adds regression tests for COSI ServiceAccount naming and S3 probe schemes. The reviewer pointed out that the patch file packages/system/seaweedfs/patches/cosi-sa-componentname.patch referenced in the Makefile is missing from the pull request, which will cause the build to fail.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

tar xzvf - --strip 3 -C charts seaweedfs-$${version}/k8s/charts/seaweedfs
patch --no-backup-if-mismatch -p4 < patches/resize-api-server-annotation.diff
patch --no-backup-if-mismatch -p4 < patches/disable-ca-key-rotation.patch
patch --no-backup-if-mismatch -p4 < patches/cosi-sa-componentname.patch

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The patch file packages/system/seaweedfs/patches/cosi-sa-componentname.patch referenced here is missing from this pull request. Please make sure to add and commit this file so that make update can run successfully and the build does not fail.

References
  1. Structural changes to vendored charts should be applied via a patch file in packages//patches/ applied by the Makefile. (link)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml (1)

16-24: ⚡ Quick win

Effective regression guard for HTTP probe scheme.

The assertions correctly validate that both probes render with HTTP scheme, catching the 4.31 regression if defaults change.

Optional enhancement: Consider adding a mutation test case that overrides s3.readinessProbe.httpGet.scheme: HTTPS via the set: block and documents the expected runtime failure in a comment, demonstrating that the test would catch incorrect configuration. This provides defense-in-depth verification that the template correctly wires the scheme value.

Example structure:

  - it: documents that HTTPS scheme breaks s3 (mutation guard)
    set:
      s3.readinessProbe.httpGet.scheme: HTTPS
    asserts:
      - equal:
          path: spec.template.spec.containers[0].readinessProbe.httpGet.scheme
          value: HTTPS  # Would render but fail at runtime with "HTTP response to HTTPS client"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml` around lines 16 -
24, Add a mutation-style test case to the existing s3_probe_scheme_test.yaml
that sets s3.readinessProbe.httpGet.scheme to HTTPS (using the set: block) and
asserts the rendered value at
spec.template.spec.containers[0].readinessProbe.httpGet.scheme equals HTTPS;
include an explanatory comment in the test describing that this mutation would
render but fail at runtime with "HTTP response to HTTPS client" to document the
regression-guard intent and ensure the template wiring is validated as a
defense-in-depth check.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml`:
- Around line 16-24: Add a mutation-style test case to the existing
s3_probe_scheme_test.yaml that sets s3.readinessProbe.httpGet.scheme to HTTPS
(using the set: block) and asserts the rendered value at
spec.template.spec.containers[0].readinessProbe.httpGet.scheme equals HTTPS;
include an explanatory comment in the test describing that this mutation would
render but fail at runtime with "HTTP response to HTTPS client" to document the
regression-guard intent and ensure the template wiring is validated as a
defense-in-depth check.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 444f9929-0fcf-4106-b036-e38299a4b6f6

📥 Commits

Reviewing files that changed from the base of the PR and between 8348f96 and 1b5db2a.

📒 Files selected for processing (9)
  • hack/e2e-install-cozystack.bats
  • packages/apps/tenant/templates/seaweedfs.yaml
  • packages/system/seaweedfs/Makefile
  • packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-cluster-role.yaml
  • packages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-service-account.yaml
  • packages/system/seaweedfs/patches/cosi-sa-componentname.patch
  • packages/system/seaweedfs/tests/cosi_serviceaccount_test.yaml
  • packages/system/seaweedfs/tests/s3_probe_scheme_test.yaml
  • packages/system/seaweedfs/values.yaml

@myasnikovdaniil

Copy link
Copy Markdown
Contributor Author

Superseded by #2916, which fixes the same two root causes (S3 TLS on the main port + COSI provisioner ServiceAccount name) and is merged to main. The timeout backstop from this PR was not carried over; if it's still wanted on loaded runners it can come back as a focused change. Closing in favor of #2916.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/storage Issues or PRs related to storage (linstor, seaweedfs, bucket, velero, harbor) area/testing Issues or PRs related to testing (e2e, bats, unit tests) backport Should change be backported on previous release kind/bug Categorizes issue or PR as related to a bug kind/regression Categorizes issue or PR as related to a regression from a prior release size/L This PR changes 100-499 lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant