ci: consolidated platform & e2e stabilization batch - #2948
Conversation
…t provisioning delay (#2946) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…volumeSizeLimitMB (#2943) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… apps (#2874) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…2928) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…eral (#2873) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR centralizes Docker Buildx cache configuration into a ChangesCentralized Docker Buildx Registry Cache
Cilium Endpoint Leak Self-Healer for e2e
e2e Infrastructure Hardening and Diagnostics
SeaweedFS COSI and Configuration
Miscellaneous Infrastructure Fixes
Sequence DiagramssequenceDiagram
participant GHA as GitHub Actions
participant Make as make build
participant Buildx as docker buildx
participant OCIR as Registry
rect rgba(70, 130, 180, 0.5)
Note over GHA,OCIR: main push — cache warm-up
GHA->>Make: WRITE_CACHE=1, IMAGE_TAG=main
Make->>Buildx: $(call cache-args) → --cache-from + --cache-to mode=max
Buildx->>OCIR: pull existing cache
Buildx->>OCIR: push mode=max cache manifest
end
rect rgba(60, 179, 113, 0.5)
Note over GHA,OCIR: PR build — read-only cache
GHA->>Make: WRITE_CACHE=0 (default)
Make->>Buildx: $(call cache-args) → --cache-from only
Buildx->>OCIR: pull cached layers (no write)
end
sequenceDiagram
participant Healer as cilium-leak-healer Job
participant K8sAPI as Kubernetes API
participant CiliumAgent as cilium-agent
loop every POLL_INTERVAL seconds
Healer->>K8sAPI: get events reason=FailedCreatePodSandBox
K8sAPI-->>Healer: event list with "IP already in use"
Healer->>K8sAPI: get pod phase (freshness check)
K8sAPI-->>Healer: pod phase
alt pod Running/Succeeded
Healer->>Healer: skip
else pod wedged
Healer->>CiliumAgent: cilium-dbg endpoint get ipv4:<ip>
CiliumAgent-->>Healer: endpoint JSON or empty
alt endpoint absent
Healer->>Healer: skip (already cleared)
else endpoint present
Healer->>K8sAPI: get owning pod IP+phase
K8sAPI-->>Healer: pod status
alt live Running pod owns IP
Healer->>Healer: REFUSE log
else confirmed orphan
Healer->>CiliumAgent: cilium-dbg endpoint disconnect ipv4:<ip>
CiliumAgent-->>Healer: HEAL log
end
end
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related issues
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request consolidates eight critical platform and CI stabilization fixes into a single, dependency-ordered batch. The primary goal is to resolve intermittent CI flakes caused by resource contention, CNI endpoint leaks, and timing issues in the integration test suite. By addressing these issues collectively, the PR ensures a stable and reliable CI environment for the project. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces a unified buildx caching mechanism using a new cache-args macro across all package Makefiles, and adds an in-cluster watchdog Job (cilium-leak-healer) to mitigate Cilium in-memory endpoint leaks during E2E tests. It also improves the robustness of E2E setup scripts, such as using a shared deadline helper in e2e-post-install-prep.sh, fixing port-forwarding in bucket.bats, and adjusting SeaweedFS volume limits. Review feedback suggests refining the Cilium leak healer script: first, by ensuring pods in the Pending phase are not incorrectly treated as stale when checking owner phases, and second, by replacing the inline && with a standard if statement around the while loop for better shell compatibility and readability.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if [ -n "$epns" ] && [ -n "$eppod" ]; then | ||
| owner_ip=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.podIP}' 2>/dev/null) | ||
| owner_phase=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.phase}' 2>/dev/null) | ||
| if [ "$owner_ip" = "$ip" ] && [ "$owner_phase" = "Running" ]; then |
There was a problem hiding this comment.
Checking [ "$owner_phase" = "Running" ] might incorrectly classify a live pod in the Pending phase (for example, while it is running init containers or pulling images) as stale. If its IPAM allocation is still active, disconnecting its endpoint could disrupt its startup.
Consider refusing the disconnection if the owner pod is in any non-terminal phase (i.e., not Succeeded and not Failed).
| if [ "$owner_ip" = "$ip" ] && [ "$owner_phase" = "Running" ]; then | |
| if [ "$owner_ip" = "$ip" ] && [ "$owner_phase" != "Succeeded" ] && [ "$owner_phase" != "Failed" ]; then |
| [ -n "$events" ] && while IFS='|' read -r ns pod msg; do | ||
| [ -n "$ns" ] && [ -n "$pod" ] || continue | ||
| ip=$(printf '%s' "$msg" | grep -oE 'ipv4:[0-9]{1,3}(\.[0-9]{1,3}){3}' | head -1 | cut -d: -f2) | ||
| [ -n "$ip" ] || continue | ||
|
|
||
| # Only act on a pod that is still wedged (re-checked here = freshness gate). | ||
| phase=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.status.phase}' 2>/dev/null) | ||
| case "$phase" in Running|Succeeded|"") continue ;; esac | ||
|
|
||
| node=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null) | ||
| [ -n "$node" ] || continue | ||
| agent=$(agent_on_node "$node") | ||
| [ -n "$agent" ] || { log "no cilium-agent on node=$node for $ns/$pod ip=$ip"; continue; } | ||
|
|
||
| # Does an endpoint in this agent's registry hold the disputed IP at all? | ||
| # (The wedged pod itself was never registered — CreateEndpoint rejected it | ||
| # before exposing — so any endpoint holding the IP is the stale one.) | ||
| ep_json=$(kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \ | ||
| cilium-dbg endpoint get "ipv4:$ip" -o json 2>/dev/null) | ||
| [ -n "$ep_json" ] && [ "$ep_json" != "[]" ] || \ | ||
| { log "no endpoint holds ip=$ip on node=$node (already cleared?) for $ns/$pod"; continue; } | ||
|
|
||
| epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null) | ||
| epns=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-namespace" // empty' 2>/dev/null) | ||
| eppod=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-pod-name" // empty' 2>/dev/null) | ||
|
|
||
| # If that endpoint claims a pod that is alive on the cluster with this IP, | ||
| # it is a LIVE endpoint -> a genuine duplicate-IP, NOT the leak. Refuse and | ||
| # shout so the real problem is visible instead of silently broken. | ||
| if [ -n "$epns" ] && [ -n "$eppod" ]; then | ||
| owner_ip=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.podIP}' 2>/dev/null) | ||
| owner_phase=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.phase}' 2>/dev/null) | ||
| if [ "$owner_ip" = "$ip" ] && [ "$owner_phase" = "Running" ]; then | ||
| log "REFUSE ip=$ip on node=$node: held by LIVE pod $epns/$eppod (genuine duplicate IP, not the leak) — investigate" | ||
| continue | ||
| fi | ||
| fi | ||
|
|
||
| # Confirmed orphan: endpoint $epid holds $ip, backs no live pod, and pod | ||
| # $ns/$pod is wedged requesting the same IP. Evict only this endpoint. | ||
| log "HEAL node=$node agent=$agent ep=$epid ip=$ip wedged=$ns/$pod -> endpoint disconnect ipv4:$ip" | ||
| out=$(kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \ | ||
| cilium-dbg endpoint disconnect "ipv4:$ip" 2>&1) | ||
| log " result: ${out:-<none>}" | ||
| done <<EOF | ||
| $events | ||
| EOF |
There was a problem hiding this comment.
Mixing the && operator with a while loop and a here-doc (<<EOF) can be hard to read and may lead to unexpected parsing behaviors in some shell implementations.
Using a standard if statement is more readable and robust.
if [ -n "$events" ]; then
while IFS='|' read -r ns pod msg; do
[ -n "$ns" ] && [ -n "$pod" ] || continue
ip=$(printf '%s' "$msg" | grep -oE 'ipv4:[0-9]{1,3}(\.[0-9]{1,3}){3}' | head -1 | cut -d: -f2)
[ -n "$ip" ] || continue
# Only act on a pod that is still wedged (re-checked here = freshness gate).
phase=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.status.phase}' 2>/dev/null)
case "$phase" in Running|Succeeded|"") continue ;; esac
node=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
[ -n "$node" ] || continue
agent=$(agent_on_node "$node")
[ -n "$agent" ] || { log "no cilium-agent on node=$node for $ns/$pod ip=$ip"; continue; }
# Does an endpoint in this agent's registry hold the disputed IP at all?
# (The wedged pod itself was never registered — CreateEndpoint rejected it
# before exposing — so any endpoint holding the IP is the stale one.)
ep_json=$(kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \
cilium-dbg endpoint get "ipv4:$ip" -o json 2>/dev/null)
[ -n "$ep_json" ] && [ "$ep_json" != "[]" ] || \
{ log "no endpoint holds ip=$ip on node=$node (already cleared?) for $ns/$pod"; continue; }
epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null)
epns=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-namespace" // empty' 2>/dev/null)
eppod=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-pod-name" // empty' 2>/dev/null)
# If that endpoint claims a pod that is alive on the cluster with this IP,
# it is a LIVE endpoint -> a genuine duplicate-IP, NOT the leak. Refuse and
# shout so the real problem is visible instead of silently broken.
if [ -n "$epns" ] && [ -n "$eppod" ]; then
owner_ip=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.podIP}' 2>/dev/null)
owner_phase=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.phase}' 2>/dev/null)
if [ "$owner_ip" = "$ip" ] && [ "$owner_phase" != "Succeeded" ] && [ "$owner_phase" != "Failed" ]; then
log "REFUSE ip=$ip on node=$node: held by LIVE pod $epns/$eppod (genuine duplicate IP, not the leak) — investigate"
continue
fi
fi
# Confirmed orphan: endpoint $epid holds $ip, backs no live pod, and pod
# $ns/$pod is wedged requesting the same IP. Evict only this endpoint.
log "HEAL node=$node agent=$agent ep=$epid ip=$ip wedged=$ns/$pod -> endpoint disconnect ipv4:$ip"
out=$(kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \
cilium-dbg endpoint disconnect "ipv4:$ip" 2>&1)
log " result: ${out:-<none>}"
done <<EOF
$events
EOF
fiThere was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/build-main.yaml:
- Around line 44-48: Pin the workflow actions to specific commit SHAs instead of
version tags to prevent tag-retargeting attacks. Replace actions/checkout@v4
with its commit SHA digest, docker/login-action@v3 with its SHA, and
docker/setup-buildx-action@v3 with its SHA. Additionally, in the
actions/checkout step, add persist-credentials: false to the with block to
disable credential persistence and reduce the token exposure surface in this
privileged workflow.
In @.github/workflows/pull-requests.yaml:
- Around line 67-72: The helm plugin installation and flux script installation
commands lack version pinning and checksum verification, creating supply-chain
risks. For the helm plugin install command, add the --version flag to pin a
specific version of helm-unittest. For the flux installation, replace the direct
pipe to sudo bash with a safer approach: download the install script, verify its
checksum against the published checksums.txt file from the flux release, and
then execute it only after verification succeeds. This ensures both
installations are deterministic and verified across CI builds.
In `@hack/e2e-cilium-endpoint-leak-healer.sh`:
- Around line 50-55: The agent_on_node function blindly selects the first pod
from the list using .items[0] in the jsonpath query without verifying it is in a
Running state, which can result in selecting Pending, Terminating, or Failed
pods during DaemonSet rollouts. Modify the function to filter for Running pods
by adding a field selector for status.phase=Running in the kubectl get pod
command or by updating the jsonpath query to select only from pods with running
status, ensuring that a Running pod's name is returned instead of the first pod
regardless of its state.
In `@hack/e2e-cilium-leak-healer.yaml`:
- Around line 76-96: The healer container in the pod specification lacks an
explicit securityContext, which is problematic given that the pod uses
hostNetwork: true and has pods/exec RBAC permissions. Add a securityContext
field to the healer container that restricts unnecessary capabilities by
dropping all capabilities and only adding back those required for the healing
operation, sets readOnlyRootFilesystem to true since the ConfigMap mount is
read-only, ensures the container runs as a non-root user, and prevents privilege
escalation. This hardening will reduce the attack surface if the container is
compromised while remaining compatible with the container's read-only
configuration.
- Around line 31-35: The pods/exec resource permission is currently granted
cluster-wide through the ClusterRole, but the healer script only executes within
the cozy-cilium namespace. To follow least-privilege principle, split the
permissions: keep the read-only cluster-wide discovery rules (events, pods) in
the existing ClusterRole, and create a separate namespace-scoped Role in the
cozy-cilium namespace that contains the pods/exec permission with verbs create,
then create a RoleBinding in cozy-cilium that binds this Role to the same
service account. This restricts exec access to only the cozy-cilium namespace as
the script intends.
In `@hack/e2e-install-cozystack.bats`:
- Around line 18-21: The kubectl commands creating the cilium-leak-healer
ConfigMap and applying the e2e-cilium-leak-healer.yaml manifest are both using
the || true operator to suppress failures. Remove the || true from both the
kubectl create configmap command on line 20 and the kubectl apply command on
line 21. This ensures that if either the ConfigMap creation or the Job manifest
application fails due to missing files, permission issues, or namespace
problems, the script will fail explicitly rather than silently, preventing the
watchdog from running unnoticed.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: cfb86fdc-13ec-4134-be94-4c6125dcb6ba
📒 Files selected for processing (44)
.github/workflows/build-main.yaml.github/workflows/pull-requests.yamlhack/common-envs.mkhack/e2e-apps/bucket.batshack/e2e-cilium-endpoint-leak-healer.shhack/e2e-cilium-leak-healer.yamlhack/e2e-install-cozystack.batshack/e2e-post-install-prep.shpackages/apps/clickhouse/Makefilepackages/apps/http-cache/Makefilepackages/apps/kubernetes/Makefilepackages/apps/mariadb/Makefilepackages/core/installer/Makefilepackages/core/platform/Makefilepackages/core/talos/Makefilepackages/core/testing/Makefilepackages/extra/monitoring/Makefilepackages/system/backup-controller/Makefilepackages/system/backupstrategy-controller/Makefilepackages/system/bucket/Makefilepackages/system/capi-providers-core/files/core-components.yamlpackages/system/cilium/Makefilepackages/system/cozystack-api/Makefilepackages/system/cozystack-controller/Makefilepackages/system/dashboard/Makefilepackages/system/flux-plunger/Makefilepackages/system/grafana-operator/Makefilepackages/system/kamaji/Makefilepackages/system/keycloak-operator/Makefilepackages/system/kilo/Makefilepackages/system/kubeovn-plunger/Makefilepackages/system/kubeovn-webhook/Makefilepackages/system/lineage-controller-webhook/Makefilepackages/system/linstor-gui/Makefilepackages/system/linstor/Makefilepackages/system/metallb/Makefilepackages/system/metallb/tests/metallb_test.yamlpackages/system/monitoring/Makefilepackages/system/multus/Makefilepackages/system/objectstorage-controller/Makefilepackages/system/redis-operator/Makefilepackages/system/seaweedfs/charts/seaweedfs/templates/cosi/cosi-bucket-class.yamlpackages/system/seaweedfs/charts/seaweedfs/templates/s3/s3-service.yamlpackages/system/seaweedfs/values.yaml
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| fetch-tags: true |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n .github/workflows/build-main.yamlRepository: cozystack/cozystack
Length of output: 3864
🌐 Web query:
GitHub Actions security best practices SHA pinning vs version tags 2024 2025
💡 Result:
In GitHub Actions, pinning to a full-length commit SHA is the recommended security best practice, as it provides an immutable reference to the action's code [1][2][3]. Using version tags (e.g., @v4) or branch names (e.g., @main) is inherently risky because these references are mutable; a compromised maintainer or an attacker who gains access to the action's repository can retarget them to point to malicious code without notice [4][3][5]. SHA pinning prevents "tag-retargeting" supply chain attacks [4][6]. Because a commit SHA uniquely identifies a specific state of the repository, it guarantees that your workflow executes the exact code you reviewed and approved, regardless of changes made to tags or branches upstream [2][3]. To implement this effectively: 1. Use full 40-character commit SHAs in your workflow files (e.g., uses: actions/checkout@3df4ab1) [1][7]. 2. Include the original version tag as a comment to maintain human readability and simplify updates (e.g., uses: actions/checkout@3df4ab1 # v4.0.0) [4][6][7]. 3. Automate the update process using tools like Dependabot or Renovate (using specific presets like helpers:pinGitHubActionDigests) to keep SHAs current without manual intervention [4][8][6][7]. 4. Enforce SHA pinning at the organization or repository level via GitHub's policy settings, which can cause workflows using unpinned actions to fail automatically [1][3][9]. While SHA pinning is a critical security control, it is not a complete solution. It does not protect against malicious code that was present in the chosen commit, nor does it necessarily prevent actions from pulling in other mutable dependencies (such as Docker images tagged:latest) at runtime [8][5]. Organizations should combine SHA pinning with other security practices, such as applying the principle of least privilege to GITHUB_TOKEN permissions, utilizing OIDC for cloud authentication, and conducting regular audits of third-party actions [10][11][7].
Citations:
- 1: https://docs.github.com/en/actions/reference/security/secure-use
- 2: https://www.stepsecurity.io/blog/pinning-github-actions-for-enhanced-security-a-complete-guide
- 3: https://www.romainlespinasse.dev/posts/github-actions-commit-sha-pinning/
- 4: https://safeguard.sh/resources/blog/github-actions-pinning-sha-vs-tag
- 5: https://mikael.barbero.tech/blog/post/2026-03-24-stop-trusting-mutable-references/
- 6: https://architecture.lullabot.com/adr/20260505-pin-github-actions-to-hash/
- 7: https://secure-pipelines.com/ci-cd-security/github-actions-security-cheat-sheet/
- 8: https://emmer.dev/blog/pin-your-github-actions-to-protect-against-mutability/
- 9: https://github.blog/changelog/2025-08-15-github-actions-policy-now-supports-blocking-and-sha-pinning-actions/
- 10: https://cheatsheetseries.owasp.org/cheatsheets/GitHub_Actions_Security_Cheat_Sheet.html
- 11: https://blog.gitguardian.com/github-actions-security-cheat-sheet/
🌐 Web query:
actions/checkout persist-credentials default behavior
💡 Result:
The default behavior for the persist-credentials input in the GitHub actions/checkout action is true [1][2][3]. When set to true (the default), the action configures the authentication token or SSH key used for the checkout within the local git environment [1][3][4]. This allows subsequent steps in your workflow to execute authenticated git commands, such as git fetch, without needing to manually re-authenticate [1][3]. In versions of actions/checkout prior to v6.0.0, these credentials were stored directly in the local.git/config file [1][5][6]. Starting with v6.0.0, the action was updated to store credentials in a separate file under $RUNNER_TEMP to improve security [1][5][6]. Regardless of the storage method, the action ensures that these credentials are removed during the post-job cleanup process [1][3][7]. If you wish to prevent the action from configuring these credentials, you can opt-out by explicitly setting persist-credentials: false in your workflow file [1][3][8].
Citations:
- 1: https://github.com/actions/checkout/?tab=readme-ov-file
- 2: https://github.com/actions/checkout/blob/34e114876b0b11c390a56381ad16ebd13914f8d5/action.yml
- 3: https://github.com/actions/checkout/blob/master/adrs/0153-checkout-v2.md
- 4: https://github.com/actions/checkout/blob/72f2cec99f417b1a1c5e2e88945068983b7965f9/action.yml
- 5: actions/checkout@v5.0.1...v6.0.0
- 6: https://github.com/actions/checkout/tree/v6.0.2?tab=readme-ov-file
- 7: https://github.com/actions/checkout/blob/85e6279cec87321a52edac9c87bce653a07cf6c2/README.md
- 8: actions/checkout@c170eef
Pin workflow actions to commit SHA and disable checkout credential persistence.
actions/checkout@v4, docker/login-action@v3, and docker/setup-buildx-action@v3 use version tags instead of commit SHAs, allowing tag-retargeting attacks if a dependency is compromised. Additionally, checkout persists credentials by default, exposing an unnecessary token surface in this privileged workflow.
🔧 Proposed hardening
- name: Checkout code
- uses: actions/checkout@v4
+ uses: actions/checkout@3df4ab11eba7bda6032a0b82a6bb43b11571feac # v4.0.0
with:
fetch-depth: 0
fetch-tags: true
+ persist-credentials: false
- name: Login to OCIR
- uses: docker/login-action@v3
+ uses: docker/login-action@9780b0c56d5d59944534694c45ea9bc1d00c757e # v3.3.0
- name: Set up Buildx (docker-container driver)
id: buildx
- uses: docker/setup-buildx-action@v3
+ uses: docker/setup-buildx-action@d4c1175270439c506a768202a39eb90a979b35acb # v3.6.0Replace SHAs with the actual digests for your target versions, or use Dependabot with helpers:pinGitHubActionDigests preset to automate updates.
Also applies to: lines 57, 70
🧰 Tools
🪛 zizmor (1.25.2)
[warning] 44-48: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 45-45: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/build-main.yaml around lines 44 - 48, Pin the workflow
actions to specific commit SHAs instead of version tags to prevent
tag-retargeting attacks. Replace actions/checkout@v4 with its commit SHA digest,
docker/login-action@v3 with its SHA, and docker/setup-buildx-action@v3 with its
SHA. Additionally, in the actions/checkout step, add persist-credentials: false
to the with block to disable credential persistence and reduce the token
exposure surface in this privileged workflow.
Source: Linters/SAST tools
| run: | | ||
| helm plugin list 2>/dev/null | grep -q unittest \ | ||
| || helm plugin install https://github.com/helm-unittest/helm-unittest | ||
| command -v flux >/dev/null \ | ||
| || curl -s https://fluxcd.io/install.sh | sudo bash | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n .github/workflows/pull-requests.yaml | sed -n '60,80p'Repository: cozystack/cozystack
Length of output: 1142
🏁 Script executed:
# Check if there are any version constraints or requirements files
find . -type f \( -name "requirements*.txt" -o -name "requirements*.yaml" -o -name "constraints*" -o -name ".tool-versions" \) | head -20Repository: cozystack/cozystack
Length of output: 45
🏁 Script executed:
# Search for helm plugin install patterns elsewhere to understand versioning convention
rg "helm plugin install" -A 2Repository: cozystack/cozystack
Length of output: 284
🏁 Script executed:
# Check for flux version references in the repo
rg "flux.*version" -iRepository: cozystack/cozystack
Length of output: 3264
🌐 Web query:
helm-unittest plugin version pinning command syntax --version
💡 Result:
To pin a specific version of the helm-unittest plugin during installation, use the --version flag with the standard helm plugin install command [1][2]. The command syntax is: helm plugin install https://github.com/helm-unittest/helm-unittest --version <version_number> For example, to install version 1.0.3: helm plugin install https://github.com/helm-unittest/helm-unittest --version 1.0.3 Important considerations: 1. Compatibility with Helm 4: If you are using Helm 4, ensure you omit the.git extension from the repository URL [3]. Additionally, because Helm 4 does not support webhooks GPG verification for plugin installation, you may need to include the --verify=false flag [4][3]: helm plugin install https://github.com/helm-unittest/helm-unittest --version 1.0.3 --verify=false 2. Troubleshooting: If you encounter installation errors, ensure your Helm cache is clean and that you are using a compatible version of Helm [5][6]. The plugin documentation notes that installation issues can sometimes arise if previous installations or cached files conflict with the new version [1][5]. 3. Default behavior: If the --version flag is omitted, the helm plugin install command will install the latest available version by default [4][7].
Citations:
- 1: Cannot install latest version helm-unittest/helm-unittest#856
- 2: https://helm.sh/docs/topics/plugins/
- 3: README installation section is out-of-date helm-unittest/helm-unittest#831
- 4: https://pkg.go.dev/github.com/helm-unittest/helm-unittest
- 5: Error when installing via helm plugin install helm-unittest/helm-unittest#790
- 6: Unable to install the plugin helm-unittest/helm-unittest#130
- 7: https://github.com/helm-unittest/helm-unittest/
🌐 Web query:
flux fluxcd release checksums.txt GitHub releases
💡 Result:
Flux release checksums are published as a file named flux_checksums.txt within the assets of every release on the fluxcd/flux2 GitHub repository [1][2][3]. These checksum files allow users to verify the integrity of the downloaded artifacts (such as CLI binaries, SBOMs, and manifests) by comparing the file hashes against those listed in the text file [4][5]. In addition to the raw checksum text file, Flux provides cryptographic signatures and certificates for these files to ensure their authenticity [6][7]: - fluxchecksums.txt: Contains the SHA-256 hashes of the release assets [1][4]. - fluxchecksums.txt.sig: The digital signature of the checksums file [1][5]. - flux_checksums.txt.pem: The certificate associated with the signature [1][5]. These files can be verified using the Sigstore Cosign tool [5]. A common command to verify the checksum file is: cosign verify-blob checksums.txt \ --signature checksums.txt.sig \ --certificate checksums.txt.pem \ --certificate-identity="https://github.com/fluxcd/flux2/.github/workflows/release.yml@refs/tags/v" \ --certificate-oidc-issuer="https://token.actions.githubusercontent.com" For automated installation, the official Flux install script (install/flux.sh) automatically fetches the checksum file to verify the downloaded binary [4]. Beyond manual checksum verification, the project also provides SLSA provenance attestations and signed container images for enhanced security [6][8][9].
Citations:
- 1: https://github.com/fluxcd/flux2/releases/tag/v2.8.8
- 2: https://github.com/fluxcd/flux2/releases/tag/v2.8.7
- 3: https://github.com/fluxcd/flux2/releases/tag/v2.8.5
- 4: https://github.com/fluxcd/flux2/blob/main/install/flux.sh
- 5: https://oneuptime.com/blog/post/2026-03-13-how-to-verify-flux-cli-binary-signatures/view
- 6: https://v2-5.docs.fluxcd.io/flux/releases/
- 7: https://github.com/fluxcd/website/blob/main/content/en/flux/releases/_index.md
- 8: https://v2-0.docs.fluxcd.io/flux/security/
- 9: https://fluxcd.io/flux/releases/
Avoid unpinned remote installer execution with sudo bash.
Lines 69 and 71 execute unpinned plugin and script installations. Piping the flux install script directly to sudo bash bypasses the checksum verification that flux actually provides in its releases, introducing preventable supply-chain risk and nondeterministic CI behavior across builds.
Both can be hardened:
helm plugin installsupports--versionflag for version pinning- flux publishes
checksums.txtfor all releases; the official install script verifies them, but the direct pipe approach does not
🔧 Proposed hardening pattern
- helm plugin list 2>/dev/null | grep -q unittest \
- || helm plugin install https://github.com/helm-unittest/helm-unittest
+ helm plugin list 2>/dev/null | grep -q unittest \
+ || helm plugin install https://github.com/helm-unittest/helm-unittest --version v<pin>
- command -v flux >/dev/null \
- || curl -s https://fluxcd.io/install.sh | sudo bash
+ command -v flux >/dev/null || {
+ FLUX_VERSION=v<pin>
+ ARCHIVE="flux_${FLUX_VERSION#v}_linux_amd64.tar.gz"
+ curl -fsSLo /tmp/${ARCHIVE} "https://github.com/fluxcd/flux2/releases/download/${FLUX_VERSION}/${ARCHIVE}"
+ curl -fsSLo /tmp/flux_checksums.txt "https://github.com/fluxcd/flux2/releases/download/${FLUX_VERSION}/checksums.txt"
+ grep " ${ARCHIVE}$" /tmp/flux_checksums.txt | sha256sum -c -
+ tar -xzf /tmp/${ARCHIVE} -C /tmp flux
+ sudo install -m 0755 /tmp/flux /usr/local/bin/flux
+ }📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| run: | | |
| helm plugin list 2>/dev/null | grep -q unittest \ | |
| || helm plugin install https://github.com/helm-unittest/helm-unittest | |
| command -v flux >/dev/null \ | |
| || curl -s https://fluxcd.io/install.sh | sudo bash | |
| run: | | |
| helm plugin list 2>/dev/null | grep -q unittest \ | |
| || helm plugin install https://github.com/helm-unittest/helm-unittest --version v<pin> | |
| command -v flux >/dev/null || { | |
| FLUX_VERSION=v<pin> | |
| ARCHIVE="flux_${FLUX_VERSION#v}_linux_amd64.tar.gz" | |
| curl -fsSLo /tmp/${ARCHIVE} "https://github.com/fluxcd/flux2/releases/download/${FLUX_VERSION}/${ARCHIVE}" | |
| curl -fsSLo /tmp/flux_checksums.txt "https://github.com/fluxcd/flux2/releases/download/${FLUX_VERSION}/checksums.txt" | |
| grep " ${ARCHIVE}$" /tmp/flux_checksums.txt | sha256sum -c - | |
| tar -xzf /tmp/${ARCHIVE} -C /tmp flux | |
| sudo install -m 0755 /tmp/flux /usr/local/bin/flux | |
| } | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/pull-requests.yaml around lines 67 - 72, The helm plugin
installation and flux script installation commands lack version pinning and
checksum verification, creating supply-chain risks. For the helm plugin install
command, add the --version flag to pin a specific version of helm-unittest. For
the flux installation, replace the direct pipe to sudo bash with a safer
approach: download the install script, verify its checksum against the published
checksums.txt file from the flux release, and then execute it only after
verification succeeds. This ensures both installations are deterministic and
verified across CI builds.
| # cilium-agent pod running on a given node (DaemonSet selector k8s-app=cilium). | ||
| agent_on_node() { | ||
| kubectl get pod -n "$CILIUM_NS" -l k8s-app=cilium \ | ||
| --field-selector "spec.nodeName=$1" \ | ||
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's look at the actual file structure and content
cat -n hack/e2e-cilium-endpoint-leak-healer.sh | head -70Repository: cozystack/cozystack
Length of output: 4374
🏁 Script executed:
# Search for usages of agent_on_node function
rg "agent_on_node" hack/e2e-cilium-endpoint-leak-healer.shRepository: cozystack/cozystack
Length of output: 116
🏁 Script executed:
# Look for kubectl exec calls that might use the output
rg "kubectl exec" hack/e2e-cilium-endpoint-leak-healer.sh -A 2 -B 2Repository: cozystack/cozystack
Length of output: 759
🏁 Script executed:
# Test JSONPath filtering syntax used in the suggestion
# First, let's check what kubectl version is expected and if JSONPath filters work
grep -E "(kubectl|version)" hack/e2e-cilium-endpoint-leak-healer.sh | head -5Repository: cozystack/cozystack
Length of output: 436
🏁 Script executed:
# Let's also check if there are any other similar patterns in the codebase for filtering by status
rg "status.phase" --type shRepository: cozystack/cozystack
Length of output: 3398
🏁 Script executed:
# Check if the file has any comments about pod selection reliability
rg -B2 -A2 "items\[0\]" hack/e2e-cilium-endpoint-leak-healer.shRepository: cozystack/cozystack
Length of output: 218
🏁 Script executed:
# Check the full context of the agent_on_node function usage and how errors are handled
sed -n '50,120p' hack/e2e-cilium-endpoint-leak-healer.shRepository: cozystack/cozystack
Length of output: 3940
🏁 Script executed:
# Look for similar pattern with status filtering in foundationdb helpers mentioned in grep results
cat -n examples/backups/foundationdb/00-helpers.sh | grep -B3 -A3 "status.phase=Running"Repository: cozystack/cozystack
Length of output: 523
🏁 Script executed:
# Verify the kubectl JSONPath filter syntax is valid
# Check if there are any JSONPath range filters in the codebase
rg "range.*items\[" --type sh | head -10Repository: cozystack/cozystack
Length of output: 1327
🏁 Script executed:
# Also check what kubectl version/features are expected
fd -e yaml | xargs grep -l "kubectl" | head -3 | xargs cat | grep -i "image.*kubectl" | head -3Repository: cozystack/cozystack
Length of output: 156
🏁 Script executed:
# Verify both approaches work with kubectl's field-selector and JSONPath
# Check if there are examples of both approaches in the codebase
# Approach 1: field-selector with status.phase (simpler)
grep -n "field-selector.*status.phase" hack/e2e-cilium-endpoint-leak-healer.sh examples/backups/foundationdb/*.sh
# Approach 2: JSONPath filtering (what the review suggests)
grep -n "items\[?.*status" hack/*.sh | head -5Repository: cozystack/cozystack
Length of output: 695
🏁 Script executed:
# Let's verify the actual behavior difference
# Check what happens when agent_on_node returns empty or non-running pod
sed -n '82,92p' hack/e2e-cilium-endpoint-leak-healer.shRepository: cozystack/cozystack
Length of output: 720
🏁 Script executed:
# Verify that kubectl exec actually fails on non-Running pods
# Let's check the kubectl documentation pattern in the codebase
# Also verify the condition check at line 89 only checks for empty string
sed -n '87,91p' hack/e2e-cilium-endpoint-leak-healer.shRepository: cozystack/cozystack
Length of output: 433
🏁 Script executed:
# Check if there's any error logging when kubectl exec fails silently
# The error is redirected to /dev/null, so failures are not logged
sed -n '100,115p' hack/e2e-cilium-endpoint-leak-healer.sh | cat -nRepository: cozystack/cozystack
Length of output: 1080
Select a Running cilium-agent pod instead of blindly taking .items[0].
The current code at line 54 can resolve to a Pending, Terminating, or Failed pod during DaemonSet rollouts or restarts. This causes kubectl exec to fail silently (stderr redirected to /dev/null), skipping endpoint healing for that node until the next poll interval.
Suggested patch
agent_on_node() {
kubectl get pod -n "$CILIUM_NS" -l k8s-app=cilium \
--field-selector "spec.nodeName=$1" \
- -o jsonpath='{.items[0].metadata.name}' 2>/dev/null
+ -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}' 2>/dev/null \
+ | head -n1
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # cilium-agent pod running on a given node (DaemonSet selector k8s-app=cilium). | |
| agent_on_node() { | |
| kubectl get pod -n "$CILIUM_NS" -l k8s-app=cilium \ | |
| --field-selector "spec.nodeName=$1" \ | |
| -o jsonpath='{.items[0].metadata.name}' 2>/dev/null | |
| } | |
| # cilium-agent pod running on a given node (DaemonSet selector k8s-app=cilium). | |
| agent_on_node() { | |
| kubectl get pod -n "$CILIUM_NS" -l k8s-app=cilium \ | |
| --field-selector "spec.nodeName=$1" \ | |
| -o jsonpath='{range .items[?(@.status.phase=="Running")]}{.metadata.name}{"\n"}{end}' 2>/dev/null \ | |
| | head -n1 | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-cilium-endpoint-leak-healer.sh` around lines 50 - 55, The
agent_on_node function blindly selects the first pod from the list using
.items[0] in the jsonpath query without verifying it is in a Running state,
which can result in selecting Pending, Terminating, or Failed pods during
DaemonSet rollouts. Modify the function to filter for Running pods by adding a
field selector for status.phase=Running in the kubectl get pod command or by
updating the jsonpath query to select only from pods with running status,
ensuring that a Running pod's name is returned instead of the first pod
regardless of its state.
| # run `cilium-dbg endpoint ...` inside the owning node's cilium-agent | ||
| - apiGroups: [""] | ||
| resources: ["pods/exec"] | ||
| verbs: ["create"] | ||
| --- |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, locate and read the YAML file
find . -name "e2e-cilium-leak-healer.yaml" -type fRepository: cozystack/cozystack
Length of output: 98
🏁 Script executed:
# Once found, read the file to see the complete RBAC structure
cat -n hack/e2e-cilium-leak-healer.yamlRepository: cozystack/cozystack
Length of output: 4158
🏁 Script executed:
# Search for references to cozy-cilium and exec in the repository
rg "cozy-cilium" --type yaml -B 2 -A 2Repository: cozystack/cozystack
Length of output: 5452
🏁 Script executed:
# Search for exec invocations or patterns in shell scripts
rg "exec" hack/ -B 2 -A 2Repository: cozystack/cozystack
Length of output: 6447
Scope pods/exec to cozy-cilium namespace instead of granting it cluster-wide.
The healer script (hack/e2e-cilium-endpoint-leak-healer.sh) explicitly limits exec calls to the cozy-cilium namespace using -n "$CILIUM_NS", but the current ClusterRole allows exec into any pod cluster-wide. This violates least-privilege: keep the ClusterRole for read-only cluster-wide discovery rules (events, pods), and move pods/exec to a namespace-scoped Role in cozy-cilium.
Suggested patch direction
# In ClusterRole: keep read-only cluster-wide discovery rules,
# remove cluster-wide pods/exec.
- - apiGroups: [""]
- resources: ["pods/exec"]
- verbs: ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+ name: cilium-leak-healer-exec
+ namespace: cozy-cilium
+rules:
+ - apiGroups: [""]
+ resources: ["pods/exec"]
+ verbs: ["create"]
+
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+ name: cilium-leak-healer-exec
+ namespace: cozy-cilium
+roleRef:
+ apiGroup: rbac.authorization.k8s.io
+ kind: Role
+ name: cilium-leak-healer-exec
+subjects:
+ - kind: ServiceAccount
+ name: cilium-leak-healer
+ namespace: kube-system📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # run `cilium-dbg endpoint ...` inside the owning node's cilium-agent | |
| - apiGroups: [""] | |
| resources: ["pods/exec"] | |
| verbs: ["create"] | |
| --- | |
| # run `cilium-dbg endpoint ...` inside the owning node's cilium-agent | |
| --- | |
| apiVersion: rbac.authorization.k8s.io/v1 | |
| kind: Role | |
| metadata: | |
| name: cilium-leak-healer-exec | |
| namespace: cozy-cilium | |
| rules: | |
| - apiGroups: [""] | |
| resources: ["pods/exec"] | |
| verbs: ["create"] | |
| --- | |
| apiVersion: rbac.authorization.k8s.io/v1 | |
| kind: RoleBinding | |
| metadata: | |
| name: cilium-leak-healer-exec | |
| namespace: cozy-cilium | |
| roleRef: | |
| apiGroup: rbac.authorization.k8s.io | |
| kind: Role | |
| name: cilium-leak-healer-exec | |
| subjects: | |
| - kind: ServiceAccount | |
| name: cilium-leak-healer | |
| namespace: kube-system |
🧰 Tools
🪛 Trivy (0.69.3)
[error] 31-33: Exec into Pods
ClusterRole 'cilium-leak-healer' should not have access to resource '["pods/exec"]' for verbs ["create", "update", "patch", "delete", "deletecollection", "impersonate", "*"]
Rule: KSV-0053
(IaC/Kubernetes)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-cilium-leak-healer.yaml` around lines 31 - 35, The pods/exec
resource permission is currently granted cluster-wide through the ClusterRole,
but the healer script only executes within the cozy-cilium namespace. To follow
least-privilege principle, split the permissions: keep the read-only
cluster-wide discovery rules (events, pods) in the existing ClusterRole, and
create a separate namespace-scoped Role in the cozy-cilium namespace that
contains the pods/exec permission with verbs create, then create a RoleBinding
in cozy-cilium that binds this Role to the same service account. This restricts
exec access to only the cozy-cilium namespace as the script intends.
Source: Linters/SAST tools
| tolerations: | ||
| - operator: Exists | ||
| containers: | ||
| - name: healer | ||
| # Reuse the in-tree, digest-pinned kubectl+jq image already shipped by | ||
| # the installer (packages/core/installer/templates/cozy-system-labels.yaml). | ||
| image: docker.io/alpine/k8s:1.33.4@sha256:b0523f0a244ddc4c8e055aa335c040d3d78b3ead5528f4544395f7f9f69c7b68 | ||
| command: ["/bin/sh", "/opt/healer/heal.sh"] | ||
| env: | ||
| - name: CILIUM_NS | ||
| value: cozy-cilium | ||
| volumeMounts: | ||
| - name: script | ||
| mountPath: /opt/healer | ||
| resources: | ||
| requests: | ||
| cpu: 10m | ||
| memory: 32Mi | ||
| limits: | ||
| memory: 128Mi | ||
| volumes: |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat hack/e2e-cilium-leak-healer.yamlRepository: cozystack/cozystack
Length of output: 3465
Add explicit container security context; default settings are unnecessarily permissive with hostNetwork and exec RBAC.
The pod uses hostNetwork: true (required for healing the leak during cluster bootstrap) and grants pods/exec permissions. Without explicit container hardening, a compromise gains elevated capabilities. The ConfigMap-only mount is read-only, making the suggested hardening straightforward to implement.
Suggested patch
containers:
- name: healer
image: docker.io/alpine/k8s:1.33.4@sha256:b0523f0a244ddc4c8e055aa335c040d3d78b3ead5528f4544395f7f9f69c7b68
command: ["/bin/sh", "/opt/healer/heal.sh"]
+ securityContext:
+ allowPrivilegeEscalation: false
+ readOnlyRootFilesystem: true
+ capabilities:
+ drop: ["ALL"]📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| tolerations: | |
| - operator: Exists | |
| containers: | |
| - name: healer | |
| # Reuse the in-tree, digest-pinned kubectl+jq image already shipped by | |
| # the installer (packages/core/installer/templates/cozy-system-labels.yaml). | |
| image: docker.io/alpine/k8s:1.33.4@sha256:b0523f0a244ddc4c8e055aa335c040d3d78b3ead5528f4544395f7f9f69c7b68 | |
| command: ["/bin/sh", "/opt/healer/heal.sh"] | |
| env: | |
| - name: CILIUM_NS | |
| value: cozy-cilium | |
| volumeMounts: | |
| - name: script | |
| mountPath: /opt/healer | |
| resources: | |
| requests: | |
| cpu: 10m | |
| memory: 32Mi | |
| limits: | |
| memory: 128Mi | |
| volumes: | |
| tolerations: | |
| - operator: Exists | |
| containers: | |
| - name: healer | |
| # Reuse the in-tree, digest-pinned kubectl+jq image already shipped by | |
| # the installer (packages/core/installer/templates/cozy-system-labels.yaml). | |
| image: docker.io/alpine/k8s:1.33.4@sha256:b0523f0a244ddc4c8e055aa335c040d3d78b3ead5528f4544395f7f9f69c7b68 | |
| command: ["/bin/sh", "/opt/healer/heal.sh"] | |
| securityContext: | |
| allowPrivilegeEscalation: false | |
| readOnlyRootFilesystem: true | |
| capabilities: | |
| drop: ["ALL"] | |
| env: | |
| - name: CILIUM_NS | |
| value: cozy-cilium | |
| volumeMounts: | |
| - name: script | |
| mountPath: /opt/healer | |
| resources: | |
| requests: | |
| cpu: 10m | |
| memory: 32Mi | |
| limits: | |
| memory: 128Mi | |
| volumes: |
🧰 Tools
🪛 Trivy (0.69.3)
[error] 55-96: Access to host network
Job 'cilium-leak-healer' should not set 'spec.template.spec.hostNetwork' to true
Rule: KSV-0009
(IaC/Kubernetes)
[error] 76-92: Root file system is not read-only
Container 'healer' of Job 'cilium-leak-healer' should set 'securityContext.readOnlyRootFilesystem' to true
Rule: KSV-0014
(IaC/Kubernetes)
[error] 76-92: Default security context configured
container cilium-leak-healer in kube-system namespace is using the default security context
Rule: KSV-0118
(IaC/Kubernetes)
[error] 64-96: Default security context configured
job cilium-leak-healer in kube-system namespace is using the default security context, which allows root privileges
Rule: KSV-0118
(IaC/Kubernetes)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@hack/e2e-cilium-leak-healer.yaml` around lines 76 - 96, The healer container
in the pod specification lacks an explicit securityContext, which is problematic
given that the pod uses hostNetwork: true and has pods/exec RBAC permissions.
Add a securityContext field to the healer container that restricts unnecessary
capabilities by dropping all capabilities and only adding back those required
for the healing operation, sets readOnlyRootFilesystem to true since the
ConfigMap mount is read-only, ensures the container runs as a non-root user, and
prevents privilege escalation. This hardening will reduce the attack surface if
the container is compromised while remaining compatible with the container's
read-only configuration.
Source: Linters/SAST tools
…viction churn In Auto mode the VPA Updater evicts running pods to apply new recommendations. During install that eviction churn recreates VictoriaMetrics pods faster than Cilium releases their endpoint IPs, tripping the cluster-pool IPAM "IP already in use" (putEndpointIdInvalid) leak that wedges a pod and stalls the install. Initial sets requests at pod creation without ever evicting, removing the trigger (the cilium-leak healer remains as defense for non-VPA churn paths). Also raise vmstorage/vmselect minAllowed floors to 200m/512Mi: under Initial a pod is sized once at creation and never re-sized at runtime, so the floor is the safety net against the heavy query/storage tiers being left under-provisioned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Auto mode evicts running etcd members to apply VPA recommendations, churning the StatefulSet (disruptive to quorum and a Cilium IP-reuse-leak trigger under load). Initial sizes at pod creation without evicting; the existing 250m/256Mi minAllowed floor covers the no-runtime-resize tradeoff. Same eviction-churn class as the monitoring VPA fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…bats Reconfiguring the Etcd CR triggers a HelmRelease upgrade that renders the EtcdBackupSchedule; hr/etcd and etcdcluster can report ready on the prior revision before the new resource materializes, so the immediate get raced to NotFound. Poll for it (timeout 120s), mirroring the CronJob wait already used later in the same test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
…t budgets kubernetes-latest flaked because the worker-readiness wait was split into 8m-to-join + 3m-to-become-Ready. Under sandbox load a slow KubeVirt VM boot consumes most of the join budget, then the tenant cluster's cilium CNI needs several more minutes to make the freshly-joined nodes Ready, overflowing the fixed 3m Ready window even though the CNI converges fine (its HelmRelease reached True shortly after). Replace with one 12m deadline polling for >=2 Ready nodes, robust to wherever the time goes. Same single-deadline shape as the LINSTOR post-install fix (#2928). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
… hook The e2e runner hack/cozytest.sh converts only @test blocks into functions and runs those; it never invokes bats setup_file/teardown_file. So the healer's setup_file deploy silently never executed (confirmed: the Job was absent from the cozyreport while the same manifest deploys and runs fine on a real Talos cozystack cluster). Move the deploy into a real, first @test so cozytest.sh runs it (with visible set -x trace) before the install churn, and assert the Job landed without ever failing the suite (band-aid). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
kubernetes-{latest,previous} intermittently failed their node-Ready wait
with the tenant API "connection refused" on localhost, while the cluster was
in fact healthy (host snapshot: both CAPI Machines NodeHealthy=True, TCP
Ready 2/2). Root cause: `kubectl port-forward service/kubernetes-<name>`
pins to ONE of the two Kamaji control-plane pods; when that pod's
kube-apiserver container restarted once (observed restartCount=1), the
forward broke and did not recover, so the 12m wait ran against a dead socket
and the failure-path describe/snapshot also got nothing.
Expose the tenant API with a test-scoped LoadBalancer Service selecting the
Kamaji control-plane pods instead. It load-balances across ALL ready apiserver
endpoints, so a single pod's apiserver restart is routed around transparently;
the host runs MetalLB on the sandbox /24 (pool 192.168.123.200-250) so the IP
is directly routable from the test (the in-tenant LB test already curls such
an address). The kubeconfig points at the LB IP with insecure-skip-tls-verify
(the MetalLB IP isn't in the apiserver cert SANs; e2e only) and the CA dropped.
Bonus: the LB endpoint is stable until teardown, so the on-failure tenant
crust-gather snapshot can finally reach the tenant (the port-forward was dead
by snapshot time, which is why every prior tenant snapshot came back empty).
Removes the port-forward supervisor + pidfile machinery. Verified against the
real hack/cozytest.sh under dash (CI's /bin/sh) in ubuntu:22.04: no pipeline
deadlock, the in-subshell snapshot trap fires and captures the tenant, and the
LB Service is cleaned up on both the success path and cozy_cleanup.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
No code change. The 'debug' PR label was pinning E2E to the contended self-hosted runner; with it removed this synchronize event re-evaluates runs-on to the ephemeral oracle-vm pool. Empty commit because the workflow only triggers on opened/synchronize/reopened (not label changes) and a plain re-run reuses the original event payload (stale labels). Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The leak-healer only refused to disconnect an endpoint whose owning pod was Running, so a Pending owner that already held a live IPAM allocation (status.podIP is populated once the sandbox is created, before the pod goes Running) could have its endpoint evicted mid-startup. Refuse for any non-terminal phase (!= Succeeded && != Failed); only a terminal owner is safely past needing its endpoint. Addresses non-blocking follow-up 4 on #2948. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
REST.Update wraps the HelmRelease write in retry.RetryOnConflict so a stale resourceVersion (the helm-controller racing the status subresource) is refreshed and retried instead of surfacing a 409. Add a unit test using a fake client whose interceptor returns one Conflict on the first Update and succeeds on the second, asserting Update returns nil and writes exactly twice. Addresses non-blocking follow-up 3 on #2948. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The build-main cache warmer is a privileged workflow (packages:write + an OCIR push token) and never pushes git, so set persist-credentials: false to keep the checkout's GITHUB_TOKEN out of .git/config where a later build step could read it. (SHA-pinning the actions is handled repo-wide in #2849.) Addresses non-blocking follow-up 7 on #2948. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
Aleksei Sviridkin (@lexfrei) thanks for the thorough review — every blocker and follow-up is now addressed or consciously deferred. Blocker
Follow-ups
Deferred (with reasoning)
|
The per-test failure snapshots (host in cozytest.sh, tenant in run-kubernetes.sh) ran `crust-gather collect` unbounded. On a contended/degraded cluster the collect can hang indefinitely — observed wedging the E2E install step for ~2h35m after a test failure until the job-level cancel, with no output after the capture line. `|| true` only catches a bad exit code, never a hang. Wrap both invocations in `timeout -k 30 300`: a 5-minute cap (hard SIGKILL 30s later if the collect ignores SIGTERM) bounds the hang; a partial snapshot still beats a multi-hour wedge, and timeout's exit 124 is already swallowed by `|| true`. Assisted-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
E2E re-run update + root cause of the storage failureRe-ran E2E on
The same SHA passes and fails on different subsystems each run, so the red E2E is environmental flakiness on the sandbox, not the change — none of these paths are touched by this PR. srv1 root cause (run 3)
Reads as a LINSTOR autoplace / tie-breaker reconcile wedge (orchestration-level), not a disk fault. The exact Two
|
|
Aleksei Sviridkin (@lexfrei) ready for another look when you have a moment 🙏
Could you re-review to clear the |
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
LGTM — the SeaweedFS customizations are now carried as patches wired into make update, so a re-vendor no longer drops them. That resolves my earlier blocker, and nothing else in the batch is a merge risk.
Business context: consolidated landing branch for 8 interdependent platform/CI stabilization fixes, reviewable commit-by-commit.
Resolved blocker
The -lock/-readonly BucketClasses and the s3 service-name override now live in patches/cosi-bucket-class-lock-readonly.patch and patches/s3-service-name.patch, both applied from the update target. A future re-vendor re-applies them instead of silently dropping hand-edits.
Production-surface review (no blockers)
- cozystack-api conflict retry (
pkg/registry/apps/application/rest.go):retry.RetryOnConflictre-reads the live HelmRelease, copies its resourceVersion and retries. Last-write-wins is correct here since the HelmRelease spec is fully derived from the Application. Covered byrest_conflict_test.go. volumeSizeLimitMB: 30000 -> 1000(seaweedfsvalues.yaml): not a capacity regression. The volume servers run-max 0(maxVolumes: 0), so SeaweedFS derives the volume count from disk size and total capacity is unchanged — only volume-file granularity changes. 1000 is the upstream chart default, and it also unblocks the default 10Gi PVC config, not just the E2E sandbox.- VPA
Auto -> Initial(etcd, monitoring, monitoring-agents): production-wide but justified.Autoevicts running pods (etcd quorum disruption plus install-time churn that trips the Cilium IP-reuse leak).Initialtogether with the raisedminAllowedfloors is the safer combination for these workloads. The tradeoff (no live resize of already-running pods) is acceptable and documented inline. - capi startupProbe: standard ~5-minute startup budget before liveness takes over. Fine.
Non-blocking follow-ups
- The
ci:type undersells the scope — the batch also changes the aggregated API server, etcd and monitoring behavior. The PR body is transparent about it, so this is only a labeling nit. - The e2e Cilium leak-healer Job (under
hack/) runs withhostNetworkpluspods/execand no explicit container securityContext. It is ephemeral test-cluster tooling and never ships underpackages/, but hardening it is cheap if you touch it again.
## What this PR does The `Build cache (main)` workflow (`build-main.yaml`, added in #2948) failed on its first run: https://github.com/cozystack/cozystack/actions/runs/27929277423 The `Set up Buildx` step ran without `DOCKER_CONFIG`, so `docker/setup-buildx-action` created the docker-container builder under the runner's default `~/.docker`. The build step runs with `DOCKER_CONFIG=$RUNNER_TEMP/.docker` (seeded earlier, holding the OCIR auth), so buildx couldn't find the builder and failed immediately: ``` ERROR: no builder "builder-6fb46242-…" found ``` The post-job `docker buildx rm` still succeeded — because the action's cleanup runs under the default config where the builder actually lived — which confirms the builder existed but in the wrong config store. This sets `DOCKER_CONFIG` on the `Set up Buildx` step so the builder instance is created in the same config the build step reads. Impact: only the main cache-warmer was broken. PR builds `--cache-from` this ref, so a missing cache is a cold build, not a failure — but the warm-cache speedup from #2948 wasn't being delivered until this fix. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Improved build pipeline stability by ensuring consistent Docker configuration across workflow steps. This enhances the reliability of the container build and deployment process. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
What
Consolidated landing branch for a set of interdependent platform/CI stabilization fixes. It began as the source PRs in the table below (in dependency order) and has since grown with review-driven fixes and a few production-behavior changes — the full current inventory is under Scope below. Landing them as one unit lets CI run against the real combined state instead of a fragile bottom-up merge train.
Source PRs (dependency order)
Why consolidated
These form a dependency DAG (verified from CI logs): e.g. 2943 needs 2946, 2938 needs 2874, 2928 needs 2938/2939, 2873 needs 2928. Tested bottom-up, each lower PR runs with none of the fixes above it and can't go green alone. This branch carries all of them, so CI runs against the real combined state.
Verification status
CI is green on the latest head (
5653c30): full-suite E2E passes end-to-end and is reproducible (2 of 3 runs on this SHA green). The one red run was a LINSTOR tie-breaker / DRBD-metadata infra flake on the sandbox, unrelated to the diff — a different environmental subsystem failed each run (details in the comments below). The earlier 3-hour crust-gather snapshot hang is fixed in5653c30.Scope beyond the original 8
The branch has grown past the 8 source PRs above with review-driven fixes and a few production-behavior changes. For reviewer transparency, the full set:
Review fixes (Aleksei Sviridkin (@lexfrei) review):
8f41910converts the SeaweedFS-lock/-readonlyBucketClasses and the s3 service-name override intopatches/, wired intomake update, so a re-vendor no longer drops them.cc99230grants the cilium leak-healerdeleteon pods (both delete remedies were RBAC-forbidden).014de3fadds a unit test for the HelmRelease update conflict-retry path.6deddc6makes the leak-healer refuse a disconnect for any non-terminal owner phase, not justRunning.f3914d2setspersist-credentials: falseon the build-cache checkout (SHA-pinning is handled repo-wide in ci: harden workflow token permissions and pin actions by SHA #2849).Production-behavior changes (not in the table above):
updateMode: Auto → Initialfor etcd (6e9ff90) and monitoring (0997105), plusvmselect/vmstorageminAllowedfloors — stops install-time eviction churn.retry.RetryOnConflicton the Application→HelmRelease update path (728f676).E2E robustness follow-ups: per-test crust-gather snapshots (
50e5f94), EtcdBackupSchedule wait (763b85e), tenant-node single-deadline wait (9e43d45), harbor BucketClaim 10m budget (d7f244f), tenant API via LoadBalancer (9beda4e), crust-gather pin (95f42aa).Not addressed (by decision): FU2 — the s3 Service keeps
seaweedfs.name-based naming via patch (rendered output unchanged; cozystack rendersfullname == seaweedfs, so it matches thecomponentName/fullnamesiblings in practice). FU6 — fork PR build push is handled by the existing mirror-to-same-repo-branch workflow.Relates to
#2946 #2944 #2943 #2874 #2938 #2939 #2928 #2873 — if this lands, those can be closed; otherwise they remain the granular per-PR review path.
Summary by CodeRabbit
New Features
Improvements
Tests