feat/improvement-kubernetes-tests - #1508
Conversation
The Cozystack Kubernetes tests are improved to better verify cluster state on worker nodes. This patch adds checks for the Kubernetes version on each worker node and validates the installation of necessary releases in tenant Kubernetes clusters, such as CoreDNS, Cilium, and others. These improvements ensure that tenant clusters are correctly set up before running workloads and that all required components are present. ```release-note [tests] Improve Kubernetes tests: check worker node versions and validate installation of required releases (CoreDNS, Cilium, etc.) Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
WalkthroughUpdates the Kubernetes app Makefile to set KUBERNETES_VERSION to v1.33. Enhances hack/e2e-apps/run-kubernetes.sh with explicit variable handling, expanded readiness waits, kubelet version validation, background port-forwarding with API/version checks, node-count gating, conditional cleanup, and a temporary v1.32 compatibility allowance. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant Dev as Script (run-kubernetes.sh)
participant K8s as mgmt-cluster (kubectl)
participant TCC as TenantControlPlane
participant PF as Port-forward
participant API as Tenant API Server
Dev->>K8s: kubectl apply manifests
Dev->>K8s: kubectl wait (namespace, Kamaji CP, TCP readiness)
Dev->>K8s: kubectl wait (deployments, machine deployment)
par Background
Dev->>PF: Start port-forward (bg)
and Readiness
Dev->>K8s: kubectl wait components (cilium, coredns, csi, ingress-nginx, vsnap-crd)
end
Dev->>API: kube version check via new kubeconfig
Dev->>API: Get nodes, verify count==2
alt Kubelet versions match expected
Dev->>Dev: proceed
else v1.32 compatibility allowance
Note right of Dev: Temporary leniency (TODO)
else mismatch
Dev-->>Dev: exit with error
end
Dev->>K8s: Conditional cleanup then delete (on teardown)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello IvanHunters, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly improves the Cozystack Kubernetes end-to-end tests by adding comprehensive verification steps for tenant cluster state. It ensures that worker nodes are running the correct Kubernetes version and that all necessary core components are properly installed and ready, making the tests more reliable and robust. Additionally, it includes an update to the default Kubernetes version used in the project. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request improves the Kubernetes e2e tests by adding checks for worker node versions and ensuring required components like Cilium and CoreDNS are ready. The changes in run-kubernetes.sh are substantial and add valuable assertions to the test suite. I've provided a couple of suggestions to improve the robustness and readability of the new shell script logic. The update to the Makefile to bump the default Kubernetes version is consistent with the testing improvements.
| ;; | ||
| esac | ||
|
|
||
| for v in $versions; do |
There was a problem hiding this comment.
The for v in $versions construct is subject to word splitting based on the value of $IFS. If a version string ever contains whitespace, this loop will process its parts as separate items. While kubeletVersion is not expected to have spaces, using a more robust method to iterate over the list of versions is a good practice. Consider reading the versions into an array.
For example:
read -r -a versions_array <<< "$versions"
for v in "${versions_array[@]}"; do
# ... loop body
done| case "$k8s_version" in | ||
| v1.32 | v1.32.* | v1.32-*) | ||
| case "$v" in | ||
| v1.32 | v1.32.* | v1.32-* | v1.33 | v1.33.* | v1.33-*) ;; | ||
| *) node_ok=false; break ;; | ||
| esac | ||
| ;; | ||
| *) | ||
| case "$v" in | ||
| "$k8s_version" | "$k8s_version".* | "$k8s_version"-*) ;; | ||
| *) node_ok=false; break ;; | ||
| esac | ||
| ;; | ||
| esac |
There was a problem hiding this comment.
This nested case statement for version checking is a bit complex and can be simplified. The patterns for matching versions can be made more concise using globbing (*). This will improve readability and maintainability.
| case "$k8s_version" in | |
| v1.32 | v1.32.* | v1.32-*) | |
| case "$v" in | |
| v1.32 | v1.32.* | v1.32-* | v1.33 | v1.33.* | v1.33-*) ;; | |
| *) node_ok=false; break ;; | |
| esac | |
| ;; | |
| *) | |
| case "$v" in | |
| "$k8s_version" | "$k8s_version".* | "$k8s_version"-*) ;; | |
| *) node_ok=false; break ;; | |
| esac | |
| ;; | |
| esac | |
| case "$k8s_version" in | |
| v1.32*) | |
| case "$v" in | |
| v1.32* | v1.33*) ;; | |
| *) node_ok=false; break ;; | |
| esac | |
| ;; | |
| *) | |
| case "$v" in | |
| "$k8s_version" | "$k8s_version".* | "$k8s_version"-*) ;; | |
| *) node_ok=false; break ;; | |
| esac | |
| ;; | |
| esac |
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
hack/e2e-apps/run-kubernetes.sh(2 hunks)packages/apps/kubernetes/Makefile(1 hunks)
🧰 Additional context used
🪛 checkmake (0.2.2)
packages/apps/kubernetes/Makefile
[warning] 1-1: Missing required phony target "all"
(minphony)
[warning] 1-1: Missing required phony target "clean"
(minphony)
[warning] 1-1: Missing required phony target "test"
(minphony)
| # Update the kubeconfig to use localhost for the API server | ||
| yq -i ".clusters[0].cluster.server = \"https://localhost:${port}\"" tenantkubeconfig | ||
| bash -c 'timeout 200s kubectl port-forward service/kubernetes-'"${test_name}"' -n tenant-test '"${port}"':6443 > /dev/null 2>&1 &' | ||
| timeout 20 sh -ec 'until kubectl --kubeconfig tenantkubeconfig version 2>/dev/null | grep -Fq "Server Version: ${k8s_version}"; do sleep 5; done' |
There was a problem hiding this comment.
Remember subject line to highlight.
Let's produce.
Need comment text: subject e.g. "Restore actual server version check" etc.
We must be concise, direct.
Let's craft text:
"Restore the actual version guard."
Explain: Because script single-quoted, ${k8s_version} not expanded, so grep string degrade to "Server Version: ". Means mismatch not caught. Provide diff.
Add ``.
Add diff using diff block.
Ok.
After comment, update scratchpad to mark file done.
Let's craft comment.
Need to ensure adhesives referencing Line 78. Provide condemnation.
Let's produce.
Then final file_end with scratchpad concluding.
Finally finalize entire review message summarizing? Wait instructions: For each file, we must produce start and end markers, even if no comments. Already for first file we have markers without comments. For second file we will have comment.
Also final output should only include review comments? The instructions not ask summary. Should we provide final message with overall? They say "Generate review comments on each changed code segment... For each file, wrap..." So final output likely just these structured comments. No final summary required? But typical RBC? We'll follow instructions: after finishing last file_end, no extra text.
Make sure scratchpad update after comment: mark done.
Let's produce comment. Need to include scratchpad update before comment? Should update scratchpad before comment? We'll include comment around instructions: At start of file we already inserted scratchpad with tasks. After comment, update [done]. maybe new scratchpad comment? In general they expect scratchpad to be updated with statuses. After comment we can update to mark done.
Ok.
Add comment.
Then finish.
Let's craft.
First update scratchpad to note issue. Need to insert new scratchpad comment before comment? maybe to show we identified issue? instructions not strict but should maintain.
Add new scratchpad comment with note.
Then comment.
Then final scratchpad to mark done.
Finally file_end.
After everything, final message should end.
Let's craft step-by-step.
- Insert scratchpad update. Use hidden comment.
Maybe:
<!-- [scratchpad]
[analysis]
- `k8s_version` not expanded in grep -> check degraded.
[actions]
Add review comment lines 78.
-->
Then comment.
After comment, final scratchpad to mark file done.
Finally <!-- file_end -->.
Ok.
Let's produce final output accordingly.
🤖 Prompt for AI Agents
In hack/e2e-apps/run-kubernetes.sh around line 78, the Server Version check uses
single quotes so ${k8s_version} is not expanded and grep matches an empty
string; change the shell quoting so the variable is expanded (e.g., use double
quotes for the sh -ec string or build the command without single quotes) and
ensure the final grep pattern includes the expanded ${k8s_version} so the
timeout loop actually verifies the Kubernetes server version.
The Cozystack Kubernetes tests are improved to better verify cluster state on worker nodes. This patch adds checks for the Kubernetes version on each worker node and validates the installation of necessary releases in tenant Kubernetes clusters, such as CoreDNS, Cilium, and others. These improvements ensure that tenant clusters are correctly set up before running workloads and that all required components are present.
Summary by CodeRabbit
New Features
Chores