feat(cozyreport): collect Flux/cert-manager/host context + summary.txt - #2553

Actionable comments posted: 4

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@hack/cozyreport-summary.sh`:
- Around line 31-34: The "## OOMKilled in last 30 min" heading is misleading
because the kubectl command (kubectl get events -A --field-selector
reason=OOMKilling --sort-by=.lastTimestamp) only limits count, not time; either
change the heading to "Recent OOMKilled events" or actually filter events to the
last 30 minutes: run kubectl get events -A -o json and use a JSON processor (jq
or python) to compare each event.metadata.creationTimestamp/.lastTimestamp
against (now - 30m), then sort by .lastTimestamp and print the tail as before;
update the echo line (echo "## OOMKilled in last 30 min") and replace the
kubectl pipeline accordingly so the output matches the label.
- Around line 53-56: The loop that prints non-ready Flux resources (for kind in
helmrepositories.source.toolkit.fluxcd.io
ocirepositories.source.toolkit.fluxcd.io
gitrepositories.source.toolkit.fluxcd.io) omits
externalartifacts.source.toolkit.fluxcd.io, so ExternalArtifact failures never
appear in summary.txt; update that loop to include
externalartifacts.source.toolkit.fluxcd.io (keeping the existing k extraction
${kind%%.*}) so ExternalArtifact resources are queried and printed the same way
as the other Flux kinds.

In `@hack/cozyreport.sh`:
- Around line 253-254: The call to hack/cozyreport-summary.sh fails unless the
CWD is the repo root; compute the script directory near the top (e.g., set
SCRIPT_DIR using: SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)) and
replace the invocation echo "Generating summary..." hack/cozyreport-summary.sh >
$REPORT_DIR/summary.txt 2>&1 || true with a resolved path using
"$SCRIPT_DIR/cozyreport-summary.sh" so the summary runs regardless of caller
CWD; keep REPORT_DIR usage the same.
- Around line 243-248: The script currently hardcodes the Talos node IPs
(192.168.123.11-13) so talosctl calls only capture those machines; replace the
fixed list in the for loop with a dynamic node enumeration using talosctl (e.g.,
capture the node list from a talosctl nodes/list command using --talosconfig
/workspace/talosconfig and iterate over that output), and add a safe fallback to
the original static list if the talosctl query fails; update the loop that calls
talosctl (the lines invoking talosctl --talosconfig /workspace/talosconfig -n
$node dmesg/logs) so it uses the dynamically discovered $node values and
preserves the existing DIR/talos-$node-*.txt naming and the "|| true" behavior.

Code Review

This pull request enhances the diagnostic capabilities of the cozyreport utility by adding modules for Flux controller state, cert-manager, Kubernetes events, Helm release secrets, and Cozystack applications, alongside a new sandbox-host diagnostic module. I have provided feedback regarding potential logic errors in the Helm and Flux resource readiness checks, the need for robust CRD existence verification to prevent script termination, and a recommendation to dynamically discover node IPs for the sandbox-host module rather than using hardcoded values.

LGTM. Addresses the diagnostic-coverage gap surfaced from #2500 with a clean, additive change to two shell scripts. All bot findings from the first review pass are folded into b40ab29:

• "OOMKilled in last 30 min" heading relabeled to match the actual --field-selector reason=OOMKilling semantics (no time bound) — pragmatic call to avoid pulling jq/python into an otherwise pure-sh script.
• externalartifacts.source.toolkit.fluxcd.io now in the summary loop, with the loop switched from column-based awk to JSONPath so layout differences between source CRDs (HelmRepository's columns differ from OCIRepository) don't silently misclassify resources. Per-kind kubectl get crd … || continue guard keeps the script working on clusters without Flux v2.
• Dynamic node IP discovery from kubectl get nodes with a fallback to the E2E sandbox IPs — the sandbox-host module now works on arbitrary clusters.
• cozyreport-summary.sh invoked via $SCRIPT_DIR (resolved through CDPATH= + dirname $0) so cozyreport.sh works from any CWD, not just repo root.
• kubectl get hr -A | awk '$4 != "True"' is correct against helm-controller v1.4.3 (api/v2/helmrelease_types.go exports Age/Ready/Status printcolumns; with -A columns are NAMESPACE NAME AGE READY STATUS, so $4 is Ready). The Gemini comment claiming $3 is wrong — keep what's there.
• base64 -d | base64 -d | gzip -d for Helm storage Secrets is correct: helm itself base64-encodes the gzipped JSON, then Kubernetes wraps the resulting string in Secret data which is base64-encoded again, so two decodes are required.

E2E Tests is currently red on this PR, but the failure is in Install Cozystack → Wait for Cluster-API provider deployments on all three retry attempts — well before collect-report runs, and unrelated to anything in this diff (the new code only executes at the end of an install, on report collection). Looks like a CAPI bootstrap flake on the oracle runners. A clean re-run should pass; please push an empty commit to trigger one before merge so the gate is actually green rather than waved through.

Non-blocking nits worth knowing about, none required for merge:

• Variable quoting in both scripts is sparse — $NAMESPACE, $NAME, $REPORT_DIR, etc. unquoted across most loops. Kubernetes resource names can't contain spaces (they're DNS-label-validated) so this won't bite in practice, but set -euo pipefail + consistent quoting would harden the scripts against future changes that pipe in less-trusted strings (e.g. message bodies from events).
• cozyreport.sh doesn't set -e; cozyreport-summary.sh does. Inconsistent — first is best-effort by design ("collect what you can, log errors into the file"), second is fail-fast wrapped in || true at the call site. Worth a one-line comment at the top of cozyreport.sh explaining the choice so the next maintainer doesn't "fix" it.
• Pipelines like kubectl get … | awk … swallow kubectl exit codes (no pipefail). For a diagnostic script, recoverable failure is fine, but it's worth a one-line set -o pipefail at the top of cozyreport-summary.sh (already set -eu) so failed kubectl calls surface as missing sections rather than silently empty ones.
• The kubectl get events --field-selector reason=OOMKilling filter catches kernel-OOM events emitted by kubelet, but not container exit-137 cases that don't escalate to a node-level OOMKilling event (e.g. cgroup-only kills with no kernel pressure). For a future iteration, kubectl get pod -A -o jsonpath='{range .items[*]}{range .status.containerStatuses[?(@.lastState.terminated.reason=="OOMKilled")]}{…}{end}{end}' catches the container-state path. Out of scope here.