DO NOT MERGE — instrument tenant Talos bootstrap timing - #3144
DO NOT MERGE — instrument tenant Talos bootstrap timing#3144Aleksei Sviridkin (lexfrei) wants to merge 1 commit into
Conversation
Throwaway diagnostic to capture why the tenant worker MachineDeployment md0 sometimes fails its status.replicas=2 wait: is the TalosConfigTemplate late because an input is slow, the reconcile Job pod is slow/stuck, the Job is recreated (churn), or the template only lands after the wait times out. Adds a background poller (hack/e2e-apps/talos-debug-poller.bash) that, from just before the install through the bringup, every ~15s appends a timestamped [TALOS-DEBUG] dump against the management cluster: the parent HelmRelease conditions/remediation counters/history, the talos-reconcile Job(s) creationTimestamp+uid+status+backoffLimit, its pod(s) phase/waiting-reason/restarts, the TalosConfigTemplate first-appearance, the four runtime-wait inputs, and the MachineDeployment/MachineSet status and events. Wired in from kubernetes-latest.bats and kubernetes-previous.bats (not run-kubernetes.sh) so Test-Impact-Analysis runs only the two kubernetes tests instead of the full suite. The md0 wait budget is unchanged so the failure window is observable; a post-failure tail captures whether the template lands just after. Revert once the timeline is captured. Nothing here ships. Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Diagnostic complete — it served its purpose. The instrumented run pinned the root cause: an ordering deadlock where the post-install hook that creates the TalosConfigTemplate never runs because the install stalls in its main-resource wait (deadlocked on the cilium sub-release, which needs worker nodes that need the template). The structural fix is #3145. Closing this throwaway diagnostic. |
Throwaway diagnostic. Not for merge, not gated — close after the data is captured.
Goal: classify why the tenant worker
MachineDeploymentmd0intermittently fails itsstatus.replicas=2wait. md0 is gated on theTalosConfigTemplateexisting; this run captures whether the template is late because an input is slow, the talos-reconcile Job pod is slow/stuck (Pending / ImagePull / CrashLoop), the Job is recreated (churn), or the template only lands after the wait times out.What it does: a background poller (
hack/e2e-apps/talos-debug-poller.bash) runs from just before the install through the bringup and, every ~15s, appends a timestamped[TALOS-DEBUG ...]block (against the management cluster) covering:kubernetes-<name>HelmRelease: Ready/reason, attempted revision, install/upgrade failure counters, and per-revision history statuses (the remediation/churn axis);TalosConfigTemplate <rel>-md0first-appearance timestamp;MachineDeployment/MachineSetstatus and events (the "cannot create a new MachineSet when templates do not exist" line lands here).Scope: wired in from
kubernetes-latest.batsandkubernetes-previous.bats(notrun-kubernetes.sh) so Test-Impact-Analysis runs only the two kubernetes tests, not the full suite. The md0 wait budget is unchanged so the failure window stays observable; a post-failure tail captures whether the template lands just after.Reading the output: grep the e2e job log (or the
cozyreportartifact) for[TALOS-DEBUG]. Both kubernetes-latest and kubernetes-previous run, giving two samples; md0 is bimodal (sometimes ~46s, sometimes >10m), so a fast run is the happy-path baseline and a re-run may be needed to catch a slow one.Revert (the two
.batshooks + the.bashhelper) once the timeline is captured.