feat(kubernetes-nodes): split off worker node pool into its own app (Phase 2) - #2620

…or Talos CSR signer

Replace the upstream clastix/cluster-api-control-plane-provider-kamaji
v0.16.0 image with a Cozystack-built variant
(ghcr.io/cozystack/cozystack/cluster-api-control-plane-provider-kamaji:v0.16.0-talos-csr-signer.0)
that propagates KamajiControlPlane.spec.network.additionalServicePorts
into the resulting TenantControlPlane.spec.controlPlane.service.additionalPorts
field, allowing a clastix/talos-csr-signer sidecar to be exposed alongside
the kube-apiserver on the tenant API endpoint.

Patch source: aenix-io/cluster-api-control-plane-provider-kamaji@cozystack/v0.16.0-additional-service-ports;
upstream PR will be filed separately against the master branch (v1alpha2).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

…ontrolPlane

Lay down the infrastructure half of the Phase 1 Talos migration inside
the existing monolithic kubernetes chart, without yet switching the
worker bootstrap path. Specifically:

  - templates/talos/talos-pki.yaml: cert-manager Issuer + Certificate
    that issues a self-signed Ed25519 Talos CA (<release>-talos-ca)
    and a server TLS cert for the trustd listener
    (<release>-talos-tls-cert).
  - templates/talos/talos-secrets.yaml: helm-rendered Secret
    (<release>-talos-secrets) carrying random token, clusterId and
    clusterSecret values. Stability across upgrades is via lookup.
  - templates/cluster.yaml: KamajiControlPlane now declares a
    talos-csr-signer additionalContainer mounting the two secrets above,
    plus an additionalServicePort 50001/TCP propagated through the
    patched Kamaji control-plane provider.

What is still missing for end-to-end Talos workers (will follow in
subsequent commits): TalosConfigTemplate replacing KubeadmConfigTemplate
with a fully-rendered worker machineconfig, switching the
KubevirtMachineTemplate system disk from containerDisk to DataVolume
http source, and a Job that materialises a kubeadm bootstrap-token
Secret in the tenant apiserver.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Complete the Phase 1 chart rewrite that turns workers into
Talos-bootstrapped KubeVirt VMs:

- templates/talos/talos-secrets.yaml: extend the Helm-rendered Secret
  with bootstrap-token-id and bootstrap-token-secret so the same chart
  controls the kubeadm-format token used as cluster.token in the
  worker machineconfig.

- templates/talos/bootstrap-token-tenant-job.yaml: post-install /
  post-upgrade Helm hook Job that materialises the matching
  bootstrap-token-<id> Secret inside the tenant kube-system namespace,
  using <release>-admin-kubeconfig from Kamaji.

- templates/cluster.yaml:
    * Replace KubeadmConfigTemplate (~85 lines of cloud-init kubeadm
      flow) with TalosConfigTemplate using generateType=none and a
      Helm-rendered Talos worker machineconfig as spec.template.spec.data.
      Values are looked up at render time from <release>-talos-secrets,
      <release>-talos-ca (cert-manager) and <release>-ca (Kamaji).
    * Add a disk-system DataVolume that imports
      factory.talos.dev/image/<schematic>/v1.13.0/nocloud-amd64.raw.xz
      via CDI; system disk volume now references this DataVolume
      instead of a containerDisk pulled from a Cozystack-built image.
    * Switch MachineDeployment.bootstrap.configRef from
      KubeadmConfigTemplate (bootstrap.cluster.x-k8s.io/v1beta1) to
      TalosConfigTemplate (bootstrap.cluster.x-k8s.io/v1alpha3).
    * Gate TalosConfigTemplate rendering on all three Secret
      dependencies being ready, to avoid a chicken-and-egg deadlock on
      first install (cert-manager Certificate and Kamaji ca are
      produced asynchronously by other components of this same chart).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

bitnami/kubectl:1.33 tag does not exist on Docker Hub; switch to
docker.io/alpine/k8s:1.33.4 which is already the conventional kubectl
image used elsewhere in cozystack (see core/installer).

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

CDI rejects the import when the target PVC is smaller than the
decompressed nocloud-amd64.raw image plus reserved sector overhead;
4Gi was just enough for the image but CDI's headroom check kept
failing at ~99% with 'DataVolume too small to contain image'. 8Gi
clears the check with margin for future image growth.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

…check

Two related fixes for Talos workers booting on KubeVirt:

1. KubeVirt CAPK injects bootstrap data through a cloudInitConfigDrive
   volume (OpenStack metadata service shape, user_data field). The
   nocloud-amd64 Talos image silently ignores ConfigDrive and never
   applies the machineconfig. Switch the system DataVolume http source
   to openstack-amd64.raw.xz so Talos's openstack platform driver
   reads the cloud-init data on first boot.

2. KubevirtMachineTemplate's default virtualMachineBootstrapCheck is
   'ssh', which has no chance of succeeding against Talos (no SSH
   daemon by design). Set checkStrategy=none so CAPK stops blocking
   Machine.Ready on a probe that will never connect.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

The system disk volume on 4Ki-native storage was exposed to QEMU
without a blockSize hint, so the guest saw 512-byte sectors backed
by a block device that requires 4Ki-aligned O_DIRECT writes. Talos
partition setup then triggered libvirt I/O errors and KubeVirt
paused the VM (error_policy=stop), preventing the worker from
joining the tenant cluster.

Set blockSize.matchVolume on the system disk to align with the
existing kubelet disk and let the guest see the real physical
sector size.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Talos creates its own EPHEMERAL partition on the install disk and
mounts it at /var (which already contains /var/lib/kubelet), so a
dedicated second virtio disk is redundant. Use a single system disk
sized from nodeGroup.diskSize and let Talos lay out its partitions
on top.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

SeaBIOS rejects virtio-blk devices that report 4096-byte logical
sector size ("block size 4096 is unsupported"), which is what the
underlying 4Ki-native block storage exposes once blockSize.matchVolume
is enabled. Switch the VM firmware to EFI; the Talos openstack image
already ships an EFI partition, and OVMF handles 4Ki sectors natively.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Reverting to SeaBIOS bootflow on the 4Ki-native backing storage:
EFI/OVMF could not locate the Talos openstack image bootloader on
virtio-blk, and SeaBIOS refuses to boot a virtio-blk device that
advertises 4096-byte logical sectors. Set blockSize.custom with
logical=512/physical=4096 so the guest sees classic 512-byte logical
sectors (SeaBIOS happy) while the qemu side preserves 4Ki physical
alignment for the underlying DRBD device.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Talos workers resolve cluster.controlPlane.endpoint via cloud-init
DNS (the host cluster CoreDNS), not the tenant cluster CoreDNS that
does not exist yet. The host CoreDNS serves the host cluster's
domain (e.g. cozy.local on Cozystack), so the short
"<release>.<ns>.svc" form gets NXDOMAIN. Expand the endpoint into a
fully qualified <release>.<ns>.svc.<host-cluster-domain> form using
the cluster-domain value injected from cozystack-values.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Without explicit cluster.network.podSubnets/serviceSubnets in the
worker machineconfig, Talos assumes the upstream defaults
(10.244.0.0/16 / 10.96.0.0/12) and refuses to start kubelet because
the VM NIC address ends up inside its assumed pod CIDR
(address-overlap diagnostic). Pass the tenant CIDRs that the CAPI
Cluster already advertises so Talos sees a consistent layout.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

…oint

The Talos worker bootstraps before any cluster DNS exists, so a
DNS-based controlPlane.endpoint never resolves: cloud-init points
DNS at the host cluster CoreDNS, which neither serves the tenant
cluster's cluster.local zone nor the host's <svc>.<ns>.svc.cozy.local
form for tenant services in our setup. Look up the Kamaji-managed
apiserver Service via helm lookup and embed its ClusterIP directly,
which routes through host kube-proxy/Cilium to the Kamaji pod.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

The cilium DaemonSet runs with hostNetwork=true and inherits the
host node's resolv.conf, which on Talos workers points at the host
cluster CoreDNS. That resolver does not serve the tenant cluster
cluster.local zone, so the previous short form
"<release>.<ns>.svc" returned NXDOMAIN and cilium init container
"config" crashlooped. Look up the Kamaji apiserver service and embed
its ClusterIP directly so cilium contacts the apiserver without any
DNS.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

…kers

With rotate-server-certificates=true, the kubelet refuses to serve
TLS until its kubernetes.io/kubelet-serving CSR is approved. The
cluster has no kubelet-serving approver (talos-csr-signer is for
Talos Machine PKI on port 50001, not Kubernetes CSRs), so the CSRs
pile up Pending forever and any apiserver/konnectivity hit on
kubelet:10250 fails with "tls: internal error" (no Logs / Exec /
Metrics from tenant nodes). Match the prior Ubuntu+kubeadm worker
behaviour and let kubelet use a self-signed serving cert; the Kamaji
apiserver already skips kubelet TLS verification.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Talos workers reach the Kamaji-managed apiserver by ClusterIP (see
the worker controlPlane.endpoint), and Talos's internal
secrets.APIController submits its apid CSR over that same endpoint.
The default Kamaji apiserver certificate only carries the public
ingress hostname and the standard kubernetes.default.svc SANs, so
the IP-based dial fails with "x509: cannot validate certificate for
<ClusterIP> because it doesn't contain any IP SANs" and apid never
comes up. Add the Service short FQDN, the cluster.local FQDN, and
the looked-up ClusterIP to KamajiControlPlane.spec.network.certSANs
so the apiserver cert covers every form workers may dial.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Talos workers reach trustd (the talos-csr-signer sidecar on port
50001) through the Kamaji apiserver Service ClusterIP, but the
sidecar's TLS cert only carried DNS SANs. The Talos
secrets.APIController then failed its apid CSR with "cannot
validate certificate for <ClusterIP>: doesn't contain any IP SANs",
so apid never started on workers. Look up the Service ClusterIP and
add it as an ipAddress on the trustd Certificate.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Talos workers run an internal k8s.EndpointController that watches
the Endpoint and EndpointSlice named "kubernetes" in the default
namespace to discover apiserver endpoints (failover for the
controlPlane.endpoint pin). The default system:node ClusterRole
denies this, so every worker fills its console with
"endpointslices ... forbidden" warnings. Reuse the post-install
hook Job to apply a minimal ClusterRole/Binding granting
system:nodes get/list/watch on those two named resources only.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

The previous ClusterRole used resourceNames=["kubernetes"] which
Kubernetes only honours for list/watch when the client supplies a
matching metadata.name field selector. Talos's informer issues an
unbounded list/watch, so the rule never matched and the forbidden
warnings persisted. Replace with a namespace-scoped Role/RoleBinding
in default (the only namespace where the "kubernetes" endpoint
lives) and drop the previously-applied cluster-scoped objects.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

The kubeadm-era assertions referenced KubevirtMachineTemplate at
documentIndex 5 (after KubeadmConfigTemplate) and probed Ubuntu-only
fields: disk-kubelet name, source.blank, and the /persistent +
update-k8s.sh wiring inside KubeadmConfigTemplate. After the Phase 1
Talos migration, KubeadmConfigTemplate is gone, the worker carries a
single disk-system DataVolume sourced from the Talos factory image,
and the system disk uses blockSize.custom=512/4096. Update the
KvMT-focused assertions to match (new documentIndex, disk-system
metadata, factory URL, custom blockSize) and drop the kubelet
reservation test along with the persistent-disk and update-k8s.sh
sections; their direct replacement lives inside the
TalosConfigTemplate, which is lookup-gated and therefore not visible
in the unit-test render.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

Drop in the empty packages/apps/kubernetes-nodes/ skeleton (Chart.yaml,
Makefile, README, charts/cozy-lib link, values.yaml + schema, a
_helpers.tpl that resolves the parent CAPI Cluster CR for linkage and
ownerReferences). Templates and the API types come in follow-up
commits — this commit only lays out the directory so the rest of the
phase 2 split can land incrementally.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>