feat(networking): cozyplane eBPF CNI + cozyplane-kpr networking variant (drops Cilium) - #3149
Draft
Timofei Larkin (lllamnyp) wants to merge 33 commits into
Draft
feat(networking): cozyplane eBPF CNI + cozyplane-kpr networking variant (drops Cilium)#3149Timofei Larkin (lllamnyp) wants to merge 33 commits into
Timofei Larkin (lllamnyp) wants to merge 33 commits into
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Timofei Larkin (lllamnyp)
force-pushed
the
feat/cozyplane
branch
2 times, most recently
from
July 2, 2026 15:14
f37bd80 to
1f18873
Compare
Timofei Larkin (lllamnyp)
added a commit
that referenced
this pull request
Jul 6, 2026
…ate (#3184) ## What this PR does Closes two coverage gaps found while shadow-observing the API-owner-review gate (#3167) against real PRs in the first hours after merge: 1. **CRD discovery was gated on directory name** (`crds/`, `definitions/`, `manifests/`), so a CRD shipped inside a chart's `templates/` dir — a valid, existing convention when the CRD is wrapped in a `{{- if }}` guard — was invisible to the gate. This was not hypothetical: `#3149` (cozyplane) adds four brand-new CRDs (`VPC`, `VPCBinding`, `VPCPeering`, `Port`) exactly this way, and none of them tripped a review requirement. 2. **Groups served only by an external aggregated apiserver had no representation at all.** The same `#3149` also registers `sdn.cozystack.io` via a `kind: APIService` pointing at a prebuilt image built outside this repository — no CRD, no vendored Go types, nothing to diff. ### Fix - CRD/APIService discovery is now content-based: every `.yaml`/`.yml` file under `packages/` and `internal/` is a candidate, and whether it's kept depends on whether it parses into the right `kind` with a first-party group — not which directory it sits in. - Both parsers now tolerate a Helm directive line (`{{- if }}`, `{{- end }}`, a comment block) sharing a `---`-delimited document with otherwise-valid YAML, rather than dropping the whole document. This matters because the closing `{{- end }}` of a wrapping conditional has no `---` before it, so it lands in the same document as the manifest it closes. - Added a fourth resource source, `SourceAPIService`, for `kind: APIService` manifests registering a first-party group. Like the existing static Go-backed groups, these carry no checked-in schema, so they participate only in new-group / new-resource / removal detection — never a fabricated breaking-change diff against a schema this repo can't see. Verified against the real `feat/cozyplane` branch: the gate now reports all four new CRDs, plus the removal of the old `securitygroups`/`APIService` registration for the same group — a full API-surface migration that was previously invisible end to end. ### Release note ```release-note fix(ci): the API-owner-review gate now discovers CRDs and APIService registrations by content instead of directory name, closing a gap where a CRD or an externally-served aggregated API group could ship without triggering the required review. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Discovery now includes APIService registrations, enabling detection of groups/versions exposed only via aggregated APIService manifests (including merging versions from multiple templates). * **Bug Fixes** * Improved YAML discovery to tolerate mixed/templated multi-document streams: non-CRD/non-APIService documents no longer stop discovery, and embedded template directives are handled more reliably. * File processing is more resilient during snapshot loading (includes a safety cap on YAML size). * **Tests** * Added coverage for APIService discovery, templated neighbor documents, and correct exclusion of third-party manifests. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…ariant A "cozyplane" networking variant that replaces BOTH kube-ovn and Cilium (and kube-proxy) with cozyplane's eBPF CNI plus cozyplane-kpr for in-cluster Services (socket-LB, importing Cilium's LB control plane — no Cilium agent). - packages/system/cozyplane: the CNI/agent/apiserver/controller (image bumped to the build with the off-VPC-transit fix; the old 0.3.17 pin had that regression, which broke grouped-pod TCP egress via the gateway). - packages/system/cozyplane-kpr: NEW — the kube-proxy replacement DaemonSet (hostNetwork, privileged, host bpffs+cgroup2 mounts, an init container that mounts bpffs; on Talos it reaches the apiserver via KubePrism https://localhost:7445 since the kubernetes.default ClusterIP is unserved until kpr runs). Scoped RBAC (services/endpointslices/nodes read + cilium.io). - networking.yaml: the "cozyplane" variant = cozyplane + cozyplane-kpr, no Cilium; drop the old cozyplane-cilium variant and system/cilium/values-cozyplane.yaml. - platform system bundle + values: accept networkingVariant "cozyplane". Single IPAM authority (cozyplane) over the podCIDR — removes the Cilium standalone-IPAM coexistence hazard (lllamnyp/cozyplane#12). Known gaps carried as follow-ups: host-firewall + default-network NetworkPolicy (were Cilium's), and external NodePort + VM-guest ClusterIP (cozyplane-kpr increment 3). Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Timofei Larkin (lllamnyp)
force-pushed
the
feat/cozyplane
branch
from
July 7, 2026 16:05
85c6dbc to
7f057be
Compare
…ory-backed) main's etcd operator moved from etcd.aenix.io/v1alpha1 to etcd-operator.cozystack.io/v1alpha2 (group+version), and the spec reshaped. Migrate cozyplane-etcd: - v1alpha2 EtcdCluster, storage.medium=Memory (no PVC/storageClass) so the aggregated API's etcd bootstraps WITHOUT a storage layer — the CNI installs first, breaking the CNI->apiserver->etcd->storage->CNI cycle. Replicated raft survives single-pod restarts; not durable across a full-cluster restart (a persistent mode is a follow-up once storage is sequenced after the CNI). - TLS via the operator's cert-manager integration: it emits the server/operator-client/peer Certificates itself given issuers. Drop the hand-rolled server+peer certs; keep the CA issuers and the apiserver's own client cert (signed by the same client CA so the etcd server trusts it). - apiserver dials the operator's client Service (cozyplane-etcd) — v1alpha2 members carry operator-assigned names, so per-member headless DNS is gone. helm lint + template clean with operator etcd enabled (the Talos overlay). Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…ager)
cozyplane installs first as the CNI, before cert-manager, but its chart renders
cert-manager Certificates for the apiserver+etcd — the atomic Helm release fails
("no matches for kind Certificate") and the CNI never comes up. Disable the
apiserver in the networking slot so the CNI bootstraps cert-manager-free.
Follow-up: split the chart (CNI component vs apiserver+etcd component that
dependsOn cert-manager) to bring VPC tenancy back.
Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
… Talos)
With cozyplane as CNI and no kube-proxy, the kubernetes.default ClusterIP is
unserved until cozyplane-kpr (which needs the agent) is up — so the hostNetwork
agent/responder crashloop on 'dial 10.96.0.1:443: i/o timeout' at bootstrap. Add
a kubeApiServer.{host,port} value (empty = ClusterIP) and set it to Talos
KubePrism localhost:7445 in values-talos, wired as KUBERNETES_SERVICE_HOST/PORT.
Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
….cozystack.io to cozyplane
The cozyplane networking variant runs no Cilium and no kube-ovn, but the shared
common-packages helper still instantiated four data-plane packages that assume
them, and cozystack-api still aggregated the sdn.cozystack.io group — which
cozyplane owns as CRDs. Concretely on a fresh cozyplane cluster:
- multus CrashLoopBackOff: delegates to the Cilium primary conflist
05-cilium.conflist, which cozyplane never writes (it writes 00-cozyplane).
- securitygroup-controller CrashLoopBackOff: projects sdn.cozystack.io
SecurityGroups onto CiliumNetworkPolicy (cilium.io/v2 CRD absent), AND is a
second, incompatible implementation of securitygroups.sdn.cozystack.io.
- kubeovn-webhook/-plunger: orphaned kube-ovn control plane (the webhook would
fail closed once cert-manager lands and block resource creation).
- cozystack-api aggregates v1alpha1.sdn.cozystack.io -> its explicit APIService
hijacks the group from cozyplane's CRDs the moment it deploys, 404ing every
non-SecurityGroup kind (VPC, Port, FloatingIP, ...).
Gate the four packages in common-packages on networkingVariant != cozyplane, and
add cozystackAPI.serveSDN (default true) so the platform bundle turns off the sdn
APIService for the cozyplane path. kubeovn-cilium is unchanged: all four packages
instantiate and cozystack-api keeps serving sdn.
Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…g multus dep) virtualprivatecloud-application (kubevirt) dependsOn cozystack.multus and is the kube-ovn tenant VPC app; the cozyplane variant installs neither multus nor kube-ovn and offers VPC tenancy via sdn.cozystack.io, so its Package hung forever on a multus HR that never gets created. Gate the instantiation on networkingVariant != cozyplane. Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Image with node_remotes (pod->node overlay encap) + CFG_MASQ_IP (masquerade from the egress address), fixing cross-node webhooks and pod internet egress on a spoof-guarding underlay. Adds nodes 'patch' RBAC for the node-address annotation. Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…ts install linstor, tenant, monitoring-agents, fluxcd-operator (and more) embed CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy objects and their Helm installs fail with 'no matches for kind CiliumClusterwideNetworkPolicy' when Cilium is absent. Ship the two cilium.io/v2 CRDs (from cilium 1.19.5) as a CRD-only package, gated to the cozyplane variant (the Cilium variants get them from the cilium chart). Inert — cozyplane enforces no NetworkPolicy yet; the CRDs exist only so the manifests apply. Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
hostNetwork clients on another node can now reach a VPC pod's fabric IP (the bridge reply is encapsulated instead of leaving pod-sourced; OCI dropped it). Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…ice DNAT) Assisted-By: Claude Opus 4.8 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Assisted-By: Claude Fable 5 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…endsOn cert-manager) The CNI slot stays cert-manager-free: packages/system/cozyplane now serves sdn.cozystack.io as bootstrap CRDs only (apiserver.enabled dropped for crds.enabled, apiserver/etcd templates removed; agent.yaml/crds.yaml re-vendored). The new packages/system/cozyplane-apiserver carries the aggregated apiserver + the memory-backed v1alpha2 EtcdCluster + the cert-manager PKI, values re-rooted; a new cozystack.cozyplane-apiserver PackageSource (kubeovn-webhook pattern) installs it with dependsOn cert-manager + etcd-operator, included in the system bundle on the cozyplane variant. Its APIService atomically takes the group over from the CRDs when it lands; existing CRD-stored objects must be exported and re-applied (the takeover is storage-disjoint). Assisted-By: Claude Fable 5 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Assisted-By: Claude Fable 5 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The deployed artifact was built with this template change but it was left out of the bump commit. The server registers (or takes over from CRD autoregistration) its own APIService at startup; the chart manifest is gone because Helm cannot adopt the object the kube-apiserver has already auto-created for the bootstrap CRDs. Assisted-By: Claude Fable 5 Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Sync of the cozyplane charts after the API-group split (lllamnyp/cozyplane,
docs/api-groups.md).
The tenant kinds (VPC, Port, SecurityGroup, HostFirewall, ...) are now served
ONLY by the aggregated apiserver and have no CRDs at all. One group served by
two mechanisms cannot work: a CRD keeps publishing its OpenAPI paths after an
APIService takes the group over, the two specs collide on duplicated paths, the
group's schema never serves, and `kubectl apply` of every cozyplane object
fails client-side with "failed to download openapi" while core types keep
working. That is why the takeover is gone rather than fixed — there is nothing
to take over.
The CNI package now ships only the LOCAL group's CRDs
(local.sdn.cozystack.io): FabricIP, the underlay IPAM claim, whose dependency
floor is the kube API and nothing else — everything above it, cert-manager and
etcd and the aggregated server included, runs as default-network pods.
The apiserver's etcd moves from tmpfs to a PVC on the replicated class. Its
objects are cluster state: memory-backed, a full-cluster restart loses every
VPC, Port and policy — and a node reboot is exactly that. This is the follow-up
the values file already promised ("a persistent mode is a follow-up once storage
is sequenced"), so the component now dependsOn cozystack.linstor. No cycle:
linstor depends on the CNI, not on this.
Also new in the charts: NetworkPolicy/HostFirewall/SecurityGroup RBAC for the
agent, FabricIP RBAC for the agent and controller.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
… schema) Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Bumps the cozyplane image to the build that separates a floating IP's attraction (which node answers ARP for it) from its delivery (which node runs the target pod). The announcer is elected by rendezvous hash over the Ready nodes that can serve the pool link; the request is forwarded over the overlay to the pod's node, and the reply leaves that node directly, sourced as the public address. This makes a live-migrating VM's public address independent of gratuitous ARP: the pod moves, the announcer does not. New chart value floatingHA (default true) switches it off for an underlay that accepts a pool address only from the pod's own node — the same property etp: Cluster DSR needs. Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
A floating IP is a bijection, but only its forward half is keyed by the
public address; the reverse half (floating_egress) is keyed by the target's
{net, VPC IP} alone. Two FloatingIPs on one target therefore do not coexist:
the second overwrites the first's egress entry, and the first address starts
replying from the second — its clients drop the reply and the address goes
silently dead. The datapath cannot detect this, so the controller refuses the
second binding and leaves it Pending.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
A VPC's door was a bool on an object the tenant owns (VPC.spec.egress.natGateway), so a tenant granted itself internet. It is a VPCGateway now, gated on the 'attach' verb on the referenced ExternalPool. LoadBalancer ingress into a VPC is default-DENY: a Service can no longer open a door into a tenant's VPC just by naming its pod as a backend. Bumps both charts (the apiserver must serve the new kind). Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
… should have moved) Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…memory metric Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
cozyplanenetworking variant for theisp-full(Talos) system bundlethat replaces both kube-ovn and Cilium (and kube-proxy) with three
components:
tenancy, Geneve overlay, pinned-identity Ports for VM live migration).
sdn.cozystack.io,a separate component so the CNI slot stays cert-manager-free (bootstrap
ordering below).
importing Cilium's LB control plane (
pkg/loadbalancer+pkg/socketlb) asa standalone DaemonSet with no Cilium agent, plus a per-packet ClusterIP
DNAT fallback for traffic socket-LB structurally can't see (VM guests dial
through the bridge, never through a host socket).
Select it with
bundles.system.networkingVariant: cozyplane.Why drop Cilium entirely
The previous iteration ran cozyplane alongside a standalone Cilium (kept only
for kube-proxy-replacement). That leaves two uncoordinated IPAM authorities on
the node podCIDR — Cilium self-allocates its router/health/ingress endpoints
there while cozyplane's
host-localallocates pod IPs — and they collide (asystem pod drew Cilium's
reserved:healthIP and went cross-node-unreachable).cozyplane-kprlets Cilium be removed, so cozyplane is the single podCIDRauthority and the whole coexistence hazard class is gone.
Bootstrap ordering — the chart split
The API group's serving is deliberately split in two so the install order has
no cycles:
packages/system/cozyplaneinstalls in the CNI slot with nodependencies: it serves
sdn.cozystack.ioas plain bootstrap CRDs, soVPC tenancy works from the moment the CNI lands — before cert-manager (or
anything else that needs a working CNI) can even schedule.
packages/system/cozyplane-apiserver(new) carries the aggregated APIserver, its operator-managed EtcdCluster, and its cert-manager PKI. Its
PackageSource declares
dependsOn: [cert-manager, etcd-operator](thekubeovn-webhook pattern), so it converges later and atomically takes over
the group's serving from the bootstrap CRDs via its APIService. The CRDs
stay installed, shadowed and inert — and become the serving surface again if
the apiserver is removed.
Two mechanics of that takeover shaped the implementation:
auto-registers an APIService for every served CRD group, so the object
always pre-exists (labelled
kube-aggregator.kubernetes.io/automanaged)and Helm refuses to adopt it. The apiserver binary registers/takes over its
own APIService at startup (
--ensure-apiservice-service+--ensure-apiservice-ca-injection), stripping the automanaged label so CRDautoregistration stops reverting it.
watching the shadowed CRD store until the kube-apiserver closes the idle
stream (30–60 min). Fresh installs don't care. Migrating an existing
CRD-mode install must go export → takeover → import → restart the
cozyplane controller + agents — import before restarting, so agent
startup pruning sees a populated store and no-ops instead of tearing down
the live datapath.
Contents
packages/system/cozyplane/— CNI/agent/controller + bootstrap CRDs.packages/system/cozyplane-apiserver/— new: aggregated apiserver +EtcdCluster (3 replicas) + cert-manager Certificate/Issuer;
sources/cozyplane-apiserver.yamlwires it into the system bundle on thecozyplane variant with the
dependsOnabove.packages/system/cozyplane-kpr/— the KPR DaemonSet (hostNetwork,privileged, host bpffs + cgroup2 mounts; on Talos it reaches the apiserver
via KubePrism
https://localhost:7445, since thekubernetes.defaultClusterIP is unserved until kpr itself runs). Also feeds the datapath's
per-packet service-DNAT map for default-network ClusterIPs, which is what
serves KubeVirt guests.
packages/system/cozyplane-cilium-crds/— ships thecilium.iopolicy CRDsinert, so stock components that render CiliumNetworkPolicy manifests still
install on the variant.
sources/networking.yaml— thecozyplanevariant (cozyplane +cozyplane-kpr); removes the old
cozyplane-ciliumvariant andsystem/cilium/values-cozyplane.yaml.networkingVariant: cozyplane.Validation
kubeProxyMode: nonefor the KPR suite): the cozyplanee2e suite is green at 107/107 — overlay delivery, VPC isolation/peering,
security groups, in-VPC ServiceVIPs + split-horizon DNS, floating IPs,
live-migration Port semantics, and the net-0 per-packet ClusterIP DNAT. The
chart split was validated end to end on a separate kind harness: CRD
bootstrap → apiserver install → APIService takeover → object import →
pinned-identity Port re-import.
(91/91 HelmReleases). Exercised there: KubeVirt live migration preserving a
VM's VPC IP+MAC; a VPC VM exposed publicly (FloatingIP on an OCI VLAN behind
a public NLB, SSH+HTTP end to end); VM-guest ClusterIP DNAT verified with
raw socket-LB-bypassing SYN probes; and the CRD→aggregated serving migration
performed live with the datapath intact throughout.
Known gaps (follow-ups, not blockers for the variant)
are not yet replaced.
externalTrafficPolicysemantics) — designed, tracked as cozyplane's KPR increment-3 Half B.
values switch, held for the resilient etcd-operator work (feat(platform): migrate to etcd-operator v1alpha2 #2859).