Skip to content

feat(networking): cozyplane eBPF CNI + cozyplane-kpr networking variant (drops Cilium) - #3149

Draft
Timofei Larkin (lllamnyp) wants to merge 33 commits into
mainfrom
feat/cozyplane
Draft

feat(networking): cozyplane eBPF CNI + cozyplane-kpr networking variant (drops Cilium)#3149
Timofei Larkin (lllamnyp) wants to merge 33 commits into
mainfrom
feat/cozyplane

Conversation

@lllamnyp

@lllamnyp Timofei Larkin (lllamnyp) commented Jun 30, 2026

Copy link
Copy Markdown
Member

What

Adds a cozyplane networking variant for the isp-full (Talos) system bundle
that replaces both kube-ovn and Cilium (and kube-proxy) with three
components:

  • cozyplane — the multi-tenant eBPF CNI (flat default network + per-pod VPC
    tenancy, Geneve overlay, pinned-identity Ports for VM live migration).
  • cozyplane-apiserver — the aggregated API server for sdn.cozystack.io,
    a separate component so the CNI slot stays cert-manager-free (bootstrap
    ordering below).
  • cozyplane-kpr — the kube-proxy replacement: socket-LB at the cgroup root,
    importing Cilium's LB control plane (pkg/loadbalancer + pkg/socketlb) as
    a standalone DaemonSet with no Cilium agent, plus a per-packet ClusterIP
    DNAT fallback for traffic socket-LB structurally can't see (VM guests dial
    through the bridge, never through a host socket).

Select it with bundles.system.networkingVariant: cozyplane.

Why drop Cilium entirely

The previous iteration ran cozyplane alongside a standalone Cilium (kept only
for kube-proxy-replacement). That leaves two uncoordinated IPAM authorities on
the node podCIDR
— Cilium self-allocates its router/health/ingress endpoints
there while cozyplane's host-local allocates pod IPs — and they collide (a
system pod drew Cilium's reserved:health IP and went cross-node-unreachable).
cozyplane-kpr lets Cilium be removed, so cozyplane is the single podCIDR
authority
and the whole coexistence hazard class is gone.

Bootstrap ordering — the chart split

The API group's serving is deliberately split in two so the install order has
no cycles:

  • packages/system/cozyplane installs in the CNI slot with no
    dependencies
    : it serves sdn.cozystack.io as plain bootstrap CRDs, so
    VPC tenancy works from the moment the CNI lands — before cert-manager (or
    anything else that needs a working CNI) can even schedule.
  • packages/system/cozyplane-apiserver (new) carries the aggregated API
    server, its operator-managed EtcdCluster, and its cert-manager PKI. Its
    PackageSource declares dependsOn: [cert-manager, etcd-operator] (the
    kubeovn-webhook pattern), so it converges later and atomically takes over
    the group's serving
    from the bootstrap CRDs via its APIService. The CRDs
    stay installed, shadowed and inert — and become the serving surface again if
    the apiserver is removed.

Two mechanics of that takeover shaped the implementation:

  1. The APIService cannot be a chart manifest. The kube-apiserver
    auto-registers an APIService for every served CRD group, so the object
    always pre-exists (labelled kube-aggregator.kubernetes.io/automanaged)
    and Helm refuses to adopt it. The apiserver binary registers/takes over its
    own APIService at startup (--ensure-apiservice-service +
    --ensure-apiservice-ca-injection), stripping the automanaged label so CRD
    autoregistration stops reverting it.
  2. Established watch streams don't follow the takeover — clients keep
    watching the shadowed CRD store until the kube-apiserver closes the idle
    stream (30–60 min). Fresh installs don't care. Migrating an existing
    CRD-mode install must go export → takeover → import → restart the
    cozyplane controller + agents
    — import before restarting, so agent
    startup pruning sees a populated store and no-ops instead of tearing down
    the live datapath.

Contents

  • packages/system/cozyplane/ — CNI/agent/controller + bootstrap CRDs.
  • packages/system/cozyplane-apiserver/new: aggregated apiserver +
    EtcdCluster (3 replicas) + cert-manager Certificate/Issuer;
    sources/cozyplane-apiserver.yaml wires it into the system bundle on the
    cozyplane variant with the dependsOn above.
  • packages/system/cozyplane-kpr/ — the KPR DaemonSet (hostNetwork,
    privileged, host bpffs + cgroup2 mounts; on Talos it reaches the apiserver
    via KubePrism https://localhost:7445, since the kubernetes.default
    ClusterIP is unserved until kpr itself runs). Also feeds the datapath's
    per-packet service-DNAT map for default-network ClusterIPs, which is what
    serves KubeVirt guests.
  • packages/system/cozyplane-cilium-crds/ — ships the cilium.io policy CRDs
    inert, so stock components that render CiliumNetworkPolicy manifests still
    install on the variant.
  • sources/networking.yaml — the cozyplane variant (cozyplane +
    cozyplane-kpr); removes the old cozyplane-cilium variant and
    system/cilium/values-cozyplane.yaml.
  • platform system bundle + values accept networkingVariant: cozyplane.

Validation

  • kind (dual-stack, kubeProxyMode: none for the KPR suite): the cozyplane
    e2e suite is green at 107/107 — overlay delivery, VPC isolation/peering,
    security groups, in-VPC ServiceVIPs + split-horizon DNS, floating IPs,
    live-migration Port semantics, and the net-0 per-packet ClusterIP DNAT. The
    chart split was validated end to end on a separate kind harness: CRD
    bootstrap → apiserver install → APIService takeover → object import →
    pinned-identity Port re-import.
  • Real Talos/OCI dev cluster: the full platform converges on the variant
    (91/91 HelmReleases). Exercised there: KubeVirt live migration preserving a
    VM's VPC IP+MAC; a VPC VM exposed publicly (FloatingIP on an OCI VLAN behind
    a public NLB, SSH+HTTP end to end); VM-guest ClusterIP DNAT verified with
    raw socket-LB-bypassing SYN probes; and the CRD→aggregated serving migration
    performed live with the datapath intact throughout.

Known gaps (follow-ups, not blockers for the variant)

  • host-firewall and default-network NetworkPolicy that Cilium provided
    are not yet replaced.
  • External NodePort / LoadBalancer per-packet DNAT (externalTrafficPolicy
    semantics) — designed, tracked as cozyplane's KPR increment-3 Half B.
  • The apiserver's etcd currently runs memory-backed; a PVC-backed mode is a
    values switch, held for the resilient etcd-operator work (feat(platform): migrate to etcd-operator v1alpha2 #2859).

@github-actions github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature size/XL This PR changes 500-999 lines, ignoring generated files labels Jun 30, 2026
@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7c9d4267-1c1b-4209-9540-56f4441340da

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/cozyplane

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added size/XXL This PR changes 1000+ lines, ignoring generated files and removed size/XL This PR changes 500-999 lines, ignoring generated files labels Jul 1, 2026
@lllamnyp
Timofei Larkin (lllamnyp) force-pushed the feat/cozyplane branch 2 times, most recently from f37bd80 to 1f18873 Compare July 2, 2026 15:14
Timofei Larkin (lllamnyp) added a commit that referenced this pull request Jul 6, 2026
…ate (#3184)

## What this PR does

Closes two coverage gaps found while shadow-observing the
API-owner-review gate (#3167) against real PRs in the first hours after
merge:

1. **CRD discovery was gated on directory name** (`crds/`,
`definitions/`, `manifests/`), so a CRD shipped inside a chart's
`templates/` dir — a valid, existing convention when the CRD is wrapped
in a `{{- if }}` guard — was invisible to the gate. This was not
hypothetical: `#3149` (cozyplane) adds four brand-new CRDs (`VPC`,
`VPCBinding`, `VPCPeering`, `Port`) exactly this way, and none of them
tripped a review requirement.
2. **Groups served only by an external aggregated apiserver had no
representation at all.** The same `#3149` also registers
`sdn.cozystack.io` via a `kind: APIService` pointing at a prebuilt image
built outside this repository — no CRD, no vendored Go types, nothing to
diff.

### Fix

- CRD/APIService discovery is now content-based: every `.yaml`/`.yml`
file under `packages/` and `internal/` is a candidate, and whether it's
kept depends on whether it parses into the right `kind` with a
first-party group — not which directory it sits in.
- Both parsers now tolerate a Helm directive line (`{{- if }}`, `{{- end
}}`, a comment block) sharing a `---`-delimited document with
otherwise-valid YAML, rather than dropping the whole document. This
matters because the closing `{{- end }}` of a wrapping conditional has
no `---` before it, so it lands in the same document as the manifest it
closes.
- Added a fourth resource source, `SourceAPIService`, for `kind:
APIService` manifests registering a first-party group. Like the existing
static Go-backed groups, these carry no checked-in schema, so they
participate only in new-group / new-resource / removal detection — never
a fabricated breaking-change diff against a schema this repo can't see.

Verified against the real `feat/cozyplane` branch: the gate now reports
all four new CRDs, plus the removal of the old
`securitygroups`/`APIService` registration for the same group — a full
API-surface migration that was previously invisible end to end.

### Release note

```release-note
fix(ci): the API-owner-review gate now discovers CRDs and APIService registrations by content instead of directory name, closing a gap where a CRD or an externally-served aggregated API group could ship without triggering the required review.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Discovery now includes APIService registrations, enabling detection of
groups/versions exposed only via aggregated APIService manifests
(including merging versions from multiple templates).
* **Bug Fixes**
* Improved YAML discovery to tolerate mixed/templated multi-document
streams: non-CRD/non-APIService documents no longer stop discovery, and
embedded template directives are handled more reliably.
* File processing is more resilient during snapshot loading (includes a
safety cap on YAML size).
* **Tests**
* Added coverage for APIService discovery, templated neighbor documents,
and correct exclusion of third-party manifests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
…ariant

A "cozyplane" networking variant that replaces BOTH kube-ovn and Cilium (and
kube-proxy) with cozyplane's eBPF CNI plus cozyplane-kpr for in-cluster
Services (socket-LB, importing Cilium's LB control plane — no Cilium agent).

- packages/system/cozyplane: the CNI/agent/apiserver/controller (image bumped
  to the build with the off-VPC-transit fix; the old 0.3.17 pin had that
  regression, which broke grouped-pod TCP egress via the gateway).
- packages/system/cozyplane-kpr: NEW — the kube-proxy replacement DaemonSet
  (hostNetwork, privileged, host bpffs+cgroup2 mounts, an init container that
  mounts bpffs; on Talos it reaches the apiserver via KubePrism
  https://localhost:7445 since the kubernetes.default ClusterIP is unserved
  until kpr runs). Scoped RBAC (services/endpointslices/nodes read + cilium.io).
- networking.yaml: the "cozyplane" variant = cozyplane + cozyplane-kpr, no
  Cilium; drop the old cozyplane-cilium variant and system/cilium/values-cozyplane.yaml.
- platform system bundle + values: accept networkingVariant "cozyplane".

Single IPAM authority (cozyplane) over the podCIDR — removes the Cilium
standalone-IPAM coexistence hazard (lllamnyp/cozyplane#12). Known gaps carried
as follow-ups: host-firewall + default-network NetworkPolicy (were Cilium's),
and external NodePort + VM-guest ClusterIP (cozyplane-kpr increment 3).

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
@lllamnyp Timofei Larkin (lllamnyp) changed the title feat(networking): add cozyplane eBPF CNI as a networking variant feat(networking): cozyplane eBPF CNI + cozyplane-kpr networking variant (drops Cilium) Jul 7, 2026
…ory-backed)

main's etcd operator moved from etcd.aenix.io/v1alpha1 to
etcd-operator.cozystack.io/v1alpha2 (group+version), and the spec reshaped. Migrate
cozyplane-etcd:

- v1alpha2 EtcdCluster, storage.medium=Memory (no PVC/storageClass) so the
  aggregated API's etcd bootstraps WITHOUT a storage layer — the CNI installs
  first, breaking the CNI->apiserver->etcd->storage->CNI cycle. Replicated raft
  survives single-pod restarts; not durable across a full-cluster restart (a
  persistent mode is a follow-up once storage is sequenced after the CNI).
- TLS via the operator's cert-manager integration: it emits the
  server/operator-client/peer Certificates itself given issuers. Drop the
  hand-rolled server+peer certs; keep the CA issuers and the apiserver's own
  client cert (signed by the same client CA so the etcd server trusts it).
- apiserver dials the operator's client Service (cozyplane-etcd) — v1alpha2
  members carry operator-assigned names, so per-member headless DNS is gone.

helm lint + template clean with operator etcd enabled (the Talos overlay).

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…ager)

cozyplane installs first as the CNI, before cert-manager, but its chart renders
cert-manager Certificates for the apiserver+etcd — the atomic Helm release fails
("no matches for kind Certificate") and the CNI never comes up. Disable the
apiserver in the networking slot so the CNI bootstraps cert-manager-free.
Follow-up: split the chart (CNI component vs apiserver+etcd component that
dependsOn cert-manager) to bring VPC tenancy back.

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
… Talos)

With cozyplane as CNI and no kube-proxy, the kubernetes.default ClusterIP is
unserved until cozyplane-kpr (which needs the agent) is up — so the hostNetwork
agent/responder crashloop on 'dial 10.96.0.1:443: i/o timeout' at bootstrap. Add
a kubeApiServer.{host,port} value (empty = ClusterIP) and set it to Talos
KubePrism localhost:7445 in values-talos, wired as KUBERNETES_SERVICE_HOST/PORT.

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
….cozystack.io to cozyplane

The cozyplane networking variant runs no Cilium and no kube-ovn, but the shared
common-packages helper still instantiated four data-plane packages that assume
them, and cozystack-api still aggregated the sdn.cozystack.io group — which
cozyplane owns as CRDs. Concretely on a fresh cozyplane cluster:

  - multus CrashLoopBackOff: delegates to the Cilium primary conflist
    05-cilium.conflist, which cozyplane never writes (it writes 00-cozyplane).
  - securitygroup-controller CrashLoopBackOff: projects sdn.cozystack.io
    SecurityGroups onto CiliumNetworkPolicy (cilium.io/v2 CRD absent), AND is a
    second, incompatible implementation of securitygroups.sdn.cozystack.io.
  - kubeovn-webhook/-plunger: orphaned kube-ovn control plane (the webhook would
    fail closed once cert-manager lands and block resource creation).
  - cozystack-api aggregates v1alpha1.sdn.cozystack.io -> its explicit APIService
    hijacks the group from cozyplane's CRDs the moment it deploys, 404ing every
    non-SecurityGroup kind (VPC, Port, FloatingIP, ...).

Gate the four packages in common-packages on networkingVariant != cozyplane, and
add cozystackAPI.serveSDN (default true) so the platform bundle turns off the sdn
APIService for the cozyplane path. kubeovn-cilium is unchanged: all four packages
instantiate and cozystack-api keeps serving sdn.

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…g multus dep)

virtualprivatecloud-application (kubevirt) dependsOn cozystack.multus and is the
kube-ovn tenant VPC app; the cozyplane variant installs neither multus nor
kube-ovn and offers VPC tenancy via sdn.cozystack.io, so its Package hung forever
on a multus HR that never gets created. Gate the instantiation on
networkingVariant != cozyplane.

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Image with node_remotes (pod->node overlay encap) + CFG_MASQ_IP (masquerade from
the egress address), fixing cross-node webhooks and pod internet egress on a
spoof-guarding underlay. Adds nodes 'patch' RBAC for the node-address annotation.

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…ts install

linstor, tenant, monitoring-agents, fluxcd-operator (and more) embed
CiliumNetworkPolicy / CiliumClusterwideNetworkPolicy objects and their Helm
installs fail with 'no matches for kind CiliumClusterwideNetworkPolicy' when
Cilium is absent. Ship the two cilium.io/v2 CRDs (from cilium 1.19.5) as a
CRD-only package, gated to the cozyplane variant (the Cilium variants get them
from the cilium chart). Inert — cozyplane enforces no NetworkPolicy yet; the CRDs
exist only so the manifests apply.

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
hostNetwork clients on another node can now reach a VPC pod's fabric IP (the
bridge reply is encapsulated instead of leaving pod-sourced; OCI dropped it).

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…ice DNAT)

Assisted-By: Claude Opus 4.8
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Assisted-By: Claude Fable 5
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…endsOn cert-manager)

The CNI slot stays cert-manager-free: packages/system/cozyplane now serves
sdn.cozystack.io as bootstrap CRDs only (apiserver.enabled dropped for
crds.enabled, apiserver/etcd templates removed; agent.yaml/crds.yaml
re-vendored). The new packages/system/cozyplane-apiserver carries the
aggregated apiserver + the memory-backed v1alpha2 EtcdCluster + the
cert-manager PKI, values re-rooted; a new cozystack.cozyplane-apiserver
PackageSource (kubeovn-webhook pattern) installs it with dependsOn
cert-manager + etcd-operator, included in the system bundle on the cozyplane
variant. Its APIService atomically takes the group over from the CRDs when it
lands; existing CRD-stored objects must be exported and re-applied (the
takeover is storage-disjoint).

Assisted-By: Claude Fable 5
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Assisted-By: Claude Fable 5
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The deployed artifact was built with this template change but it was left
out of the bump commit. The server registers (or takes over from CRD
autoregistration) its own APIService at startup; the chart manifest is
gone because Helm cannot adopt the object the kube-apiserver has already
auto-created for the bootstrap CRDs.

Assisted-By: Claude Fable 5
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Sync of the cozyplane charts after the API-group split (lllamnyp/cozyplane,
docs/api-groups.md).

The tenant kinds (VPC, Port, SecurityGroup, HostFirewall, ...) are now served
ONLY by the aggregated apiserver and have no CRDs at all. One group served by
two mechanisms cannot work: a CRD keeps publishing its OpenAPI paths after an
APIService takes the group over, the two specs collide on duplicated paths, the
group's schema never serves, and `kubectl apply` of every cozyplane object
fails client-side with "failed to download openapi" while core types keep
working. That is why the takeover is gone rather than fixed — there is nothing
to take over.

The CNI package now ships only the LOCAL group's CRDs
(local.sdn.cozystack.io): FabricIP, the underlay IPAM claim, whose dependency
floor is the kube API and nothing else — everything above it, cert-manager and
etcd and the aggregated server included, runs as default-network pods.

The apiserver's etcd moves from tmpfs to a PVC on the replicated class. Its
objects are cluster state: memory-backed, a full-cluster restart loses every
VPC, Port and policy — and a node reboot is exactly that. This is the follow-up
the values file already promised ("a persistent mode is a follow-up once storage
is sequenced"), so the component now dependsOn cozystack.linstor. No cycle:
linstor depends on the CNI, not on this.

Also new in the charts: NetworkPolicy/HostFirewall/SecurityGroup RBAC for the
agent, FabricIP RBAC for the agent and controller.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
… schema)

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Bumps the cozyplane image to the build that separates a floating IP's
attraction (which node answers ARP for it) from its delivery (which node runs
the target pod). The announcer is elected by rendezvous hash over the Ready
nodes that can serve the pool link; the request is forwarded over the overlay
to the pod's node, and the reply leaves that node directly, sourced as the
public address.

This makes a live-migrating VM's public address independent of gratuitous
ARP: the pod moves, the announcer does not.

New chart value floatingHA (default true) switches it off for an underlay
that accepts a pool address only from the pod's own node — the same property
etp: Cluster DSR needs.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
A floating IP is a bijection, but only its forward half is keyed by the
public address; the reverse half (floating_egress) is keyed by the target's
{net, VPC IP} alone. Two FloatingIPs on one target therefore do not coexist:
the second overwrites the first's egress entry, and the first address starts
replying from the second — its clients drop the reply and the address goes
silently dead. The datapath cannot detect this, so the controller refuses the
second binding and leaves it Pending.

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
A VPC's door was a bool on an object the tenant owns (VPC.spec.egress.natGateway),
so a tenant granted itself internet. It is a VPCGateway now, gated on the 'attach'
verb on the referenced ExternalPool. LoadBalancer ingress into a VPC is
default-DENY: a Service can no longer open a door into a tenant's VPC just by
naming its pod as a backend. Bumps both charts (the apiserver must serve the new
kind).

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
… should have moved)

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
…memory metric

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant