diff --git a/Network-Namespace/README.md b/Network-Namespace/README.md new file mode 100644 index 0000000..267cff2 --- /dev/null +++ b/Network-Namespace/README.md @@ -0,0 +1,1776 @@ + +# NetworkExtension for Apache CloudStack + +This directory contains the **NetworkExtension** `NetworkOrchestrator` extension — +a CloudStack plugin that delegates all network operations to an external device +over SSH. The device can be a Linux server (using network namespaces, +bridges, and iptables), a network appliance that accepts SSH commands, or any +other host that can run the `network-namespace-wrapper.sh` (or a compatible +script) to perform network configurations. + +The extension is implemented in +`framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/network/NetworkExtensionElement.java` +and loaded automatically by the management server — **no separate plugin JAR is +required**. + +--- + +## Table of Contents + +1. [Architecture](#architecture) +2. [Directory contents](#directory-contents) +3. [How it works](#how-it-works) +4. [Installation](#installation) + - [Management server](#management-server) + - [Remote network device](#remote-network-device) +5. [Step-by-step API setup](#step-by-step-api-setup) + - [1. Create the extension](#1-create-the-extension) + - [2. Register the extension with a physical network](#2-register-the-extension-with-a-physical-network) + - [3. Create a network offering](#3-create-a-network-offering) + - [4. Create an isolated network](#4-create-an-isolated-network) + - [5. Acquire a public IP and enable Source NAT](#5-acquire-a-public-ip-and-enable-source-nat) + - [6. Enable / disable Static NAT](#6-enable--disable-static-nat) + - [7. Add / delete Port Forwarding](#7-add--delete-port-forwarding) + - [8. Delete the network](#8-delete-the-network) + - [9. Unregister and delete the extension](#9-unregister-and-delete-the-extension) +6. [Multiple extensions on the same physical network](#multiple-extensions-on-the-same-physical-network) +7. [Wrapper script operations reference](#wrapper-script-operations-reference) +8. [Payload reference](#payload-reference) +9. [Custom actions](#custom-actions) +10. [Developer / testing notes](#developer--testing-notes) + +--- + +## Architecture + +``` +┌──────────────────────────────────────────────────────────┐ +│ CloudStack Management Server │ +│ │ +│ NetworkExtensionElement.java │ +│ │ executes (path resolved from Extension record) │ +│ ▼ │ +│ /usr/share/cloudstack-management/extensions// │ +│ .sh (network-namespace.sh) │ +└──────────────────────┬───────────────────────────────────┘ + │ SSH (host : port from extension details) + │ credentials from extension_resource_map_details + ▼ +┌──────────────────────────────────────────────────────────┐ +│ Remote Network Device (KVM Linux server) │ +│ │ +│ network-namespace-wrapper.sh [args...] │ +│ │ +│ Per-network data plane (guest VLAN 1910, network 209): │ +│ │ +│ HOST side │ +│ ───────────────────────────────────────────────── │ +│ eth1.1910 ─────────────────────────────────┐ │ +│ (VLAN sub-iface) │ │ +│ breth1-1910 (bridge) │ +│ vh-1910-d1 ─────────────────────────────────┘ │ +│ │ │ +│ NAMESPACE cs-net-209 (isolated) │ +│ cs-vpc-5 (VPC, vpc-id=5) │ +│ ───────────────────────────────────────────────── │ +│ vn-1910-d1 ← gateway IP 10.1.1.1/24 │ +│ │ +│ PUBLIC side (source-NAT IP 10.0.56.4 on VLAN 101): │ +│ │ +│ HOST side │ +│ eth1.101 ─────────────────────────────────┐ │ +│ breth1-101 (bridge) │ +│ vph-101-209 ────────────────────────────────┘ │ +│ │ │ +│ NAMESPACE cs-net-209 (or cs-vpc-) │ +│ vpn-101-209 ← source-NAT IP 10.0.56.4/32 │ +│ default route → 10.0.56.1 (upstream gateway) │ +└──────────────────────────────────────────────────────────┘ +``` + +### Naming conventions + +| Object | Name pattern | Example (VLAN 1910, net 209, pub-VLAN 101) | +|--------|--------------|-------------------------------------------| +| Namespace (isolated network) | `cs-net-` | `cs-net-209` | +| Namespace (VPC network) | `cs-vpc-` | `cs-vpc-5` | +| Guest host bridge | `br-` | `breth1-1910` | +| Guest veth – host side | `vh--` | `vh-1910-d1` | +| Guest veth – namespace side | `vn--` | `vn-1910-d1` | +| Public host bridge | `br-` | `breth1-101` | +| Public veth – host side | `vph--` | `vph-101-209` | +| Public veth – namespace side | `vpn--` | `vpn-101-209` | + +`ethX` (and `pub_ethX`) is the NIC specified in the `guest.network.device` +(and `public.network.device`) key when registering the extension on the +physical network. Both default to `eth1` when not explicitly set. + +> **Note:** when `` or `` would make the interface name exceed the +> Linux 15-character limit, the `` portion is shortened to its hex +> representation (for numeric IDs) or a 6-character MD5 prefix (for +> non-numeric IDs). + +**Key design principles:** + +* The `network-namespace.sh` script runs on the **management server**. All + connection details (`host`, `port`, `username`, `sshkey`, etc.) are passed as + two named CLI arguments injected by `NetworkExtensionElement` — the script + itself is completely generic and requires no local configuration. +* The `network-namespace-wrapper.sh` script runs on the **remote KVM device**. + It creates host-side bridges, veth pairs, and iptables rules. Bridges and + VLAN sub-interfaces live on the **host** (not inside the namespace) so that + guest VMs whose NICs are connected to `brethX-` reach the namespace + gateway without any additional configuration. +* **VPC networks** share a single namespace per VPC (`cs-vpc-`). Multiple + guest VLANs are each connected via their own veth pair (`vh--` / + `vn--`). +* **Isolated networks** each get their own namespace (`cs-net-`). +* The two scripts are intentionally decoupled: you can replace either script + with a custom implementation (Python, Go, etc.) as long as the interface + contract (arguments and exit codes) is maintained. + +--- + +## Directory contents + +| File | Installed location | Purpose | +|------|--------------------|---------| +| `network-namespace.sh` | management server | SSH proxy — executed by `NetworkExtensionElement` | +| `network-namespace-wrapper.sh` | remote network device | Performs iptables / bridge operations | +| `README.md` | — | This documentation | + +> **Source tree paths:** +> * `network-namespace.sh` → `extensions/network-namespace/network-namespace.sh` +> * `network-namespace-wrapper.sh` → `extensions/network-namespace/network-namespace-wrapper.sh` + +--- + +## How it works + +### Lifecycle of a CloudStack network operation + +1. **CloudStack** decides that a network operation must be applied (e.g. + `implement`, `addStaticNat`, `applyPortForwardingRules`). +2. **`NetworkExtensionElement`** (Java) resolves the extension that is registered + on the physical network whose name matches the network's service provider. It + reads all device details stored in `extension_resource_map_details`. +3. `NetworkExtensionElement` builds a command line: + ``` + /network-namespace.sh + ``` + The payload file includes top-level `physical-network-extension-details`, + top-level `network-extension-details`, and command-specific fields under + `payload` (except `custom-action`, which is a flat top-level payload). +4. **`network-namespace.sh`** parses the payload JSON, writes the SSH + private key to a temporary file (if `sshkey` is set in the physical-network + details), uploads the payload file to the selected host, then runs the wrapper + script remotely as ` `. +5. **`network-namespace-wrapper.sh`** parses the payload and executes the + requested operation using `ip link`, `iptables`, `ip addr`, etc. inside the + network namespace. +6. Exit codes from `network-namespace.sh`: + * `0` — success + * `1` — usage / configuration error (missing arguments, no reachable hosts) + * `2` — SSH connection or authentication error + * `3` — remote wrapper script returned non-zero + + Any non-zero exit causes CloudStack to treat the operation as failed. + +### Authentication priority (network-namespace.sh) + +1. `sshkey` field in `physical-network-extension-details` — PEM key written + to a temp file under `/tmp/.cs-extnet-key-XXXXXX/`, used with `ssh -i`. + **Preferred** — the temp file is deleted on exit. +2. `password` field — passed to `sshpass(1)` if available; `sshpass` must be + installed on the management server. +3. Neither set — relies on the SSH agent or host key on the management server. + +### Host selection (`ensure-network-device`) + +Before every network operation `NetworkExtensionElement` calls `ensure-network-device` +on `network-namespace.sh` (locally, **no SSH**). This selects the KVM host for the +network: + +1. **Sticky re-validation**: if a host was previously selected (from + `payload.current_details.host` or `network-extension-details.host`) *and* that + host is still in the candidate list *and* still reachable, it is kept. +2. **Hash-based selection**: for new or failed-over networks a stable preferred index + is computed as `CRC32() mod len(hosts)` where the routing key is + `vpc_id` for VPC networks (ensuring all tiers land on the same host) or + `network_id` for isolated networks. Hosts are probed in order starting at that + index until one answers. +3. The result is printed as a single-line JSON object: + ```json + {"host":"192.168.1.10","namespace":"cs-net-42"} + ``` + CloudStack stores this in `network_details.extension.details` and forwards it + to later calls through top-level `network-extension-details`. + +You can override the remote wrapper path for testing: +```bash +CS_NET_SCRIPT_PATH=/custom/path/wrapper.sh network-namespace.sh implement-network ... +``` + +--- + +## Installation + +### Management server + +During package installation the `network-namespace.sh` script is deployed to: + +``` +/usr/share/cloudstack-management/extensions//.sh +``` + +The extension path is set to `network-namespace` at creation time; +`NetworkExtensionElement` looks for `.sh` inside the directory. +In **developer mode** the extensions directory defaults to `extensions/` relative +to the repo root, so `extensions/network-namespace/network-namespace.sh` is +found automatically. + +### Remote network device + +Copy `network-namespace-wrapper.sh` to **each** remote device that will act as the +network gateway, inside a subdirectory named after the extension: + +```bash +# From the CloudStack source tree: +DEVICE=root@ +EXT_NAME=network-namespace # must match the extension name in CloudStack + +ssh ${DEVICE} "mkdir -p /etc/cloudstack/extensions/${EXT_NAME}" +scp extensions/network-namespace/network-namespace-wrapper.sh \ + ${DEVICE}:/etc/cloudstack/extensions/${EXT_NAME}/${EXT_NAME}-wrapper.sh +ssh ${DEVICE} "chmod +x /etc/cloudstack/extensions/${EXT_NAME}/${EXT_NAME}-wrapper.sh" +``` + +The wrapper derives its state directory and log path from the directory it is +installed in: + +* **State:** `/var/lib/cloudstack//` + (e.g. `/var/lib/cloudstack/network-namespace/`) +* **Log (wrapper, on KVM host):** `/var/log/cloudstack/extensions//.log` + (e.g. `/var/log/cloudstack/extensions/network-namespace/network-namespace.log`) +* **Log (proxy, on management server):** `/var/log/cloudstack/extensions/.log` + (e.g. `/var/log/cloudstack/extensions/network-namespace.log`) + +Additional per-network service logs are also written to the same directory on the +KVM host: `dnsmasq-.log`, `apache2-.log`, +`passwd-.log`. + +**Prerequisites on the remote device:** + +| Package / tool | Purpose | +|----------------|---------| +| `iproute2` (`ip`, `ip netns`) | Namespace, bridge, veth, route management | +| `iptables` + `iptables-save` | NAT and filter rules inside namespace | +| `arping` | Gratuitous ARP after public IP assignment | +| `dnsmasq` | DHCP and DNS service inside namespace | +| `haproxy` | Load balancing inside namespace | +| `apache2` (Debian/Ubuntu) or `httpd` (RHEL/CentOS) | Metadata / user-data HTTP service (port 80) | +| `python3` | DHCP options parsing, haproxy config generation, vm-data processing | +| `util-linux` (`flock`) | Serialise concurrent operations per network | +| `sshd` | Reachable from the management server on the configured port (default 22) | + +The SSH user must have permission to run `ip`, `iptables`, `iptables-save`, +and `ip netns exec` (root or passwordless `sudo` for those commands). + +--- + +## Step-by-step API setup + +All examples below use `cmk` (the CloudStack CLI). Replace ``, +``, etc. with real values from your environment. + +### 1. Create the extension + +```bash +cmk createExtension \ + name=my-extnet \ + type=NetworkOrchestrator \ + path=network-namespace \ + details[0].network.services="SourceNat,StaticNat,PortForwarding,Firewall,Gateway" \ + details[1].network.service.capabilities="{\"SourceNat\":{\"SupportedSourceNatTypes\":\"peraccount\",\"RedundantRouter\":\"false\"},\"Firewall\":{\"TrafficStatistics\":\"per public ip\"}}" +``` + +The two details declare which services this extension provides and their +CloudStack capability values. These are consulted when listing network service +providers and when validating network offerings. + +**`network.services`** — comma-separated list of service names: +``` +SourceNat,StaticNat,PortForwarding,Firewall,Gateway +``` +Valid service names include: `Vpn`, `Dhcp`, `Dns`, `SourceNat`, +`PortForwarding`, `Lb`, `UserData`, `StaticNat`, `NetworkACL`, `Firewall`, +`Gateway`, `SecurityGroup`. + +**`network.service.capabilities`** — JSON object mapping each service to its +CloudStack `Capability` key/value pairs: +```json +{ + "SourceNat": { + "SupportedSourceNatTypes": "peraccount", + "RedundantRouter": "false" + }, + "Firewall": { + "TrafficStatistics": "per public ip" + } +} +``` + +Services listed in `network.services` that have no entry in +`network.service.capabilities` (e.g. `StaticNat`, `PortForwarding`, +`Gateway`) are still offered — CloudStack treats missing capability values as +"no constraint" and accepts any value when creating the network offering. + +If you omit both details entirely, the extension defaults to an empty set of +services and no capabilities. + +> **Backward compatibility:** the old combined `network.capabilities` JSON +> key (with a `"services"` array and `"capabilities"` object in one blob) is +> still accepted but deprecated. Prefer the split keys above. + +Verify the extension was created and its state is `Enabled`: +```bash +cmk listExtensions name=my-extnet +``` + +To enable or disable the extension: +```bash +cmk updateExtension id= state=Enabled +cmk updateExtension id= state=Disabled +``` + +### 2. Register the extension with a physical network + +```bash +cmk registerExtension \ + id= \ + resourcetype=PhysicalNetwork \ + resourceid= +``` + +This creates a **Network Service Provider** (NSP) entry named `my-extnet` on the +physical network and enables it automatically. The NSP name is the **extension +name** — not the generic string `NetworkExtension`. + +After registering, set the connection details for the remote KVM device(s): + +```bash +cmk updateRegisteredExtension \ + extensionid= \ + resourcetype=PhysicalNetwork \ + resourceid= \ + "details[0].hosts=192.168.10.1,192.168.10.2" \ + "details[1].username=root" \ + "details[2].sshkey=" \ + "details[3].guest.network.device=eth1" \ + "details[4].public.network.device=eth1" +``` + +> **`network.isolation.method=NetworkExtension`** must be set as an Extension +> detail (via `createExtension` or `updateExtension`), not as a physical-network +> registration detail. The network-namespace extension uses VLAN-based isolation +> and does not rely on the script output from `implement-network` to override +> the broadcast domain type, so this detail is not strictly required for basic +> operation. It is included here as best practice and for forward +> compatibility — extensions that return `network.broadcast_domain_type` or +> `network.broadcast_uri` from `implement-network` **must** set it or those +> updates will be silently ignored by CloudStack. + +The `hosts` value is a comma-separated list of KVM host IPs; `ensure-network-device` +picks one per network and stores it in `--network-extension-details`. Use `sshkey` +(PEM private key) for passwordless authentication, or `password` + `sshpass`. + +Verify: +```bash +cmk listNetworkServiceProviders physicalnetworkid= +# → a provider named "my-extnet" should appear in state Enabled +``` + +To disable or re-enable the NSP: +```bash +cmk updateNetworkServiceProvider id= state=Disabled +cmk updateNetworkServiceProvider id= state=Enabled +``` + +To unregister: +```bash +cmk unregisterExtension \ + id= \ + resourcetype=PhysicalNetwork \ + resourceid= +``` + +### 3. Create a network offering + +Use the **extension name** (`my-extnet`) as the service provider — not the +generic string `NetworkExtension`: + +```bash +cmk createNetworkOffering \ + name="My ExtNet Offering" \ + displaytext="Isolated network via my-extnet" \ + guestiptype=Isolated \ + traffictype=GUEST \ + supportedservices="SourceNat,StaticNat,PortForwarding,Firewall,Gateway" \ + "serviceProviderList[0].service=SourceNat" "serviceProviderList[0].provider=my-extnet" \ + "serviceProviderList[1].service=StaticNat" "serviceProviderList[1].provider=my-extnet" \ + "serviceProviderList[2].service=PortForwarding" "serviceProviderList[2].provider=my-extnet" \ + "serviceProviderList[3].service=Firewall" "serviceProviderList[3].provider=my-extnet" \ + "serviceProviderList[4].service=Gateway" "serviceProviderList[4].provider=my-extnet" \ + "serviceCapabilityList[0].service=SourceNat" \ + "serviceCapabilityList[0].capabilitytype=SupportedSourceNatTypes" \ + "serviceCapabilityList[0].capabilityvalue=peraccount" +``` + +Enable the offering: +```bash +cmk updateNetworkOffering id= state=Enabled +``` + +> The `serviceCapabilityList` entries must match the values declared in the +> extension's `network.service.capabilities` detail. If the extension's JSON does +> not declare a capability value for a service, CloudStack accepts any value (or no +> value) without error. + +### 4. Create an isolated network + +```bash +cmk createNetwork \ + name=my-network \ + displaytext="My isolated network" \ + networkofferingid= \ + zoneid= +``` + +When a VM is first deployed into this network, CloudStack calls +`NetworkExtensionElement.implement()`, which triggers the `implement-network` command: + +```bash +# Management server executes: +network-namespace.sh implement-network \ + --network-id 42 \ + --vlan 100 \ + --gateway 10.0.1.1 \ + --cidr 10.0.1.0/24 + +# network-namespace.sh SSHes to the host and runs inside the host: +network-namespace-wrapper.sh implement-network \ + --network-id 42 \ + --vlan 100 \ + --gateway 10.0.1.1 \ + --cidr 10.0.1.0/24 +``` + +The wrapper creates a VLAN sub-interface and Linux bridge, a guest veth pair +(`vh-100-2a`/`vn-100-2a`), assigns the gateway IP to the namespace veth, +enables IP forwarding inside the namespace, and creates per-network iptables +chains: `CS_EXTNET_42_PR` (nat PREROUTING), `CS_EXTNET_42_POST` (nat +POSTROUTING), and `CS_EXTNET_FWD_42` (filter FORWARD). + +> **Note on iptables chains:** +> | Chain | Table | Purpose | +> |-------|-------|---------| +> | `CS_EXTNET__PR` | `nat` | PREROUTING DNAT (port-forward, static-NAT) | +> | `CS_EXTNET__POST` | `nat` | POSTROUTING SNAT (source-NAT, static-NAT outbound) | +> | `CS_EXTNET_FWD_` | `filter` | FORWARD catch-all for this network | +> | `CS_EXTNET_FWRULES_` | `filter` | Firewall egress rules (inserted at pos 1 of FWD chain) | +> | `CS_EXTNET_FWI_` | `mangle` | Firewall ingress per public IP (PREROUTING, before DNAT) | +> | `CS_EXTNET_ACL_` | `filter` | VPC Network ACL (both ingress and egress; pos 1 of FWD) | +> | `CS_EXTNET__VPC_POST` | `nat` | VPC-level SNAT for entire VPC CIDR | + +### 5. Acquire a public IP and enable Source NAT + +```bash +cmk associateIpAddress networkid= +``` + +CloudStack calls `applyIps()` which issues `assign-ip` with `--source-nat true` +for the source-NAT IP: + +```bash +network-namespace.sh assign-ip \ + --network-id 42 \ + --vlan 100 \ + --public-ip 203.0.113.10 \ + --source-nat true \ + --gateway 10.0.1.1 \ + --cidr 10.0.1.0/24 +``` + +The wrapper: +1. Creates public VLAN sub-interface `eth1.` and bridge `breth1-` on the host. +2. Creates veth pair `vph--42` (host, in bridge) / `vpn--42` (namespace). +3. Assigns `203.0.113.10/32` to `vpn--42` **inside the namespace**. +4. Adds host route `203.0.113.10/32 dev vph--42` so the host can reach it. +5. Adds an iptables SNAT rule in `CS_EXTNET_42_POST`: traffic from `10.0.1.0/24` + out `vpn--42` → source `203.0.113.10`. +6. Adds an iptables FORWARD ACCEPT rule in `CS_EXTNET_FWD_42` for the guest CIDR. +7. If `--public-gateway` is set, adds/replaces the namespace default route via + `vpn--42`. + +When the IP is released (via `disassociateIpAddress`), `release-ip` is called, +which removes all associated rules and the IP address. + +### 6. Enable / disable Static NAT + +```bash +# Enable static NAT: map public IP 203.0.113.20 to VM private IP 10.0.1.5 +cmk enableStaticNat \ + ipaddressid= \ + virtualmachineid= \ + networkid= +``` + +CloudStack calls `applyStaticNats()` → `add-static-nat`: + +```bash +network-namespace.sh add-static-nat \ + --network-id 42 \ + --vlan 100 \ + --public-ip 203.0.113.20 \ + --private-ip 10.0.1.5 +``` + +iptables rules added (all run inside the namespace via `ip netns exec`): +```bash +# DNAT inbound (CS_EXTNET_42_PR = nat PREROUTING chain) +iptables -t nat -A CS_EXTNET_42_PR -d 203.0.113.20 -j DNAT --to-destination 10.0.1.5 +# SNAT outbound (CS_EXTNET_42_POST = nat POSTROUTING chain) +iptables -t nat -A CS_EXTNET_42_POST -s 10.0.1.5 -o vpn--42 -j SNAT --to-source 203.0.113.20 +# FORWARD inbound + outbound (CS_EXTNET_FWD_42 = filter FORWARD chain) +iptables -t filter -A CS_EXTNET_FWD_42 -d 10.0.1.5 -o vn-100-2a -j ACCEPT +iptables -t filter -A CS_EXTNET_FWD_42 -s 10.0.1.5 -i vn-100-2a -j ACCEPT +``` + +```bash +# Disable static NAT +cmk disableStaticNat ipaddressid= +``` + +CloudStack calls `delete-static-nat`, which removes all four rules above. + +### 7. Add / delete Port Forwarding + +```bash +# Forward TCP port 2222 on public IP 203.0.113.20 → VM port 22 +cmk createPortForwardingRule \ + ipaddressid= \ + privateport=22 \ + publicport=2222 \ + protocol=TCP \ + virtualmachineid= \ + networkid= +``` + +CloudStack calls `applyPFRules()` → `add-port-forward`: + +```bash +network-namespace.sh add-port-forward \ + --network-id 42 \ + --vlan 100 \ + --public-ip 203.0.113.20 \ + --public-port 2222 \ + --private-ip 10.0.1.5 \ + --private-port 22 \ + --protocol TCP +``` + +iptables rules added (inside the namespace): +```bash +# DNAT inbound (CS_EXTNET_42_PR = nat PREROUTING chain) +iptables -t nat -A CS_EXTNET_42_PR -p tcp -d 203.0.113.20 --dport 2222 \ + -j DNAT --to-destination 10.0.1.5:22 +# FORWARD (CS_EXTNET_FWD_42 = filter FORWARD chain) +iptables -t filter -A CS_EXTNET_FWD_42 -p tcp -d 10.0.1.5 --dport 22 \ + -o vn-100-2a -j ACCEPT +``` + +Port ranges (e.g. `80:90`) are supported and passed verbatim to iptables `--dport`. + +```bash +# Delete the rule +cmk deletePortForwardingRule id= +``` + +This calls `delete-port-forward` which removes the DNAT and FORWARD rules. + +### 8. Delete the network + +```bash +cmk deleteNetwork id= +``` + +CloudStack calls `shutdown-network` (to clean up active state) then +`destroy-network` (full removal): + +```bash +network-namespace.sh shutdown-network --network-id 42 --vlan 100 +network-namespace.sh destroy-network --network-id 42 --vlan 100 +``` + +**`shutdown-network`** wrapper actions: +1. Removes iptables jump rules and flushes/deletes per-network chains + (`CS_EXTNET_42_PR`, `CS_EXTNET_42_POST`, `CS_EXTNET_FWD_42`). +2. Stops dnsmasq, haproxy, apache2, and password-server processes. +3. Deletes public veth pairs (`vph--42` / `vpn--42`) that were + created during `assign-ip` (read from state files). +4. Deletes the guest veth host-side (`vh-100-2a`). +5. For **isolated** networks: deletes the namespace `cs-net-42`. +6. For **VPC tier** networks: preserves the shared namespace `cs-vpc-`. + +**`destroy-network`** wrapper actions (similar to `shutdown-network`, plus): +1. Deletes the guest veth host-side (`vh-100-2a`). +2. Deletes public veth pairs owned by this tier. +3. Stops per-network services. +4. Removes per-network state directory `/var/lib/cloudstack//network-42/`. +5. For **isolated** networks: deletes the namespace `cs-net-42`. +6. For **VPC tier** networks: deregisters this tier from the VPC — namespace is + only removed by a subsequent `destroy-vpc` call. + +> The host bridge `breth1-100` and VLAN sub-interface `eth1.100` are removed +> once nothing else is attached to the bridge (checked via +> `teardown_host_bridge_if_unused`). If another network/tenant is still +> sharing the same physical VLAN, or a VM tap is still attached, the bridge +> and VLAN sub-interface are left in place. + +### 9. Unregister and delete the extension + +```bash +# Disable and delete the NSP +cmk updateNetworkServiceProvider id= state=Disabled +cmk deleteNetworkServiceProvider id= + +# Remove external network device credentials (if any) +# Device credentials are stored as extension_resource_map_details for the +# extension registration. Remove or update them via `updateRegisteredExtension` +# (set cleanupdetails=true to wipe all details) or by supplying new details. +# Example: clear all registration details for a physical network: +cmk updateRegisteredExtension \ + extensionid= \ + resourcetype=PhysicalNetwork \ + resourceid= \ + cleanupdetails=true + +# Unregister the extension from the physical network +cmk unregisterExtension \ + id= \ + resourcetype=PhysicalNetwork \ + resourceid= + +# Delete the extension +# (only possible once it is unregistered from all physical networks) +cmk deleteExtension id= +``` + +--- + +## Multiple extensions on the same physical network + +Because each extension is registered as its own NSP (named after the extension), +multiple independent external network providers can coexist on the same physical +network: + +```bash +# Register two extensions, each backed by a different device +cmk registerExtension id= resourcetype=PhysicalNetwork resourceid= +cmk registerExtension id= resourcetype=PhysicalNetwork resourceid= + +# Store device connection details as registration details for each extension. +# Details are stored in extension_resource_map_details for the registration. +# Example: set hosts and guest/public network devices for ext-a on the physical network: +cmk updateRegisteredExtension \ + extensionid= \ + resourcetype=PhysicalNetwork \ + resourceid= \ + "details[0].hosts=10.0.0.1,10.0.0.2" \ + "details[1].guest.network.device=eth1" \ + "details[2].public.network.device=eth1" +``` + +When creating network offerings, reference the specific extension name: + +```bash +# Network offering backed by ext-a-name +cmk createNetworkOffering ... \ + "serviceProviderList[0].provider=ext-a-name" ... + +# Network offering backed by ext-b-name +cmk createNetworkOffering ... \ + "serviceProviderList[0].provider=ext-b-name" ... +``` + +CloudStack resolves which extension to call by: +1. Looking up the service provider name stored in `ntwk_service_map` for the + guest network. +2. Finding the registered extension on the physical network whose name matches + that provider name. +3. Calling `NetworkExtensionElement` scoped to that specific provider/extension + (via `NetworkExtensionElement.withProviderName()`). + +--- + +## IPv6 support + +The network-namespace extension supports IPv6 guest networks via stateless address auto-configuration +(SLAAC) using **radvd** (Router Advertisement Daemon), mirroring the approach used by CloudStack's +built-in VPC router (`CsVpcGuestNetwork.py`). + +### How it works + +When `network_ip6_gateway` and `network_ip6_cidr` are present in the `implement-network` payload: + +1. **`implement-network`** enables IPv6 in the namespace (`net.ipv6.conf.all.disable_ipv6=0`, + forwarding on, DAD and temporary addresses disabled) and assigns the IPv6 gateway address + to the guest veth interface inside the namespace. +2. **`config-dhcp-subnet`** or **`config-dns-subnet`** writes `/radvd/radvd.conf` and starts + radvd inside the namespace. radvd sends Router Advertisements on the guest veth, advertising + the `/64` (or configured) prefix so VMs can self-configure via SLAAC. +3. **`add-dns-entry`** adds both an A record (IPv4) and an AAAA record (IPv6, from `ip6_address`) + to the dnsmasq hosts file so guests can resolve VM hostnames over IPv6. +4. **`remove-dhcp-subnet`** / **`shutdown-network`** / **`destroy-network`** stops radvd and + removes its state directory. + +### radvd configuration + +The generated `radvd.conf` follows the same format as `CsVpcGuestNetwork.py`: + +``` +interface +{ + AdvSendAdvert on; + MinRtrAdvInterval 5; + MaxRtrAdvInterval 15; + prefix / + { + AdvOnLink on; + AdvAutonomous on; + }; + RDNSS # one block per server from dns6 + { + AdvRDNSSLifetime 30; + }; +}; +``` + +### VPC networks + +For VPC networks the shared namespace is created by `implement-vpc` (IPv6-neutral — no IPv6 is +enabled or disabled at that stage). Each VPC tier network runs its own `implement-network`, which +independently enables or disables IPv6 on its own guest veth. This allows mixed VPC topologies +where some tiers have IPv6 and others do not. + +Each IPv6 tier also runs its own **radvd** instance inside the shared namespace, bound to its own +guest veth. radvd correctly handles multiple instances in the same namespace as long as each +instance is bound to a distinct interface. + +### State files + +IPv6 state is persisted under the per-network state directory: + +| File | Content | +|------|---------| +| `network-/ip6-gateway` | IPv6 gateway address assigned to the namespace veth | +| `network-/ip6-cidr` | IPv6 CIDR of the guest subnet | +| `network-/dns6` | Comma-separated IPv6 DNS server list (RDNSS) | +| `network-/radvd/radvd.conf` | Generated radvd configuration | +| `network-/radvd/radvd.pid` | PID of the running radvd process | + +--- + +## Wrapper script operations reference + +CloudStack now invokes the wrapper through payload files. + +The `network-namespace-wrapper.sh` script runs on the remote KVM device. +It receives the command as its first positional argument followed by named +`--option value` pairs. + +All commands: +* Write timestamped entries to `/var/log/cloudstack/extensions//.log`. +* Use a per-network flock file (`${STATE_DIR}/lock-network-`) — or + `lock-vpc-` for VPC networks — to serialise concurrent operations. +* Persist state under `/var/lib/cloudstack//network-/` + (or `vpc-/` for VPC-wide shared state such as public IPs). + +### `implement-network` + +Called when CloudStack activates the network (typically on first VM deploy). + +``` +network-namespace-wrapper.sh implement-network \ + --network-id \ + --vlan \ + --gateway \ + --cidr \ + [--extension-ip ] \ + [--vpc-id ] +``` + +Actions: +1. Create namespace `cs-vpc-` (VPC) or `cs-net-` (isolated). +2. Resolve `GUEST_ETH` from `guest.network.device` in `physical-network-extension-details` + (defaults to `eth1` when absent). +3. Create VLAN sub-interface `GUEST_ETH.` on the host. +4. Create host bridge `br-` and attach `GUEST_ETH.` to it. +5. Create veth pair `vh--` (host, in bridge) / `vn--` (namespace). +6. Assign `/` (or `/` when + `--extension-ip` is not given) to `vn--` inside the namespace. + When the extension IP differs from the gateway a default route via the gateway + is also added inside the namespace. +7. IPv6 handling: + - When `--network-ip6-gateway` / `--network-ip6-cidr` are provided: enable IPv6 forwarding + inside the namespace (disable DAD, enable forwarding and accept_ra) and assign the IPv6 + gateway address to `vn--`. + - When no IPv6 is configured: disable IPv6 on the guest veth interface. + For standalone networks the global namespace disable is also applied; for VPC tier networks + the global setting is left unchanged to avoid clobbering sibling tiers that may have IPv6. +8. Enable IPv4 IP forwarding inside the namespace. +9. Create iptables chains `CS_EXTNET__PR` (nat PREROUTING DNAT), + `CS_EXTNET__POST` (nat POSTROUTING SNAT), and `CS_EXTNET_FWD_` (filter FORWARD). +10. Save VLAN, gateway, CIDR, extension-ip, IPv6 gateway, IPv6 CIDR, and namespace to state files. + +### `shutdown-network` + +Called when a network is shut down (may be restarted later). + +``` +network-namespace-wrapper.sh shutdown-network \ + --network-id [--vlan ] [--vpc-id ] +``` + +Actions: +1. Remove iptables jump rules for this network and flush/delete its chains + (`CS_EXTNET__PR`, `CS_EXTNET__POST`, `CS_EXTNET_FWD_`). +2. Delete public veth pairs (`vph--` / `vpn--`) that are + owned by this tier (guarded by per-IP `.tier` state files). +3. Delete the guest veth host-side (`vh--`). +4. Stop dnsmasq, haproxy, apache2, and password-server processes. +5. For **isolated** networks: delete the namespace `cs-net-`. +6. For **VPC tier** networks: preserve the shared namespace `cs-vpc-`. + +### `destroy-network` + +Called when the network is permanently removed. + +``` +network-namespace-wrapper.sh destroy-network \ + --network-id [--vlan ] [--vpc-id ] +``` + +Actions: +1. Delete guest veth host-side (`vh--`). +2. Delete public veth pairs that belong to this tier (guarded by `.tier` state files). +3. Stop dnsmasq, haproxy, apache2, and password-server processes. +4. Remove per-network state directory `network-/`. +5. For **isolated** networks: delete the namespace `cs-net-`. +6. For **VPC tier** networks: deregister this tier from the VPC tracking directory + (`vpc-/tiers/`) — the namespace is preserved and will be + removed by a subsequent `destroy-vpc` call. + +> The host bridge `br-` and VLAN sub-interface `GUEST_ETH.` +> are removed on destroy once the bridge has no remaining member interfaces +> (`teardown_host_bridge_if_unused`). This is a no-op if another network/tenant +> is still sharing the same physical VLAN, or a VM tap is still attached. + +### VPC lifecycle commands: `implement-vpc`, `update-vpc-source-nat-ip`, `shutdown-vpc`, `destroy-vpc` + +These commands manage VPC-level state. Called by `NetworkExtensionElement` when +implementing, shutting down, or destroying a VPC (before or after per-tier +network operations). + +#### `implement-vpc` + +``` +network-namespace-wrapper.sh implement-vpc \ + --vpc-id \ + [--vpc-cidr ] \ + [--public-ip ] [--public-vlan ] \ + [--public-gateway ] [--public-cidr ] \ + [--source-nat true|false] +``` + +Actions: +1. Create the shared VPC namespace `cs-vpc-` (idempotent). +2. Enable IPv4 IP forwarding inside the namespace. + IPv6 is managed per-tier by `implement-network` (each tier enables or disables IPv6 on its + own guest veth without affecting sibling tiers). +3. Optionally, when `--source-nat true`, `--public-ip`, and `--public-vlan` are all + provided and `--vpc-cidr` (VPC CIDR) is given: + * Create public veth pair `vph--` (host) / `vpn--` (namespace). + * Assign `` to `vpn--` inside the namespace. + * Set namespace default route via `--public-gateway` (if given). + * Add VPC-level SNAT rule in chain `CS_EXTNET__VPC_POST`: + all VPC traffic (``) out `vpn--` → ``. +4. Save VPC namespace name and CIDR to + `/var/lib/cloudstack//vpc-/`. + +> This command runs **before** any tier networks are implemented. Tier networks +> inherit the same namespace. + +#### `update-vpc-source-nat-ip` + +``` +network-namespace-wrapper.sh update-vpc-source-nat-ip \ + --vpc-id \ + --public-ip \ + [--vpc-cidr ] \ + [--public-vlan ] \ + [--public-gateway ] \ + [--public-cidr ] \ + [--source-nat true|false] +``` + +Actions: +1. Ensure the target public veth pair exists (`vph--` / `vpn--`) and assign the new public IP inside the VPC namespace. +2. Update host and namespace routes for the new source NAT egress path: + * keep host route `/32` via `vph--` + * replace namespace default route via `--public-gateway` on `vpn--` when provided. +3. Rebuild VPC SNAT chain `CS_EXTNET__VPC_POST` so exactly one SNAT rule remains: + * `-s -o vpn-- -j SNAT --to-source `. +4. Reconcile persisted VPC IP markers under + `/var/lib/cloudstack//vpc-/ips/`: + * set the new source NAT IP file to `true` + * set all other VPC public IP marker files to `false` + * persist/update `.pvlan` for the new source NAT IP. + +> This command is used by `NetworkExtensionElement.updateVpcSourceNatIp()` when +> `updateVPC` is called with `sourcenatipaddress`; it avoids full VPC restart. + +#### `shutdown-vpc` + +``` +network-namespace-wrapper.sh shutdown-vpc \ + --vpc-id +``` + +Actions: +1. Delete the VPC namespace `cs-vpc-` (which removes all interfaces + inside it, including per-tier veth pairs). + +> Called after all tier networks have been shut down. The namespace itself is the +> only resource removed — any host-side bridges and VLAN sub-interfaces are left +> intact. + +#### `destroy-vpc` + +``` +network-namespace-wrapper.sh destroy-vpc \ + --vpc-id +``` + +Actions: +1. Delete the VPC namespace `cs-vpc-` (if it still exists). +2. Remove VPC-wide state directory `/var/lib/cloudstack//vpc-/`. + +> This is the final cleanup step; after this, all VPC namespace state is gone. + +### `assign-ip` + +Called when a public IP is associated with the network (including source NAT). + +``` +network-namespace-wrapper.sh assign-ip \ + --network-id \ + --vlan \ + --public-ip \ + --source-nat true|false \ + --gateway \ + --cidr \ + --public-vlan \ + [--public-gateway ] \ + [--public-cidr ] \ + [--vpc-id ] +``` + +Actions: +1. Resolve `PUB_ETH` from `public.network.device` in `physical-network-extension-details` + (defaults to `eth1` when absent). +2. Create VLAN sub-interface `PUB_ETH.` and bridge `br-` on the host. +3. Create veth pair `vph--` (host) / `vpn--` (namespace). + Attach host end to `br-`. +4. Assign `/32` (or `/` if `--public-cidr` given) to + `vpn--` inside the namespace. +5. Add host route `/32 dev vph--` so the host can reach it. +6. Send a gratuitous ARP (`arping -U`) from `vpn--` to flush stale ARP + entries in the upstream gateway (requires `arping` installed on the KVM host; + skipped silently when not available). +7. If `--public-gateway` is given, set/replace namespace default route via + `vpn--`. +8. If `--source-nat true` (and `--vpc-id` is **not** set): + * SNAT rule: `` out `vpn--` → `` + (POSTROUTING chain `CS_EXTNET__POST`). + * FORWARD ACCEPT for `` towards `vpn--`. + * For VPC tiers (`--vpc-id` present), SNAT is managed by `implement-vpc` — + `assign-ip` skips the SNAT rules. +9. Save public VLAN to state file `ips/.pvlan` and owning tier to + `ips/.tier` (used by `add-static-nat`, `add-port-forward`, `release-ip`). + +### `release-ip` + +Called when a public IP is released / disassociated from the namespace. + +``` +network-namespace-wrapper.sh release-ip \ + --network-id \ + --public-ip \ + [--public-vlan ] \ + [--public-cidr ] \ + [--vpc-id ] +``` + +Actions: +1. Load `public_vlan` from `ips/.pvlan` state file. +2. Remove SNAT rule for guest CIDR → ``. +3. Remove any DNAT rules targeting `` from PREROUTING chain. +4. Remove host route `/32`. +5. Remove IP address from `vpn--` inside namespace. +6. If no other IPs share the same `/` combination, delete + `vph--` (host veth), then attempt to remove the public bridge + `br-` and VLAN sub-interface `PUB_ETH.` — a no-op + (`teardown_host_bridge_if_unused`) if another network/tenant is still + sharing the same public VLAN. +7. Remove state files. + +### `add-static-nat` + +Called when Static NAT (one-to-one NAT) is enabled for a public IP. + +``` +network-namespace-wrapper.sh add-static-nat \ + --network-id \ + --vlan \ + --public-ip \ + --private-ip \ + [--vpc-id ] +``` + +The `public_vlan` for this IP is loaded from `ips/.pvlan` state +(written during `assign-ip`). + +iptables rules added (chains `CS_EXTNET__PR` / `_POST` / `FWD_`): + +| Table | Chain | Rule | +|-------|-------|------| +| `nat` | `CS_EXTNET__PR` | `-d -j DNAT --to-destination ` | +| `nat` | `CS_EXTNET__POST` | `-s -o vpn-- -j SNAT --to-source ` | +| `filter` | `CS_EXTNET_FWD_` | `-d -o vn-- -j ACCEPT` | +| `filter` | `CS_EXTNET_FWD_` | `-s -i vn-- -j ACCEPT` | + +State saved to `${STATE_DIR}/network-/static-nat/`. + +### `delete-static-nat` + +``` +network-namespace-wrapper.sh delete-static-nat \ + --network-id \ + --public-ip \ + [--private-ip ] +``` + +Removes all four rules added by `add-static-nat`. If `--private-ip` is omitted, +it is read from the state file. + +### `add-port-forward` + +Called when a Port Forwarding rule is added. + +``` +network-namespace-wrapper.sh add-port-forward \ + --network-id \ + --vlan \ + --public-ip \ + --public-port \ + --private-ip \ + --private-port \ + --protocol tcp|udp +``` + +iptables rules added (inside the namespace): + +| Table | Chain | Rule | +|-------|-------|------| +| `nat` | `CS_EXTNET__PR` | `-p -d --dport -j DNAT --to-destination :` | +| `filter` | `CS_EXTNET_FWD_` | `-p -d --dport -o vn-- -j ACCEPT` | + +Port ranges (`80:90`) are passed verbatim to iptables `--dport`. + +State saved to +`${STATE_DIR}/network-/port-forward/__`. + +### `delete-port-forward` + +``` +network-namespace-wrapper.sh delete-port-forward \ + --network-id \ + --public-ip \ + --public-port \ + --private-ip \ + --private-port \ + --protocol tcp|udp +``` + +Removes the DNAT and FORWARD rules added by `add-port-forward`. + +### `prepare-nic` + +Called when a VM NIC is being attached to the network (before the VM boots). + +``` +network-namespace-wrapper.sh prepare-nic \ + --network-id \ + --vlan \ + --mac \ + --ip \ + [--hostname ] \ + [--default-nic true|false] \ + [--gateway ] \ + [--cidr ] \ + [--extension-ip ] \ + [--vpc-id ] +``` + +Actions (all idempotent; silently skipped when the service is not yet configured): +1. If dnsmasq DHCP is active for the network — add a static lease + `,[,],infinite` to the hosts file. For secondary NICs + (`default_nic=false`) the gateway DHCP option is suppressed via a + `set:norouter_` tag so the VM does not receive a competing default + route from this NIC. +2. If dnsmasq DNS is active — add a ` ` line to the hosts file. +3. Sends a SIGHUP / reload to dnsmasq so the new entries take effect + immediately. + +### `release-nic` + +Called when a VM NIC is being detached from the network (after the VM stops). + +``` +network-namespace-wrapper.sh release-nic \ + --network-id \ + --mac \ + --ip \ + [--vpc-id ] +``` + +Actions: +1. Remove the MAC's DHCP static lease and any associated gateway-suppression + option from dnsmasq. +2. Remove the VM's hostname from the dnsmasq hosts file. +3. Reload dnsmasq. +4. Delete the per-VM metadata directory + `${STATE_DIR}/network-/metadata//`. +5. Remove the VM's password entry from the passwords file. + +### `apply-fw-rules` + +Called when CloudStack applies or removes firewall rules for the network. + +``` +network-namespace-wrapper.sh apply-fw-rules \ + --network-id \ + --vlan \ + [--vpc-id ] +``` + +The `fw_rules` field in the payload is a JSON object: +```json +{ + "default_egress_allow": true, + "cidr": "10.0.1.0/24", + "rules": [ + { + "type": "ingress", + "protocol": "tcp", + "portStart": 22, + "portEnd": 22, + "publicIp": "203.0.113.10", + "sourceCidrs": ["0.0.0.0/0"] + }, + { + "type": "egress", + "protocol": "all", + "sourceCidrs": ["0.0.0.0/0"] + } + ] +} +``` + +iptables design (two independent parts, both inside the namespace): + +* **Ingress** (mangle PREROUTING, per public IP): + Per-public-IP chains `CS_EXTNET_FWI_` check traffic *before* DNAT so + the match is against the real public destination IP. Traffic not matched by + explicit ALLOW rules is dropped. + +* **Egress** (filter FORWARD, chain `CS_EXTNET_FWRULES_`): + Inserted at position 1 of `CS_EXTNET_FWD_`. Applies the + `default_egress_allow` policy (allow-by-default or deny-by-default) to VM + outbound traffic on `-i vn--`. + +### `apply-network-acl` + +Apply Network ACL (Access Control List) rules for VPC networks. + +``` +network-namespace-wrapper.sh apply-network-acl \ + --network-id \ + --vlan \ + [--vpc-id ] +``` + +The `acl_rules` field in the payload is a JSON array of ACL rule objects: +```json +[ + { + "id": 1, + "number": 100, + "trafficType": "Ingress", + "action": "Allow", + "protocol": "tcp", + "portStart": 80, + "portEnd": 80, + "sourceCidrs": ["0.0.0.0/0"] + }, + { + "id": 2, + "number": 200, + "trafficType": "Egress", + "action": "Allow", + "protocol": "all", + "destCidrs": ["0.0.0.0/0"] + } +] +``` + +iptables design: + +* A single **filter FORWARD** chain `CS_EXTNET_ACL_` handles both + ingress and egress traffic. It is inserted at position 1 of + `CS_EXTNET_FWD_` so ACL rules take precedence over catch-all ACCEPT + rules. +* `RELATED,ESTABLISHED` traffic is always accepted first (so active sessions are + not interrupted). +* Rules are applied in ascending `number` order. +* **Ingress rules** (`trafficType: Ingress`) match `-o vn--` (traffic + going *into* the VM subnet, optionally filtered by `-d `). +* **Egress rules** (`trafficType: Egress`) match `-i vn--` (traffic + *from* the VM subnet, with `sourceCidrs` used as destination filter `-d`). +* A terminal DROP rule at the end of the chain enforces the implicit deny policy. + +### `config-dhcp-subnet` / `remove-dhcp-subnet` + +Configure or tear down dnsmasq DHCP service for the network inside the namespace. + +**`config-dhcp-subnet` arguments:** +``` +network-namespace-wrapper.sh config-dhcp-subnet \ + --network-id \ + --gateway \ + --cidr \ + [--dns ] \ + [--domain ] \ + [--vpc-id ] +``` + +Actions: writes a dnsmasq configuration file under +`${STATE_DIR}/network-/dnsmasq/` and starts or reloads the dnsmasq process +inside the namespace. DNS on port 53 is **disabled** by `config-dhcp-subnet` +(use `config-dns-subnet` to enable it). + +**`remove-dhcp-subnet` arguments:** +``` +network-namespace-wrapper.sh remove-dhcp-subnet --network-id +``` + +Actions: stops dnsmasq and removes the dnsmasq configuration directory. + +### `add-dhcp-entry` / `remove-dhcp-entry` + +Add or remove a static DHCP host reservation (MAC → IP mapping) from dnsmasq. + +``` +network-namespace-wrapper.sh add-dhcp-entry \ + --network-id \ + --mac \ + --ip \ + [--hostname ] \ + [--default-nic true|false] +``` + +When `--default-nic false`, the DHCP option 3 (default gateway) is suppressed +for that MAC so the VM does not get a competing default route via a secondary NIC. + +``` +network-namespace-wrapper.sh remove-dhcp-entry \ + --network-id \ + --mac +``` + +### `set-dhcp-options` + +Set extra DHCP options for a specific NIC (identified by `--nic-id`) using a +JSON map of option-code → value pairs. + +``` +network-namespace-wrapper.sh set-dhcp-options \ + --network-id \ + --nic-id \ + --options '{"119":"example.com"}' +``` + +### `config-dns-subnet` / `remove-dns-subnet` + +Enable or disable DNS (port 53) in the dnsmasq instance. + +``` +network-namespace-wrapper.sh config-dns-subnet \ + --network-id \ + --gateway \ + --cidr \ + [--extension-ip ] \ + [--domain ] \ + [--vpc-id ] +``` + +Actions: like `config-dhcp-subnet` but enables DNS on port 53. Also registers a +`data-server` hostname entry (using `--extension-ip` if provided, otherwise +`--gateway`) for metadata service discovery. + +``` +network-namespace-wrapper.sh remove-dns-subnet --network-id +``` + +Actions: disables DNS (rewrites config to disable port 53) but keeps DHCP running. + +### `add-dns-entry` / `remove-dns-entry` + +Add or remove a hostname → IP mapping in the dnsmasq hosts file. + +``` +network-namespace-wrapper.sh add-dns-entry \ + --network-id \ + --ip \ + --hostname + +network-namespace-wrapper.sh remove-dns-entry \ + --network-id \ + --ip +``` + +### `save-vm-data` + +Write the full VM metadata/userdata/password set for a VM in a single call. +Called on network restart and VM deploy. + +``` +network-namespace-wrapper.sh save-vm-data \ + --network-id \ + --ip +``` + +The `vm_data` field in the payload is a JSON array of `{dir, file, content}` +entries (same format as `generateVmData()` in the Java layer). Each `content` +value is a plain UTF-8 string. Writes files under +`${STATE_DIR}/network-/metadata//latest/`. After writing, starts or +reloads both the **apache2 metadata HTTP service** (port 80) and the +**VR-compatible password server** (port 8080) inside the namespace. + +### `save-userdata` / `save-password` / `save-sshkey` / `save-hypervisor-hostname` + +Granular variants that write individual VM metadata fields: + +``` +network-namespace-wrapper.sh save-userdata --network-id --ip --userdata +network-namespace-wrapper.sh save-password --network-id --ip --password +network-namespace-wrapper.sh save-sshkey --network-id --ip --sshkey +network-namespace-wrapper.sh save-hypervisor-hostname \ + --network-id --ip --hypervisor-hostname +``` + +Each command writes the relevant file and restarts/reloads apache2 (and +the password server, for `save-password`). + +### `apply-lb-rules` + +Apply or revoke load-balancing rules via haproxy inside the namespace. + +``` +network-namespace-wrapper.sh apply-lb-rules \ + --network-id \ + --lb-rules \ + [--vpc-id ] +``` + +`--lb-rules` is a JSON array of LB rule objects. Set `"revoke": true` on a +rule to remove it. The wrapper regenerates the haproxy configuration from the +persistent per-rule JSON files under `${STATE_DIR}/network-/haproxy/` and +reloads haproxy inside the namespace. haproxy is stopped when no active rules +remain. + +### `restore-network` + +Batch-restore DHCP/DNS/metadata/services for all VMs on a network in a single +call. Invoked on network restart to rebuild all state at once instead of N +per-VM calls. + +``` +network-namespace-wrapper.sh restore-network \ + --network-id \ + [--gateway ] [--cidr ] [--dns ] \ + [--domain ] [--extension-ip ] [--vpc-id ] +``` + +The `restore_data` field in the payload is a JSON object (see +`buildRestoreNetworkData()` in `NetworkExtensionElement.java`). + +### `custom-action` + +``` +network-namespace-wrapper.sh custom-action \ + \ + +``` + +CloudStack now writes the custom-action request to a temporary JSON payload file +and passes that file to the wrapper script. The payload contains the network or +VPC identifiers, the action name, the caller-supplied action parameters, and +the extension detail blobs that used to be forwarded as individual CLI flags. + +Expected payload keys: + +| JSON key | Description | +|----------|-------------| +| `network_id` | Network ID for network-level actions | +| `vpc_id` | VPC ID for VPC-level actions | +| `action` | Custom action name | +| `action-params` | Caller-supplied JSON object for the action | +| `physical_network_extension_details` | Physical-network extension details JSON | +| `network_extension_details` | Per-network / per-VPC extension details JSON | + +Built-in actions: + +| Action | Description | +|--------|-------------| +| `reboot-device` | Bounces the guest veth pair (`vh--` down → up) | +| `dump-config` | Prints namespace IP addresses, iptables rules, and per-network state to stdout | +| `list-firewall-rules` | List iptables rules inside the namespace | +| `pbr-create-table` | Create or update a routing-table entry in `/etc/iproute2/rt_tables` | +| `pbr-delete-table` | Remove a routing-table entry from `/etc/iproute2/rt_tables` | +| `pbr-list-tables` | List non-comment routing-table entries from `/etc/iproute2/rt_tables` | +| `pbr-add-route` | Add/replace an `ip route` entry in a specific routing table inside the namespace | +| `pbr-delete-route` | Delete an `ip route` entry from a specific routing table inside the namespace | +| `pbr-list-routes` | List routes from one table (or all tables) inside the namespace | +| `pbr-add-rule` | Add an `ip rule` policy rule mapped to a specific routing table inside the namespace | +| `pbr-delete-rule` | Delete an `ip rule` policy rule mapped to a specific routing table inside the namespace | +| `pbr-list-rules` | List policy rules (or only rules for one table) inside the namespace | + +PBR action parameter keys (`action-params` JSON in the payload file): + +| Action | Required keys | Optional keys | +|--------|---------------|---------------| +| `pbr-create-table` | `table-id` (or `id`), `table-name` (or `table`) | — | +| `pbr-delete-table` | `table-id` or `table-name` | — | +| `pbr-list-tables` | — | — | +| `pbr-add-route` | `table`, `route` | — | +| `pbr-delete-route` | `table`, `route` | — | +| `pbr-list-routes` | — | `table` | +| `pbr-add-rule` | `table`, `rule` | — | +| `pbr-delete-rule` | `table`, `rule` | — | +| `pbr-list-rules` | — | `table` | + +Examples (equivalent to direct Linux commands): + +* `{"table-id":"100","table-name":"isp1"}` → `100 isp1` +* `{"table":"isp1","route":"default via 192.168.1.1 dev eth0"}` +* `{"table":"vpn1","route":"default dev wg0"}` +* `{"table":"isp1","rule":"from 10.10.1.0/24"}` +* `{"table":"vpn1","rule":"to 10.10.2.0/24"}` + +To add custom actions, place an executable script at +`${STATE_DIR}/hooks/custom-action-.sh` +(e.g. `/var/lib/cloudstack/network-namespace/hooks/custom-action-.sh`). +Unknown action names are delegated to the hook if present; otherwise the command +fails with a descriptive error. + +--- + +## Payload reference + +### Standard payload envelope + +```json +{ + "physical-network-extension-details": {}, + "network-extension-details": {}, + "payload": {} +} +``` + +For `custom-action`, `payload` is not nested; command fields are top-level. + +### Top-level extension details + +| Top-level key | Description | +|--------------|-------------| +| `physical-network-extension-details` | All `extension_resource_map_details` **plus** physical network metadata automatically added by `NetworkExtensionElement` (see table below). | +| `network-extension-details` | Per-network opaque JSON blob (selected host, namespace). | + +### Connection details (keys in `physical-network-extension-details`) + +These keys are explicitly set when calling `registerExtension`: + +| JSON key | Description | +|----------|-------------| +| `hosts` | Comma-separated list of candidate host IPs for HA selection | +| `host` | Single host IP (used when `hosts` is absent) | +| `port` | SSH port — default: `22` | +| `username` | SSH user — default: `root` | +| `password` | SSH password via `sshpass` — sensitive, not logged | +| `sshkey` | PEM-encoded SSH private key — sensitive, not logged; preferred over password | +| `guest.network.device` | Host NIC for guest (internal) traffic, e.g. `eth1` — defaults to `eth1` when absent | +| `public.network.device` | Host NIC for public (NAT/external) traffic, e.g. `eth1` — defaults to `eth1` when absent | + +This key is **automatically injected** by `NetworkExtensionElement` from the +physical network record: + +| JSON key | Description | +|----------|-------------| +| `physicalnetworkname` | Physical network name from CloudStack DB | + +The wrapper script uses `guest.network.device` (and `public.network.device`) to +name bridges as `br-` and veth pairs as `vh--` / +`vn--` (guest) and `vph--` / `vpn--` (public). + +### Per-network details (keys in `network-extension-details`) + +| JSON key | Description | +|----------|-------------| +| `host` | Previously selected host IP (set by `ensure-network-device`) | +| `namespace` | Linux network namespace name (e.g. `cs-net-` or `cs-vpc-`) | + +### Common keys inside `payload` (standard commands) + +#### Network-level fields + +| `payload` key | Description | +|--------------|-------------| +| `network_id` | Network ID — `CHOSEN_ID` for veth names is `` when VPC, else `` | +| `vlan` | Guest VLAN tag | +| `zone_id` | CloudStack zone ID | +| `guest_type` | Guest network type: `"isolated"`, `"shared"`, or `"l2"`. The wrapper uses this to skip iptables / NAT / public-veth operations for `shared` networks. | +| `network_state` | Guest network state: `"allocated"`, `"setup"`, `"implementing"`, `"implemented"`, `"shutdown"` or `"destroy"`. | +| `gateway` | Guest network gateway | +| `cidr` | Guest network CIDR | +| `vpc_id` | Present when the network belongs to a VPC; namespace becomes `cs-vpc-` | +| `network_ip6_gateway` | Guest IPv6 gateway address (assigned to the namespace veth), when configured | +| `network_ip6_cidr` | Guest IPv6 CIDR (e.g. `2001:db8:1::/64`), when configured | +| `extension_ip` | IP for DHCP/DNS/metadata service — equals gateway when SourceNat/Gateway is active, otherwise a dedicated placeholder IP | +| `dns` | Comma-separated IPv4 DNS server list | +| `dns6` | Comma-separated IPv6 DNS server list (advertised via RDNSS in radvd RA) | +| `domain` | Network domain suffix | +| `current_details` | `ensure-network-device` only — previous selected-device JSON, used to preserve host affinity | + +#### NIC-level fields + +| `payload` key | Description | +|--------------|-------------| +| `nic_id` | CloudStack numeric NIC ID | +| `nic_uuid` | NIC UUID — matches `external_ids:iface-id` written by the KVM agent | +| `mac` | VM NIC MAC address | +| `ip` | VM NIC IPv4 address | +| `gateway` | VM NIC IPv4 gateway | +| `netmask` | VM NIC IPv4 netmask | +| `default_nic` | `"false"` for secondary NICs (gateway DHCP option suppressed) | +| `device_id` | NIC device slot index | +| `ip6_address` | VM NIC IPv6 address, when configured | +| `ip6_gateway` | VM NIC IPv6 gateway, when available | +| `ip6_cidr` | VM NIC IPv6 CIDR, when available | + +#### Public-IP fields + +| `payload` key | Description | +|--------------|-------------| +| `public_ip` | Public IP address | +| `public_vlan` | Public IP VLAN tag | +| `public_gateway` | Gateway of the public IP segment | +| `public_cidr` | CIDR of the public IP | +| `source_nat` | `"true"` when this IP is the source-NAT IP | +| `private_ip` | VM private IP (NAT target) | + +### Action parameters (custom-action only) + +Custom-action parameters are embedded in the JSON payload file under +`action-params`. Hook scripts should read and decode the payload file directly +instead of expecting individual `--action-params` CLI arguments. + +Example payload excerpt: + +```json +{ + "action": "dump-config", + "network_id": "123", + "action-params": { + "key1": "value1", + "key2": "value2" + } +} +``` + +--- + +## Custom actions + +Define custom actions per extension via the CloudStack API: + +```bash +# Add a custom action to the extension +cmk addCustomAction \ + extensionid= \ + name=dump-config \ + description="Dump iptables rules and bridge state" \ + resourcetype=Network +``` + +Trigger the action on a network, optionally with parameters: +```bash +cmk runNetworkCustomAction \ + networkid= \ + actionid= \ + "parameters[0].key=threshold" "parameters[0].value=90" +``` + +### PBR custom-action examples + +```bash +# 1) Create action definitions (once per extension) +cmk addCustomAction extensionid= name=pbr-create-table resourcetype=Network +cmk addCustomAction extensionid= name=pbr-add-route resourcetype=Network +cmk addCustomAction extensionid= name=pbr-add-rule resourcetype=Network +cmk addCustomAction extensionid= name=pbr-list-tables resourcetype=Network +cmk addCustomAction extensionid= name=pbr-list-routes resourcetype=Network +cmk addCustomAction extensionid= name=pbr-list-rules resourcetype=Network +cmk addCustomAction extensionid= name=pbr-delete-rule resourcetype=Network +cmk addCustomAction extensionid= name=pbr-delete-route resourcetype=Network +cmk addCustomAction extensionid= name=pbr-delete-table resourcetype=Network + +# 2) Execute against a network +cmk runNetworkCustomAction networkid= actionid= \ + "parameters[0].key=table-id" "parameters[0].value=100" \ + "parameters[1].key=table-name" "parameters[1].value=isp1" + +cmk runNetworkCustomAction networkid= actionid= \ + "parameters[0].key=table" "parameters[0].value=isp1" \ + "parameters[1].key=route" "parameters[1].value=default via 192.168.1.1 dev eth0" + +cmk runNetworkCustomAction networkid= actionid= \ + "parameters[0].key=table" "parameters[0].value=isp1" \ + "parameters[1].key=rule" "parameters[1].value=from 10.10.1.0/24" +``` + +CloudStack calls `NetworkExtensionElement.runCustomAction()`, which issues: +```bash +network-namespace.sh custom-action \ + \ + +``` + +`network-namespace.sh` SSHes to the device and runs `network-namespace-wrapper.sh` +with the same ` ` shape. The wrapper +extracts `action`, `action-params`, and extension-details fields from the payload. + +--- + +## Developer / testing notes + +### VPC Support + +The extension now supports **VPC (Virtual Private Cloud)** networks in addition to +isolated networks. Key differences from isolated networks: + +* **Namespace sharing**: All tiers of a VPC share a single namespace (`cs-vpc-`) + instead of each network getting its own (`cs-net-`). +* **Host affinity**: All tiers of a VPC land on the same KVM host via stable hash-based + selection using the VPC ID as the routing key. +* **VPC-level operations**: `implement-vpc`, `update-vpc-source-nat-ip`, + `shutdown-vpc`, `destroy-vpc` commands + manage VPC-wide state (namespace creation/teardown). +* **VPC tier operations**: `implement-network`, `shutdown-network`, `destroy-network` + commands manage per-tier bridges and routes; the namespace is preserved across + tier lifecycle operations. + +### Integration tests + +The integration smoke test at +`test/integration/smoke/test_network_extension_namespace.py` +exercises the full lifecycle against real KVM hosts in the zone. + +``` +Management server + └── /usr/share/cloudstack-management/extensions// + └── network-namespace.sh ← deployed / referenced by test + SSHes to KVM host + runs network-namespace-wrapper.sh + +KVM host(s) in the zone + └── /etc/cloudstack/extensions// + └── network-namespace-wrapper.sh ← copied to KVM hosts by test setup + creates cs-net- or cs-vpc- namespaces + manages bridges, veth pairs, iptables, dnsmasq, haproxy, apache2 +``` + +The test covers: +* Create / list / update / delete external network device. +* Full network lifecycle: implement → assign-ip (source NAT) → static NAT → + port forwarding → firewall rules → DHCP/DNS → shutdown / destroy. +* VPC multi-tier networks with shared namespace and automatic host affinity. +* VPC source NAT IP update flow (`test_09_vpc_source_nat_ip_update`) including + source NAT flag flip from old public IP to new public IP. +* NSP state transitions: Disabled → Enabled → Disabled → Deleted. +* Tests `test_04`, `test_05`, `test_06` (DHCP, DNS, LB) require `arping`, + `dnsmasq`, and `haproxy` on the KVM hosts; the test skips them automatically + if these tools are not installed. +* Script cleanup on both management server and KVM hosts after each test. + +Run the test: +```bash +cd test/integration/smoke +python -m pytest test_network_extension_namespace.py \ + --with-marvin --marvin-config= \ + -s -a 'tags=advanced,smoke' 2>&1 | tee /tmp/extnet-test.log +``` + +**Prerequisites on KVM hosts:** +* `iproute2` (`ip`, `ip netns`) +* `iptables` + `iptables-save` +* `arping` (for GARP on IP assignment) +* `dnsmasq` (DHCP + DNS — required for `test_04` / DNS tests) +* `haproxy` (LB — required for `test_05` / LB tests) +* `apache2` / `httpd` (metadata HTTP service — required for UserData tests) +* `python3` (vm-data processing, haproxy config generation) +* `util-linux` (`flock`) (lock serialization) +* SSH access from management server (root or sudo-capable user) + +**Prerequisites on the Marvin / test runner node:** +* Python Marvin library installed (`pip install -r requirements.txt`) +* A valid Marvin config file pointing to the CloudStack environment +* The test runner must be able to SSH to the management server and to KVM hosts diff --git a/Network-Namespace/network-namespace-wrapper.sh b/Network-Namespace/network-namespace-wrapper.sh new file mode 100755 index 0000000..027a526 --- /dev/null +++ b/Network-Namespace/network-namespace-wrapper.sh @@ -0,0 +1,4308 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +############################################################################## +# network-namespace-wrapper.sh +# +# Network Extension wrapper script for Apache CloudStack. +# Runs on a KVM host; manages Linux network namespaces, VLAN bridges, +# and iptables rules to implement isolated guest networks. +# +# Architecture +# ============ +# +# Guest (internal) side +# --------------------- +# Each CloudStack isolated network has a VLAN. On the KVM host: +# +# ethX. – VLAN sub-interface on the physical NIC (ethX) +# br- – Linux bridge: ethX. + vh-- +# vh-- – host end of the veth pair → in the bridge +# vn-- – namespace end → assigned the extension IP +# (= gateway when SourceNat/Gateway is enabled; +# = allocated placeholder IP otherwise) +# +# ethX is read from guest.network.device in the physical-network extension +# details (defaults to eth1 when absent). +# +# Namespace +# --------- +# Isolated network : cs-net- (--network-id) +# VPC network : cs-vpc- (--vpc-id) +# +# Public (NAT) side +# ----------------- +# For each public IP assigned to the network a veth pair is created: +# +# vph-- – host end → added to br- +# vpn-- – namespace end → assigned the public IP +# +# where = public VLAN tag (from --public-vlan) +# = vpc-id if present, else network-id +# pub_ethX = read from public.network.device in extension details +# (defaults to eth1 when absent) +# +# Interface name lengths (Linux limit: 15 chars) +# vh-- max 15 (shorten_id applied when needed) ✓ +# vn-- max 15 (shorten_id applied when needed) ✓ +# vph-- max 15 ✓ +# vpn-- max 15 ✓ +# +# iptables chains (inside namespace) +# ------------------------------------ +# CS_EXTNET__PR – PREROUTING DNAT chain +# CS_EXTNET__POST – POSTROUTING SNAT chain +# CS_EXTNET_FWD_ – FORWARD filter chain +# +# Invocation (forwarded by network-namespace.sh): +# network-namespace-wrapper.sh +# +# Standard payload envelope includes top-level: +# physical-network-extension-details +# network-extension-details +# payload # command-specific keys +# +# For custom-action the payload is flat (command-specific keys are top-level). +############################################################################## + +set -e + +# --------------------------------------------------------------------------- +# Derive log path from this wrapper's own directory name so that each +# renamed deployment writes to its own log file. +# /etc/cloudstack/extensions//-wrapper.sh +# → /var/log/cloudstack/extensions//.log +# --------------------------------------------------------------------------- +_WRAPPER_SELF="$(readlink -f "$0" 2>/dev/null \ + || realpath "$0" 2>/dev/null \ + || echo "$0")" +_WRAPPER_EXT_DIR="$(basename "$(dirname "${_WRAPPER_SELF}")")" +LOG_FILE="/var/log/cloudstack/extensions/${_WRAPPER_EXT_DIR}/${_WRAPPER_EXT_DIR}.log" +mkdir -p "$(dirname "${LOG_FILE}")" 2>/dev/null || true + +STATE_DIR="/var/lib/cloudstack/${_WRAPPER_EXT_DIR}" + +# --------------------------------------------------------------------------- +# JSON helpers (no jq dependency) +# --------------------------------------------------------------------------- + +_json_get() { + # _json_get → value (unquoted string) or empty + printf '%s' "$1" | grep -o "\"$2\":\"[^\"]*\"" | cut -d'"' -f4 || true +} + +_payload_json_get() { + # _payload_json_get -> value or compact JSON for objects + python3 - "$1" "$2" <<'PY' +import json, sys +with open(sys.argv[1], encoding='utf-8') as fh: + data = json.load(fh) +cur = data +for part in sys.argv[2].split('.'): + if isinstance(cur, dict): + cur = cur.get(part) + else: + cur = None + if cur is None: + break +if cur is None: + print("") +elif isinstance(cur, (dict, list)): + print(json.dumps(cur, separators=(",", ":"))) +else: + print(str(cur)) +PY +} + +# --------------------------------------------------------------------------- +# Pre-scan all arguments for the two JSON blobs. +# --------------------------------------------------------------------------- + +PHYS_DETAILS="${CS_PHYSICAL_NETWORK_EXTENSION_DETAILS:-{}}" +EXTENSION_DETAILS="${CS_NETWORK_EXTENSION_DETAILS:-{}}" + +if [ $# -ge 3 ] && [ -f "$2" ]; then + PHYS_DETAILS=$(_payload_json_get "$2" "physical-network-extension-details") + EXTENSION_DETAILS=$(_payload_json_get "$2" "network-extension-details") +fi + +_pre_scan_args() { + local i=1 + local args=("$@") + while [ $i -le $# ]; do + case "${args[$i-1]}" in + --physical-network-extension-details) + PHYS_DETAILS="${args[$i]:-{}}" + i=$((i+2)) ;; + --network-extension-details) + EXTENSION_DETAILS="${args[$i]:-{}}" + i=$((i+2)) ;; + *) i=$((i+1)) ;; + esac + done +} + +_pre_scan_args "$@" + +# --------------------------------------------------------------------------- +# Resolve host network interfaces from physical-network extension details. +# +# Register guest.network.device and public.network.device when attaching the +# extension to a physical network, e.g.: +# details[N].key=guest.network.device details[N].value=eth1 +# details[M].key=public.network.device details[M].value=eth1 +# +# Both default to eth1 when absent. +# --------------------------------------------------------------------------- + +GUEST_ETH=$(_json_get "${PHYS_DETAILS}" "guest.network.device") +GUEST_ETH="${GUEST_ETH:-eth1}" + +PUB_ETH=$(_json_get "${PHYS_DETAILS}" "public.network.device") +PUB_ETH="${PUB_ETH:-eth1}" + +# iptables chain prefix +CHAIN_PREFIX="CS_EXTNET" + +############################################################################## +# Helpers +############################################################################## + +log() { + local ts + ts=$(date '+%Y-%m-%d %H:%M:%S') + echo "[${ts}] $*" >> "${LOG_FILE}" 2>/dev/null || true +} + +# --------------------------------------------------------------------------- +# State directory helpers +# +# Per-network state (unique to each tier / isolated network): +# ${STATE_DIR}/network-/ +# +# VPC-wide shared state (public IPs, namespace, iptables dump): +# ${STATE_DIR}/vpc-/ (VPC network) +# ${STATE_DIR}/network-/ (isolated — same as _net_state_dir) +# +# Call these only after parse_args has set NETWORK_ID / VPC_ID. +# --------------------------------------------------------------------------- +_net_state_dir() { echo "${STATE_DIR}/network-${NETWORK_ID}"; } + +_vpc_state_dir() { + if [ -n "${VPC_ID}" ]; then + echo "${STATE_DIR}/vpc-${VPC_ID}" + else + echo "${STATE_DIR}/network-${NETWORK_ID}" + fi +} + +# Save the current iptables-save output to the log and to +# /iptables.dump (full namespace snapshot). +# Call this after any command that modifies iptables rules. +_dump_iptables() { + local ns="${1:-${NAMESPACE}}" + [ -z "${ns}" ] && return + local ts + ts=$(date '+%Y-%m-%d %H:%M:%S') + local ipt_out + ipt_out=$(ip netns exec "${ns}" iptables-save 2>/dev/null || echo "(iptables-save failed)") + { + printf '\n=== iptables-save [%s] ns=%s ===\n' "${ts}" "${ns}" + printf '%s\n' "${ipt_out}" + printf '=== end iptables-save ===\n\n' + } >> "${LOG_FILE}" 2>/dev/null || true + # Persist a current snapshot under the VPC/network state dir. + if [ -n "${NETWORK_ID}" ]; then + local dump_dir; dump_dir=$(_vpc_state_dir) + mkdir -p "${dump_dir}" 2>/dev/null || true + printf '# iptables-save %s ns=%s\n%s\n' "${ts}" "${ns}" "${ipt_out}" \ + > "${dump_dir}/iptables.dump" 2>/dev/null || true + fi +} + +die() { + log "ERROR: $*" + release_lock + exit 1 +} + +ensure_dirs() { + mkdir -p "${STATE_DIR}" "$(dirname "${LOG_FILE}")" 2>/dev/null || true +} + +acquire_lock() { + local network_id="$1" + # For VPC networks serialize on the VPC ID so all tiers sharing the same + # namespace are protected by a single lock. + local lockfile + if [ -n "${VPC_ID:-}" ]; then + lockfile="${STATE_DIR}/lock-vpc-${VPC_ID}" + else + lockfile="${STATE_DIR}/lock-network-${network_id}" + fi + mkdir -p "${STATE_DIR}" + # Record the current lockfile path so release_lock can remove it later + GLOBAL_LOCKFILE="${lockfile}" + log "acquire_lock: attempting to acquire lock ${GLOBAL_LOCKFILE}" + exec 200>"${lockfile}" + flock -w 30 200 || die "Failed to acquire lock for network ${network_id}" +} + +release_lock() { + # Close the fd holding the flock (releases the lock) then remove the + # lockfile path if we have it recorded. It's OK if the file no longer + # exists; we ignore errors. + exec 200>&- 2>/dev/null || true + if [ -n "${GLOBAL_LOCKFILE:-}" ]; then + log "release_lock: releasing and removing ${GLOBAL_LOCKFILE}" + rm -f "${GLOBAL_LOCKFILE}" 2>/dev/null || true + GLOBAL_LOCKFILE="" + else + log "release_lock: no GLOBAL_LOCKFILE recorded" + fi +} + +# Ensure we attempt to release any held lock on exit (safe no-op if none). +trap 'release_lock' EXIT + +# --------------------------------------------------------------------------- +# Interface / bridge name helpers +# --------------------------------------------------------------------------- + +# Host bridge for a VLAN on a given physical NIC: br- +host_bridge_name() { + local eth="$1" vlan_raw="$2" vlan + vlan=$(normalize_vlan "${vlan_raw}") + echo "br${eth}-${vlan}" +} + +# Internal guest veth pair (keyed on VLAN ID and network id): +# vh-- (host, in bridge) +# vn-- (namespace, gets gateway IP) + +# shorten_id -> short deterministic id portion suitable for interface names +shorten_id() { + local id="$1" + [ -z "${id}" ] && echo "" && return + # If purely numeric, use hex to shorten (stable) + if printf '%d' "${id}" >/dev/null 2>&1; then + printf '%x' "${id}" + return + fi + # For non-numeric, prefer an md5 prefix if available + if command -v md5sum >/dev/null 2>&1; then + echo -n "${id}" | md5sum | awk '{print $1}' | cut -c1-6 + return + fi + # Fallback: last 6 chars + echo "${id}" | awk '{n=length($0); print substr($0, n-5)}' +} + +# normalize_vlan -> prints the normalized vlan id (strip vlan:// prefix) +normalize_vlan() { + local vlan_raw="$1" + if [ -z "${vlan_raw}" ]; then + printf '%s' "" + return + fi + if printf '%s' "${vlan_raw}" | grep -q '^vlan://'; then + printf '%s' "${vlan_raw#vlan://}" + else + printf '%s' "${vlan_raw}" + fi +} + +# Generate guest host veth name: vh-- (ensure <=15 chars) +veth_host_name() { + local vlan_raw="$1" id="$2" name short + vlan=$(normalize_vlan "${vlan_raw}") + name="vh-${vlan}-${id}" + if [ ${#name} -le 15 ]; then + echo "${name}" + return + fi + short=$(shorten_id "${id}") + name="vh-${vlan}-${short}" + if [ ${#name} -le 15 ]; then + echo "${name}" + return + fi + echo "${name:0:15}" +} + +# Generate guest namespace veth name: vn-- (ensure <=15 chars) +veth_ns_name() { + local vlan_raw="$1" id="$2" name short + vlan=$(normalize_vlan "${vlan_raw}") + name="vn-${vlan}-${id}" + if [ ${#name} -le 15 ]; then + echo "${name}" + return + fi + short=$(shorten_id "${id}") + name="vn-${vlan}-${short}" + if [ ${#name} -le 15 ]; then + echo "${name}" + return + fi + echo "${name:0:15}" +} + +# Public veth pair (keyed on public VLAN and network/vpc id): +# vph-- (host, in public bridge) +# vpn-- (namespace, gets public IP) +pub_veth_host_name() { + local pvlan_raw="$1" id="$2" pvlan + pvlan=$(normalize_vlan "${pvlan_raw}") + echo "vph-${pvlan}-${id}" +} + +pub_veth_ns_name() { + local pvlan_raw="$1" id="$2" pvlan + pvlan=$(normalize_vlan "${pvlan_raw}") + echo "vpn-${pvlan}-${id}" +} + +nat_chain() { echo "${CHAIN_PREFIX}_${1}"; } +filter_chain() { echo "${CHAIN_PREFIX}_FWD_${1}"; } +firewall_chain() { echo "${CHAIN_PREFIX}_FWRULES_${1}"; } +acl_chain() { echo "${CHAIN_PREFIX}_ACL_${1}"; } + +ensure_public_ip_on_namespace() { + local public_ip="$1" public_cidr="$2" pveth_n="$3" pveth_h="$4" addr_spec prefix + + if [ -n "${public_cidr}" ] && echo "${public_cidr}" | grep -q '/'; then + prefix=$(echo "${public_cidr}" | cut -d'/' -f2) + addr_spec="${public_ip}/${prefix}" + else + addr_spec="${public_ip}/32" + fi + + # Do not add a second address when the IP is already present with another prefix. + ip netns exec "${NAMESPACE}" ip addr show "${pveth_n}" 2>/dev/null | grep -q "${public_ip}/" || \ + ip netns exec "${NAMESPACE}" ip addr add "${addr_spec}" dev "${pveth_n}" 2>/dev/null || true + + # Keep the host route in place for inbound traffic. + ip route show | grep -q "^${public_ip}" || \ + ip route add "${public_ip}/32" dev "${pveth_h}" 2>/dev/null || true + + # Disable IPv6 on the public veth interface inside namespace to prevent + # IPv6 autoconf and link-local addresses appearing. Idempotent. + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf."${pveth_n}".disable_ipv6=1 >/dev/null 2>&1 || true +} + +# --------------------------------------------------------------------------- +# ensure_host_bridge +# Idempotently creates br- with . as a member. +# Prints the bridge name. +# --------------------------------------------------------------------------- + +ensure_host_bridge() { + local eth="$1" + local vlan_raw="$2" + local vlan=$(normalize_vlan "${vlan_raw}") + local br vif current_master + + br=$(host_bridge_name "${eth}" "${vlan}") + vif="${eth}.${vlan}" + + # VLAN sub-interface + if ! ip link show "${vif}" >/dev/null 2>&1; then + if ! ip link add link "${eth}" name "${vif}" type vlan id "${vlan}" 2>/dev/null; then + if [ "${NETWORK_STATE:-}" = "shutdown" ]; then + log "ensure_host_bridge: failed to create ${vif} (network_state=shutdown, ignoring)" + echo "${br}"; return 0 + fi + ip link add link "${eth}" name "${vif}" type vlan id "${vlan}" + fi + log "Created VLAN interface ${vif}" + fi + ip link set "${vif}" up 2>/dev/null || true + + # Bridge + if ! ip link show "${br}" >/dev/null 2>&1; then + if ! ip link add name "${br}" type bridge 2>/dev/null; then + if [ "${NETWORK_STATE:-}" = "shutdown" ]; then + log "ensure_host_bridge: failed to create ${br} (network_state=shutdown, ignoring)" + echo "${br}"; return 0 + fi + ip link add name "${br}" type bridge + fi + ip link set "${br}" up 2>/dev/null || true + log "Created host bridge ${br}" + fi + + # Attach VLAN interface to bridge (if not already there) + current_master=$(ip link show "${vif}" 2>/dev/null | grep -o 'master [^ ]*' | awk '{print $2}' || true) + if [ "${current_master}" != "${br}" ]; then + ip link set "${vif}" master "${br}" 2>/dev/null || true + fi + + echo "${br}" +} + +# teardown_host_bridge_if_unused +# Counterpart to ensure_host_bridge(): removes the VLAN sub-interface (ethX.vlan) +# and the bridge (br-) it created, but ONLY if no VM (or other +# consumer, e.g. another VPC tier/network/tenant riding the same public VLAN) +# still has an interface plugged into the bridge. +# +# ensure_host_bridge() always enslaves the VLAN uplink (ethX.vlan) into the +# bridge, so it is always present as a bridge member — it must be excluded +# from the "is anything still using this bridge" check, otherwise the bridge +# would never be considered unused. +# +# Safe to call unconditionally from any teardown path — it is a no-op +# whenever the bridge is still in use (by a VM tap or otherwise), or already +# gone. +teardown_host_bridge_if_unused() { + local eth="$1" vlan_raw="$2" vlan br vif members + vlan=$(normalize_vlan "${vlan_raw}") + br=$(host_bridge_name "${eth}" "${vlan}") + vif="${eth}.${vlan}" + + if ! ip link show "${br}" >/dev/null 2>&1; then + return 0 + fi + + if [ -d "/sys/class/net/${br}/brif" ]; then + members=$(ls -A "/sys/class/net/${br}/brif" 2>/dev/null | grep -v -x "${vif}" || true) + if [ -n "${members}" ]; then + log "teardown_host_bridge_if_unused: ${br} still has member interfaces (${members}), leaving in place" + return 0 + fi + fi + + ip link del "${br}" 2>/dev/null || true + log "Removed host bridge ${br}" + + if ip link show "${vif}" >/dev/null 2>&1; then + ip link del "${vif}" 2>/dev/null || true + log "Removed VLAN interface ${vif}" + fi +} + +# _guard_ns_teardown +# When network_state is "shutdown", "destroy", or "allocated" and the namespace +# is already gone, exit successfully — the network has been torn down or was +# never implemented on this host. +# Call this after acquire_lock in any command that does namespace operations. +_guard_ns_teardown() { + case "${NETWORK_STATE:-}" in + shutdown|destroy|allocated) ;; + *) return 0 ;; + esac + ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b" && return 0 + log "${1:-command}: namespace ${NAMESPACE} not found (network_state=${NETWORK_STATE}) — treating as success" + release_lock + exit 0 +} + +ensure_chain() { + local table="$1" chain="$2" + ip netns exec "${NAMESPACE}" iptables -t "${table}" -n -L "${chain}" >/dev/null 2>&1 || \ + ip netns exec "${NAMESPACE}" iptables -t "${table}" -N "${chain}" +} + +ensure_jump() { + local table="$1" parent="$2" chain="$3" + ip netns exec "${NAMESPACE}" iptables -t "${table}" \ + -C "${parent}" -j "${chain}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t "${table}" \ + -I "${parent}" 1 -j "${chain}" +} + +############################################################################## +# Parse common arguments +############################################################################## + +parse_args() { + local payload_file="$1" + + NETWORK_ID=$(_payload_json_get "${payload_file}" "payload.network_id") + GUEST_TYPE=$(_payload_json_get "${payload_file}" "payload.guest_type") + ZONE_ID=$(_payload_json_get "${payload_file}" "payload.zone_id") + VPC_ID=$(_payload_json_get "${payload_file}" "payload.vpc_id") + VLAN=$(_payload_json_get "${payload_file}" "payload.vlan") + GATEWAY=$(_payload_json_get "${payload_file}" "payload.gateway") + CIDR=$(_payload_json_get "${payload_file}" "payload.cidr") + PUBLIC_IP=$(_payload_json_get "${payload_file}" "payload.public_ip") + PRIVATE_IP=$(_payload_json_get "${payload_file}" "payload.private_ip") + PUBLIC_PORT=$(_payload_json_get "${payload_file}" "payload.public_port") + PRIVATE_PORT=$(_payload_json_get "${payload_file}" "payload.private_port") + PROTOCOL=$(_payload_json_get "${payload_file}" "payload.protocol") + SOURCE_NAT=$(_payload_json_get "${payload_file}" "payload.source_nat") + PUBLIC_GATEWAY=$(_payload_json_get "${payload_file}" "payload.public_gateway") + PUBLIC_CIDR=$(_payload_json_get "${payload_file}" "payload.public_cidr") + PUBLIC_VLAN=$(_payload_json_get "${payload_file}" "payload.public_vlan") + MAC=$(_payload_json_get "${payload_file}" "payload.mac") + HOSTNAME=$(_payload_json_get "${payload_file}" "payload.hostname") + DNS_SERVER=$(_payload_json_get "${payload_file}" "payload.dns") + NIC_ID=$(_payload_json_get "${payload_file}" "payload.nic_id") + NIC_UUID=$(_payload_json_get "${payload_file}" "payload.nic_uuid") + NETMASK=$(_payload_json_get "${payload_file}" "payload.netmask") + DEVICE_ID=$(_payload_json_get "${payload_file}" "payload.device_id") + VM_IP=$(_payload_json_get "${payload_file}" "payload.ip") + USERDATA=$(_payload_json_get "${payload_file}" "payload.userdata") + PASSWORD=$(_payload_json_get "${payload_file}" "payload.password") + SSH_KEY=$(_payload_json_get "${payload_file}" "payload.sshkey") + HYPERVISOR_HOSTNAME=$(_payload_json_get "${payload_file}" "payload.hypervisor_hostname") + LB_RULES_JSON=$(_payload_json_get "${payload_file}" "payload.lb_rules") + DEFAULT_NIC=$(_payload_json_get "${payload_file}" "payload.default_nic") + VM_DATA=$(_payload_json_get "${payload_file}" "payload.vm_data") + DOMAIN=$(_payload_json_get "${payload_file}" "payload.domain") + EXTENSION_IP=$(_payload_json_get "${payload_file}" "payload.extension_ip") + RESTORE_DATA=$(_payload_json_get "${payload_file}" "payload.restore_data") + FW_RULES_JSON=$(_payload_json_get "${payload_file}" "payload.fw_rules") + ACL_RULES_JSON=$(_payload_json_get "${payload_file}" "payload.acl_rules") + NETWORK_STATE=$(_payload_json_get "${payload_file}" "payload.network_state") + NETWORK_IP6_GATEWAY=$(_payload_json_get "${payload_file}" "payload.network_ip6_gateway") + NETWORK_IP6_CIDR=$(_payload_json_get "${payload_file}" "payload.network_ip6_cidr") + DNS6_SERVER=$(_payload_json_get "${payload_file}" "payload.dns6") + NIC_IP6_ADDRESS=$(_payload_json_get "${payload_file}" "payload.ip6_address") + + [ -z "${SOURCE_NAT}" ] && SOURCE_NAT="false" + [ -z "${LB_RULES_JSON}" ] && LB_RULES_JSON="[]" + [ -z "${DEFAULT_NIC}" ] && DEFAULT_NIC="true" + + [ -z "${NETWORK_ID}" ] && die "Missing --network-id" + + # Namespace: VPC → cs-vpc-; standalone → cs-net- + if [ -n "${VPC_ID}" ]; then + NAMESPACE="cs-vpc-${VPC_ID}" + else + local NS_FROM_DETAILS + NS_FROM_DETAILS=$(_json_get "${EXTENSION_DETAILS}" "namespace") + NAMESPACE="${NS_FROM_DETAILS:-cs-net-${NETWORK_ID}}" + fi + + # CHOSEN_ID selects vpc-id when present, otherwise network-id + CHOSEN_ID="${VPC_ID:-${NETWORK_ID}}" + + # Normalize VLAN if provided as 'vlan://' (common in some callers) + if [ -n "${VLAN}" ]; then + VLAN=$(normalize_vlan "${VLAN}") + fi + # Normalize PUBLIC_VLAN as well + if [ -n "${PUBLIC_VLAN}" ]; then + PUBLIC_VLAN=$(normalize_vlan "${PUBLIC_VLAN}") + fi +} + +# Load persisted state (shutdown, destroy, IP operations). +# Reads from the new network-/vpc- layout first; falls back to the +# legacy ${STATE_DIR}/${NETWORK_ID} path for backward compatibility. +_load_state() { + local nsd; nsd=$(_net_state_dir) + local vsd; vsd=$(_vpc_state_dir) + local old="${STATE_DIR}/${NETWORK_ID}" # legacy path + + _read_sf() { + # _read_sf [...] + # Sets from the first dir that contains . + local _var="$1" _file="$2"; shift 2 + eval "local _cur=\${${_var}}" + [ -n "${_cur}" ] && return + local _d + for _d in "$@"; do + if [ -f "${_d}/${_file}" ]; then + eval "${_var}=\$(cat '${_d}/${_file}')" + return + fi + done + } + + _read_sf VLAN vlan "${nsd}" "${old}" + _read_sf CIDR cidr "${nsd}" "${old}" + _read_sf GATEWAY gateway "${nsd}" "${old}" + _read_sf EXTENSION_IP extension-ip "${nsd}" "${old}" + _read_sf NETWORK_IP6_GATEWAY ip6-gateway "${nsd}" "${old}" + _read_sf NETWORK_IP6_CIDR ip6-cidr "${nsd}" "${old}" + _read_sf DNS6_SERVER dns6 "${nsd}" "${old}" + + if [ -z "${NAMESPACE}" ]; then + # Namespace is VPC-wide: check vpc state dir, then per-net, then legacy. + _read_sf NAMESPACE namespace "${vsd}" "${nsd}" "${old}" + if [ -z "${NAMESPACE}" ]; then + local NS_FROM_DETAILS + NS_FROM_DETAILS=$(_json_get "${EXTENSION_DETAILS}" "namespace") + if [ -n "${VPC_ID}" ]; then + NAMESPACE="${NS_FROM_DETAILS:-cs-vpc-${VPC_ID}}" + else + NAMESPACE="${NS_FROM_DETAILS:-cs-net-${NETWORK_ID}}" + fi + fi + fi + # CHOSEN_ID will be set by parse_args; do not read or persist legacy network_or_vpc_id +} + +############################################################################## +# Command: implement-network +# +# 1. Create namespace cs-net- +# 2. Create host bridge br- with ethX. sub-interface +# 3. Create veth pair: veth-host- (host, in bridge) / veth-ns- (namespace) +# 4. Assign gateway IP to veth-ns- inside namespace +# 5. Set up iptables chains inside namespace +############################################################################## + +cmd_implement_network() { + parse_args "$@" + acquire_lock "${NETWORK_ID}" + + log "implement-network: network=${NETWORK_ID} ns=${NAMESPACE} vlan=${VLAN} gw=${GATEWAY} cidr=${CIDR}" + + local veth_h veth_n nchain_pr nchain_post fchain + veth_h=$(veth_host_name "${VLAN}" "${CHOSEN_ID}") + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" + nchain_post="${CHAIN_PREFIX}_${NETWORK_ID}_POST" + fchain=$(filter_chain "${NETWORK_ID}") + + # ---- 1. Create namespace ---- + if ! ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns add "${NAMESPACE}" + log "Created namespace ${NAMESPACE}" + fi + ip netns exec "${NAMESPACE}" ip link set lo up 2>/dev/null || true + # Ensure per-namespace iproute2 rt_tables for PBR isolation + _pbr_ensure_table_file + + # IPv6: enable when a guest IPv6 gateway is configured; otherwise disable + # globally for standalone networks (VPC tiers only disable per-interface to + # avoid clobbering IPv6 state set by sibling tiers in the same namespace). + if [ -n "${NETWORK_IP6_GATEWAY}" ]; then + _enable_ipv6_in_namespace "${veth_n}" + log "implement-network: IPv6 enabled in namespace ${NAMESPACE}" + elif [ -z "${VPC_ID}" ]; then + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.all.disable_ipv6=1 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.default.disable_ipv6=1 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.lo.disable_ipv6=1 >/dev/null 2>&1 || true + fi + + # ---- 2. Host bridge + VLAN sub-interface ---- + if [ -n "${VLAN}" ]; then + ensure_host_bridge "${GUEST_ETH}" "${VLAN}" + local br + br=$(host_bridge_name "${GUEST_ETH}" "${VLAN}") + + # ---- 3. Guest veth pair ---- + if ! ip link show "${veth_h}" >/dev/null 2>&1; then + ip link add "${veth_h}" type veth peer name "${veth_n}" + ip link set "${veth_n}" netns "${NAMESPACE}" + ip link set "${veth_h}" master "${br}" + ip link set "${veth_h}" up + ip netns exec "${NAMESPACE}" ip link set "${veth_n}" up + log "Created guest veth ${veth_h} (host→${br}) <-> ${veth_n} (namespace)" + else + ip link set "${veth_h}" up 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set "${veth_n}" up 2>/dev/null || true + fi + fi + + # ---- 4. Assign extension IP to namespace veth ---- + # EXTENSION_IP is the IP that this namespace "owns" on the guest subnet. + # When SourceNat/Gateway service is enabled this equals the network gateway; + # otherwise it is a dedicated placeholder IP allocated for DHCP/DNS/UserData. + # If --extension-ip was not supplied fall back to the gateway. + local ext_ip; ext_ip="${EXTENSION_IP:-${GATEWAY}}" + if [ -n "${ext_ip}" ] && [ -n "${CIDR}" ]; then + local prefix + prefix=$(echo "${CIDR}" | cut -d'/' -f2) + ip netns exec "${NAMESPACE}" ip addr show "${veth_n}" 2>/dev/null | \ + grep -q "${ext_ip}/${prefix}" || \ + ip netns exec "${NAMESPACE}" ip addr add "${ext_ip}/${prefix}" dev "${veth_n}" + log "Assigned ${ext_ip}/${prefix} to ${veth_n} in ${NAMESPACE}" + + # When the extension IP differs from the gateway the namespace is not + # the default router for the guest subnet. Add a default route toward + # the actual gateway so the namespace can reach external destinations + # (e.g. metadata proxy, outbound health-checks). + if [ -n "${GATEWAY}" ] && [ "${ext_ip}" != "${GATEWAY}" ]; then + ip netns exec "${NAMESPACE}" ip route replace default \ + via "${GATEWAY}" dev "${veth_n}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" ip route add default \ + via "${GATEWAY}" dev "${veth_n}" 2>/dev/null || true + log "Default route in ${NAMESPACE}: via ${GATEWAY} dev ${veth_n}" + fi + fi + + # ---- 4b. Assign IPv6 gateway address to the guest veth ---- + if [ -n "${NETWORK_IP6_GATEWAY}" ] && [ -n "${NETWORK_IP6_CIDR}" ]; then + local ip6_prefix + ip6_prefix=$(echo "${NETWORK_IP6_CIDR}" | cut -d'/' -f2) + ip netns exec "${NAMESPACE}" ip -6 addr show "${veth_n}" 2>/dev/null | \ + grep -q "${NETWORK_IP6_GATEWAY}/" || \ + ip netns exec "${NAMESPACE}" ip -6 addr add \ + "${NETWORK_IP6_GATEWAY}/${ip6_prefix}" dev "${veth_n}" 2>/dev/null || true + log "implement-network: assigned IPv6 ${NETWORK_IP6_GATEWAY}/${ip6_prefix} to ${veth_n}" + else + # No IPv6 on this interface; disable per-interface to suppress link-local + # autoconf. For VPC tiers skip global disable to avoid breaking sibling + # tiers that may have IPv6 enabled. + ip netns exec "${NAMESPACE}" sysctl -w \ + net.ipv6.conf."${veth_n}".disable_ipv6=1 >/dev/null 2>&1 || true + fi + + if [ "${GUEST_TYPE}" = "isolated" ]; then + # ---- 5. IP forwarding (needed for Isolated networks only) ---- + ip netns exec "${NAMESPACE}" sysctl -w net.ipv4.ip_forward=1 >/dev/null 2>&1 || true + + # ---- 6. iptables chains (needed for Isolated networks only) ---- + ensure_chain nat "${nchain_pr}" + ensure_chain nat "${nchain_post}" + ensure_chain filter "${fchain}" + ensure_jump nat PREROUTING "${nchain_pr}" + ensure_jump nat POSTROUTING "${nchain_post}" + ensure_jump filter FORWARD "${fchain}" + + # Allow forwarding for guest traffic in/out of veth-ns- + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C "${fchain}" -i "${veth_n}" -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A "${fchain}" -i "${veth_n}" -j ACCEPT + + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C "${fchain}" -o "${veth_n}" -m state --state RELATED,ESTABLISHED -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A "${fchain}" -o "${veth_n}" -m state --state RELATED,ESTABLISHED -j ACCEPT + fi + + # ---- 7. Persist state ---- + # Per-network state → network-/ + local nsd; nsd=$(_net_state_dir) + mkdir -p "${nsd}" + echo "${VLAN}" > "${nsd}/vlan" + echo "${GATEWAY}" > "${nsd}/gateway" + echo "${CIDR}" > "${nsd}/cidr" + echo "${ext_ip}" > "${nsd}/extension-ip" + if [ -n "${NETWORK_IP6_GATEWAY}" ]; then + echo "${NETWORK_IP6_GATEWAY}" > "${nsd}/ip6-gateway" + fi + if [ -n "${NETWORK_IP6_CIDR}" ]; then + echo "${NETWORK_IP6_CIDR}" > "${nsd}/ip6-cidr" + fi + if [ -n "${DNS6_SERVER}" ]; then + echo "${DNS6_SERVER}" > "${nsd}/dns6" + fi + + # Namespace + VPC tier tracking → vpc-/ (or network-/ for isolated) + local vsd; vsd=$(_vpc_state_dir) + mkdir -p "${vsd}" + echo "${NAMESPACE}" > "${vsd}/namespace" + # Register this network as an active tier under the VPC (for destroy tracking) + if [ -n "${VPC_ID}" ]; then + mkdir -p "${vsd}/tiers" + touch "${vsd}/tiers/${NETWORK_ID}" + fi + + _dump_iptables "${NAMESPACE}" + release_lock + log "implement-network: done network=${NETWORK_ID} namespace=${NAMESPACE}" +} + +############################################################################## +# Command: shutdown-network +# Flush iptables chains and remove this network's veth pairs. +# For VPC networks the shared namespace is preserved (other tiers still use it). +# For isolated networks the namespace is also removed. +############################################################################## + +cmd_shutdown_network() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + log "shutdown-network: network=${NETWORK_ID} ns=${NAMESPACE} guest_type=${GUEST_TYPE} vpc=${VPC_ID}" + + # iptables NAT/FILTER chains only exist for non-Shared networks (Shared + # networks have no routing/NAT role; chains are never created for them). + if [ "${GUEST_TYPE}" = "isolated" ]; then + local nchain_pr nchain_post fchain + nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" + nchain_post="${CHAIN_PREFIX}_${NETWORK_ID}_POST" + fchain=$(filter_chain "${NETWORK_ID}") + + # Remove iptables chain jumps for this network + ip netns exec "${NAMESPACE}" iptables -t nat -D PREROUTING -j "${nchain_pr}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t nat -D POSTROUTING -j "${nchain_post}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter -D FORWARD -j "${fchain}" 2>/dev/null || true + + # Flush and delete this network's chains + ip netns exec "${NAMESPACE}" iptables -t nat -F "${nchain_pr}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t nat -X "${nchain_pr}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t nat -F "${nchain_post}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t nat -X "${nchain_post}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter -F "${fchain}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter -X "${fchain}" 2>/dev/null || true + fi + + # Public veth pairs and NAT state only exist for non-Shared networks. + local vsd; vsd=$(_vpc_state_dir) + if [ "${GUEST_TYPE}" = "isolated" ]; then + # Remove public veth pairs owned by THIS tier only (guarded by .tier file). + # IPs owned by other tiers are left untouched so those tiers keep working. + # Backward compat: if no .tier file exists assume the IP belongs here. + if [ -d "${vsd}/ips" ]; then + for f in "${vsd}/ips/"*.pvlan; do + [ -f "${f}" ] || continue + local tier_f; tier_f="${f%.pvlan}.tier" + if [ -f "${tier_f}" ]; then + local owner_tier; owner_tier=$(cat "${tier_f}" 2>/dev/null || true) + if [ -n "${owner_tier}" ] && [ "${owner_tier}" != "${NETWORK_ID}" ]; then + log "shutdown-network: skipping veth for $(basename "${f%.pvlan}") (owned by tier ${owner_tier})" + continue + fi + fi + local pvlan pveth_h + pvlan=$(cat "${f}") + pveth_h=$(pub_veth_host_name "${pvlan}" "${CHOSEN_ID}") + ip link del "${pveth_h}" 2>/dev/null || true + log "shutdown-network: removed public veth ${pveth_h}" + done + fi + fi + + # Remove this tier's guest veth pair (host-side) + local veth_h + veth_h=$(veth_host_name "${VLAN}" "${CHOSEN_ID}") + ip link del "${veth_h}" 2>/dev/null || true + log "shutdown-network: removed guest veth ${veth_h}" + + # Clean transient state directories. + # Shared networks have no public IP / NAT / port-forward state, so only + # isolated/VPC networks need this cleanup. + if [ "${GUEST_TYPE}" = "isolated" ]; then + if [ -z "${VPC_ID}" ]; then + rm -rf "${vsd}/ips" "${vsd}/static-nat" "${vsd}/port-forward" + else + rm -rf "${STATE_DIR}/network-${NETWORK_ID}/static-nat" \ + "${STATE_DIR}/network-${NETWORK_ID}/port-forward" 2>/dev/null || true + fi + fi + + # Stop per-network services (dnsmasq, radvd, haproxy, apache2, passwd-server) + _svc_stop_dnsmasq + _svc_stop_radvd + _svc_stop_haproxy + _svc_stop_apache2 + _svc_stop_passwd_server + + # For isolated and Shared networks delete the namespace directly. + # VPC namespaces are shared across tiers and must only be deleted + # when the last tier is destroyed (shutdown-vpc). + if [ -z "${VPC_ID}" ]; then + ip netns del "${NAMESPACE}" 2>/dev/null || true + rm -rf "/etc/netns/${NAMESPACE}" 2>/dev/null || true + log "shutdown-network: deleted namespace ${NAMESPACE}" + else + log "shutdown-network: preserved shared namespace ${NAMESPACE} (VPC tier)" + fi + + release_lock + log "shutdown-network: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: destroy-network +# Delete this network's state entirely. +# For VPC tier networks the shared namespace is preserved — the namespace is +# removed only when shutdownVpc() calls shutdown-vpc or destroy-vpc. +############################################################################## + +cmd_destroy_network() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + log "destroy-network: network=${NETWORK_ID} ns=${NAMESPACE} guest_type=${GUEST_TYPE} vpc=${VPC_ID}" + + # Remove this tier's guest veth host-side + local veth_h + veth_h=$(veth_host_name "${VLAN}" "${CHOSEN_ID}") + ip link del "${veth_h}" 2>/dev/null || true + + # The guest VLAN bridge/sub-interface (ensure_host_bridge) is where VM taps + # are plugged in directly by the hypervisor. Destroying a network implies + # no VMs remain on it, so it is safe to tear these down here (guarded by + # teardown_host_bridge_if_unused() in case anything unexpected is still + # attached). + teardown_host_bridge_if_unused "${GUEST_ETH}" "${VLAN}" + + local vsd; vsd=$(_vpc_state_dir) + + # Public veth pairs and their state files only exist for Isolated networks + # (Shared networks have no NAT/routing role, so no public veths are created). + if [ "${GUEST_TYPE}" = "isolated" ]; then + # Remove public veth pairs that belong to THIS tier (guarded by .tier file). + # IPs owned by other tiers are left untouched so those tiers keep working. + # Backward compat: if no .tier file exists assume the IP belongs here. + if [ -d "${vsd}/ips" ]; then + for f in "${vsd}/ips/"*.pvlan; do + [ -f "${f}" ] || continue + local tier_f; tier_f="${f%.pvlan}.tier" + if [ -f "${tier_f}" ]; then + local owner_tier; owner_tier=$(cat "${tier_f}" 2>/dev/null || true) + if [ -n "${owner_tier}" ] && [ "${owner_tier}" != "${NETWORK_ID}" ]; then + log "destroy-network: skipping veth for $(basename "${f%.pvlan}") (owned by tier ${owner_tier})" + continue + fi + fi + local pvlan pveth_h + pvlan=$(cat "${f}") + pveth_h=$(pub_veth_host_name "${pvlan}" "${CHOSEN_ID}") + ip link del "${pveth_h}" 2>/dev/null || true + rm -f "${f}" "${f%.pvlan}" "${tier_f}" 2>/dev/null || true + # Public bridge may be shared by other tiers/networks/tenants on + # the same public VLAN; only removed once nothing else uses it. + teardown_host_bridge_if_unused "${PUB_ETH}" "${pvlan}" + done + fi + fi + + # Stop per-network services before removing state + _svc_stop_dnsmasq + _svc_stop_radvd + _svc_stop_haproxy + _svc_stop_apache2 + _svc_stop_passwd_server + + # Remove this network's per-tier state directory + rm -rf "$(_net_state_dir)" + + # Deregister this tier from VPC tracking, or delete the namespace for + # standalone networks (Isolated and Shared each own their own namespace). + if [ -n "${VPC_ID}" ]; then + rm -f "${vsd}/tiers/${NETWORK_ID}" 2>/dev/null || true + # The VPC namespace is managed by shutdown-vpc / destroy-vpc — do NOT delete it here. + log "destroy-network: deregistered tier ${NETWORK_ID} from VPC ${VPC_ID} (namespace preserved)" + else + # Isolated or Shared: each has its own namespace, remove it entirely. + if ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns del "${NAMESPACE}" + rm -rf "/etc/netns/${NAMESPACE}" 2>/dev/null || true + log "destroy-network: deleted namespace ${NAMESPACE}" + fi + fi + + release_lock + log "destroy-network: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: assign-ip +# +# Creates a public veth pair for the given public IP/VLAN, assigns the IP +# to the namespace end, and configures source NAT if requested. +############################################################################## + +cmd_assign_ip() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + _guard_ns_teardown "assign-ip" + log "assign-ip: network=${NETWORK_ID} ns=${NAMESPACE} ip=${PUBLIC_IP} source_nat=${SOURCE_NAT}" + [ -z "${PUBLIC_IP}" ] && die "Missing --public-ip" + [ -z "${PUBLIC_VLAN}" ] && die "Missing --public-vlan" + + local pveth_h pveth_n nchain_post fchain + pveth_h=$(pub_veth_host_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + nchain_post="${CHAIN_PREFIX}_${NETWORK_ID}_POST" + fchain=$(filter_chain "${NETWORK_ID}") + + # ---- Ensure public host bridge ---- + ensure_host_bridge "${PUB_ETH}" "${PUBLIC_VLAN}" + local pub_br + pub_br=$(host_bridge_name "${PUB_ETH}" "${PUBLIC_VLAN}") + + # ---- Create public veth pair (idempotent) ---- + if ! ip link show "${pveth_h}" >/dev/null 2>&1; then + ip link add "${pveth_h}" type veth peer name "${pveth_n}" + ip link set "${pveth_n}" netns "${NAMESPACE}" + ip link set "${pveth_h}" master "${pub_br}" + ip link set "${pveth_h}" up + ip netns exec "${NAMESPACE}" ip link set "${pveth_n}" up + log "Created public veth ${pveth_h} (host→${pub_br}) <-> ${pveth_n} (namespace)" + else + ip link set "${pveth_h}" up 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set "${pveth_n}" up 2>/dev/null || true + fi + + # ---- Assign public IP to namespace end ---- + local ADDR_SPEC + if [ -n "${PUBLIC_CIDR}" ] && echo "${PUBLIC_CIDR}" | grep -q '/'; then + local PREFIX + PREFIX=$(echo "${PUBLIC_CIDR}" | cut -d'/' -f2) + ADDR_SPEC="${PUBLIC_IP}/${PREFIX}" + else + ADDR_SPEC="${PUBLIC_IP}/32" + fi + ip netns exec "${NAMESPACE}" ip addr show "${pveth_n}" 2>/dev/null | \ + grep -q "${PUBLIC_IP}/" || \ + ip netns exec "${NAMESPACE}" ip addr add "${ADDR_SPEC}" dev "${pveth_n}" + + # ---- Host route for incoming traffic ---- + ip route show | grep -q "^${PUBLIC_IP}" || \ + ip route add "${PUBLIC_IP}/32" dev "${pveth_h}" 2>/dev/null || true + + # ---- Gratuitous ARP to flush upstream gateway's ARP cache ---- + # After a network restart the veth is recreated with a new MAC address. + # Without a gratuitous ARP the upstream gateway retains the stale ARP entry + # for the old MAC and packets cannot reach the new veth. + # Use _find_arping to locate the binary in PATH and common sbin locations. + local _arping_bin; _arping_bin=$(_find_arping) || true + if [ -n "${_arping_bin}" ]; then + ip netns exec "${NAMESPACE}" "${_arping_bin}" -c 3 -U -I "${pveth_n}" "${PUBLIC_IP}" \ + >/dev/null 2>&1 || true + log "assign-ip: sent gratuitous ARP for ${PUBLIC_IP} on ${pveth_n}" + else + log "assign-ip: arping not found — skipping gratuitous ARP for ${PUBLIC_IP}" + fi + + # ---- Default route inside namespace toward upstream gateway ---- + if [ -n "${PUBLIC_GATEWAY}" ]; then + ip netns exec "${NAMESPACE}" ip route replace default \ + via "${PUBLIC_GATEWAY}" dev "${pveth_n}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" ip route add default \ + via "${PUBLIC_GATEWAY}" dev "${pveth_n}" 2>/dev/null || true + log "Default route in ${NAMESPACE}: via ${PUBLIC_GATEWAY} dev ${pveth_n}" + fi + + # ---- Source NAT ---- + # For VPC tiers the SNAT rule covers the entire VPC CIDR and is set up by + # implement-vpc (using --vpc-cidr). Duplicate SNAT rules here would conflict. + if [ "${SOURCE_NAT}" = "true" ] && [ -n "${CIDR}" ] && [ -z "${VPC_ID}" ]; then + ip netns exec "${NAMESPACE}" iptables -t nat \ + -C "${nchain_post}" -s "${CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t nat \ + -A "${nchain_post}" -s "${CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C "${fchain}" -o "${pveth_n}" -s "${CIDR}" -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A "${fchain}" -o "${pveth_n}" -s "${CIDR}" -j ACCEPT + log "Source NAT: ${CIDR} -> ${PUBLIC_IP} via ${pveth_n}" + elif [ "${SOURCE_NAT}" = "true" ] && [ -n "${VPC_ID}" ]; then + log "assign-ip: skipping SNAT rules for VPC tier (managed by implement-vpc)" + fi + + # ---- Persist state ---- + # Public IP state is VPC-wide (shared across all tiers in a VPC) + local vsd; vsd=$(_vpc_state_dir) + mkdir -p "${vsd}/ips" + echo "${SOURCE_NAT}" > "${vsd}/ips/${PUBLIC_IP}" + # Save public VLAN so add-static-nat / add-port-forward can look it up + echo "${PUBLIC_VLAN}" > "${vsd}/ips/${PUBLIC_IP}.pvlan" + # Save owning tier (network ID) so cmd_destroy only cleans up its own IPs + echo "${NETWORK_ID}" > "${vsd}/ips/${PUBLIC_IP}.tier" + + _dump_iptables "${NAMESPACE}" + release_lock + log "assign-ip: done ${PUBLIC_IP} on network ${NETWORK_ID}" +} + +############################################################################## +# Command: release-ip +############################################################################## + +cmd_release_ip() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + log "release-ip: network=${NETWORK_ID} ns=${NAMESPACE} ip=${PUBLIC_IP}" + [ -z "${PUBLIC_IP}" ] && die "Missing --public-ip" + + # Restore PUBLIC_VLAN from state if not on CLI + local vsd; vsd=$(_vpc_state_dir) + if [ -z "${PUBLIC_VLAN}" ] && [ -f "${vsd}/ips/${PUBLIC_IP}.pvlan" ]; then + PUBLIC_VLAN=$(cat "${vsd}/ips/${PUBLIC_IP}.pvlan") + fi + [ -z "${PUBLIC_VLAN}" ] && die "release-ip: cannot determine public VLAN for ${PUBLIC_IP}" + + local pveth_h pveth_n nchain_post fchain + pveth_h=$(pub_veth_host_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + nchain_post="${CHAIN_PREFIX}_${NETWORK_ID}_POST" + fchain=$(filter_chain "${NETWORK_ID}") + + # Remove SNAT rule + if [ -n "${CIDR}" ]; then + ip netns exec "${NAMESPACE}" iptables -t nat \ + -D "${nchain_post}" -s "${CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -o "${pveth_n}" -s "${CIDR}" -j ACCEPT 2>/dev/null || true + fi + + # Remove DNAT rules for this public IP + local nchain_pr + nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" + ip netns exec "${NAMESPACE}" iptables -t nat -S "${nchain_pr}" 2>/dev/null | \ + grep -- "-d ${PUBLIC_IP}" | \ + while read -r rule; do + ip netns exec "${NAMESPACE}" iptables -t nat \ + -D "${nchain_pr}" ${rule#-A ${nchain_pr}} 2>/dev/null || true + done + + # Remove host route + ip route del "${PUBLIC_IP}/32" 2>/dev/null || true + + # Remove IP from namespace veth + if [ -n "${PUBLIC_CIDR}" ] && echo "${PUBLIC_CIDR}" | grep -q '/'; then + local PREFIX + PREFIX=$(echo "${PUBLIC_CIDR}" | cut -d'/' -f2) + ip netns exec "${NAMESPACE}" ip addr del "${PUBLIC_IP}/${PREFIX}" dev "${pveth_n}" 2>/dev/null || true + fi + ip netns exec "${NAMESPACE}" ip addr del "${PUBLIC_IP}/32" dev "${pveth_n}" 2>/dev/null || true + + # Delete public veth if no other IPs share the same public VLAN + network id + local remaining + remaining=$(find "${vsd}/ips/" -name "*.pvlan" \ + ! -name "${PUBLIC_IP}.pvlan" -exec grep -l "^${PUBLIC_VLAN}$" {} \; 2>/dev/null | wc -l) + if [ "${remaining}" -eq 0 ]; then + ip link del "${pveth_h}" 2>/dev/null || true + log "release-ip: removed public veth ${pveth_h}" + # Public bridge may still be shared by other networks/tenants on the + # same public VLAN; only removed once nothing else uses it. + teardown_host_bridge_if_unused "${PUB_ETH}" "${PUBLIC_VLAN}" + fi + + # Remove default route if no IPs remain + if [ -n "${PUBLIC_GATEWAY}" ]; then + local total_ips + total_ips=$(find "${vsd}/ips/" -maxdepth 1 \ + -not -name "*.pvlan" -type f 2>/dev/null | wc -l) + if [ "${total_ips}" -le 1 ]; then + ip netns exec "${NAMESPACE}" ip route del default \ + via "${PUBLIC_GATEWAY}" dev "${pveth_n}" 2>/dev/null || true + fi + fi + + rm -f "${vsd}/ips/${PUBLIC_IP}" \ + "${vsd}/ips/${PUBLIC_IP}.pvlan" \ + "${vsd}/ips/${PUBLIC_IP}.tier" + + _dump_iptables "${NAMESPACE}" + release_lock + log "release-ip: done ${PUBLIC_IP} on network ${NETWORK_ID}" +} + +############################################################################## +# Command: add-static-nat +############################################################################## + +cmd_add_static_nat() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + _guard_ns_teardown "add-static-nat" + log "add-static-nat: network=${NETWORK_ID} ns=${NAMESPACE} ${PUBLIC_IP} <-> ${PRIVATE_IP}" + [ -z "${PUBLIC_IP}" ] && die "Missing --public-ip" + [ -z "${PRIVATE_IP}" ] && die "Missing --private-ip" + + # Restore PUBLIC_VLAN from state (written by assign-ip) + local vsd; vsd=$(_vpc_state_dir) + if [ -z "${PUBLIC_VLAN}" ] && [ -f "${vsd}/ips/${PUBLIC_IP}.pvlan" ]; then + PUBLIC_VLAN=$(cat "${vsd}/ips/${PUBLIC_IP}.pvlan") + fi + [ -z "${PUBLIC_VLAN}" ] && die "add-static-nat: cannot determine public VLAN for ${PUBLIC_IP}" + + local pveth_h pveth_n veth_n nchain_pr nchain_post fchain + pveth_h=$(pub_veth_host_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" + nchain_post="${CHAIN_PREFIX}_${NETWORK_ID}_POST" + fchain=$(filter_chain "${NETWORK_ID}") + + ensure_public_ip_on_namespace "${PUBLIC_IP}" "${PUBLIC_CIDR}" "${pveth_n}" "${pveth_h}" + + # DNAT: inbound public IP → private IP (PREROUTING) + ip netns exec "${NAMESPACE}" iptables -t nat \ + -C "${nchain_pr}" -d "${PUBLIC_IP}" -j DNAT --to-destination "${PRIVATE_IP}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t nat \ + -A "${nchain_pr}" -d "${PUBLIC_IP}" -j DNAT --to-destination "${PRIVATE_IP}" + + # SNAT: outbound private IP → public IP (POSTROUTING, out public veth) + ip netns exec "${NAMESPACE}" iptables -t nat \ + -C "${nchain_post}" -s "${PRIVATE_IP}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t nat \ + -A "${nchain_post}" -s "${PRIVATE_IP}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" + + # FORWARD: allow traffic to/from private IP via guest veth + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C "${fchain}" -d "${PRIVATE_IP}" -o "${veth_n}" -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A "${fchain}" -d "${PRIVATE_IP}" -o "${veth_n}" -j ACCEPT + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C "${fchain}" -s "${PRIVATE_IP}" -i "${veth_n}" -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A "${fchain}" -s "${PRIVATE_IP}" -i "${veth_n}" -j ACCEPT + + mkdir -p "${vsd}/static-nat" + echo "${PRIVATE_IP}" > "${vsd}/static-nat/${PUBLIC_IP}" + + _dump_iptables "${NAMESPACE}" + release_lock + log "add-static-nat: done ${PUBLIC_IP} <-> ${PRIVATE_IP} in ${NAMESPACE}" +} + +############################################################################## +# Command: delete-static-nat +############################################################################## + +cmd_delete_static_nat() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + log "delete-static-nat: network=${NETWORK_ID} ns=${NAMESPACE} ${PUBLIC_IP}" + [ -z "${PUBLIC_IP}" ] && die "Missing --public-ip" + + local vsd; vsd=$(_vpc_state_dir) + if [ -z "${PRIVATE_IP}" ] && [ -f "${vsd}/static-nat/${PUBLIC_IP}" ]; then + PRIVATE_IP=$(cat "${vsd}/static-nat/${PUBLIC_IP}") + fi + [ -z "${PRIVATE_IP}" ] && die "Missing --private-ip and no saved state" + + # Restore PUBLIC_VLAN from state + if [ -z "${PUBLIC_VLAN}" ] && [ -f "${vsd}/ips/${PUBLIC_IP}.pvlan" ]; then + PUBLIC_VLAN=$(cat "${vsd}/ips/${PUBLIC_IP}.pvlan") + fi + [ -z "${PUBLIC_VLAN}" ] && die "delete-static-nat: cannot determine public VLAN for ${PUBLIC_IP}" + + local pveth_n veth_n nchain_pr nchain_post fchain + pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" + nchain_post="${CHAIN_PREFIX}_${NETWORK_ID}_POST" + fchain=$(filter_chain "${NETWORK_ID}") + + ip netns exec "${NAMESPACE}" iptables -t nat \ + -D "${nchain_pr}" -d "${PUBLIC_IP}" -j DNAT --to-destination "${PRIVATE_IP}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t nat \ + -D "${nchain_post}" -s "${PRIVATE_IP}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -d "${PRIVATE_IP}" -o "${veth_n}" -j ACCEPT 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -s "${PRIVATE_IP}" -i "${veth_n}" -j ACCEPT 2>/dev/null || true + + rm -f "${vsd}/static-nat/${PUBLIC_IP}" + + _dump_iptables "${NAMESPACE}" + release_lock + log "delete-static-nat: done ${PUBLIC_IP} <-> ${PRIVATE_IP}" +} + +############################################################################## +# Command: add-port-forward +############################################################################## + +cmd_add_port_forward() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + _guard_ns_teardown "add-port-forward" + log "add-port-forward: network=${NETWORK_ID} ns=${NAMESPACE} ${PUBLIC_IP}:${PUBLIC_PORT} -> ${PRIVATE_IP}:${PRIVATE_PORT} (${PROTOCOL})" + [ -z "${PUBLIC_IP}" ] && die "Missing --public-ip" + [ -z "${PUBLIC_PORT}" ] && die "Missing --public-port" + [ -z "${PRIVATE_IP}" ] && die "Missing --private-ip" + [ -z "${PRIVATE_PORT}" ] && die "Missing --private-port" + [ -z "${PROTOCOL}" ] && PROTOCOL="tcp" + + # Restore PUBLIC_VLAN from state + local vsd; vsd=$(_vpc_state_dir) + if [ -z "${PUBLIC_VLAN}" ] && [ -f "${vsd}/ips/${PUBLIC_IP}.pvlan" ]; then + PUBLIC_VLAN=$(cat "${vsd}/ips/${PUBLIC_IP}.pvlan") + fi + [ -z "${PUBLIC_VLAN}" ] && die "add-port-forward: cannot determine public VLAN for ${PUBLIC_IP}" + + local pveth_h pveth_n veth_n nchain_pr fchain + pveth_h=$(pub_veth_host_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${CHOSEN_ID}") + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" + fchain=$(filter_chain "${NETWORK_ID}") + + ensure_public_ip_on_namespace "${PUBLIC_IP}" "${PUBLIC_CIDR}" "${pveth_n}" "${pveth_h}" + + # DNAT + ip netns exec "${NAMESPACE}" iptables -t nat \ + -C "${nchain_pr}" -p "${PROTOCOL}" -d "${PUBLIC_IP}" --dport "${PUBLIC_PORT}" \ + -j DNAT --to-destination "${PRIVATE_IP}:${PRIVATE_PORT}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t nat \ + -A "${nchain_pr}" -p "${PROTOCOL}" -d "${PUBLIC_IP}" --dport "${PUBLIC_PORT}" \ + -j DNAT --to-destination "${PRIVATE_IP}:${PRIVATE_PORT}" + + # Allow forwarding for the mapped private port via guest veth + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C "${fchain}" -p "${PROTOCOL}" -d "${PRIVATE_IP}" --dport "${PRIVATE_PORT}" \ + -o "${veth_n}" -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A "${fchain}" -p "${PROTOCOL}" -d "${PRIVATE_IP}" --dport "${PRIVATE_PORT}" \ + -o "${veth_n}" -j ACCEPT + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C "${fchain}" -p "${PROTOCOL}" -s "${PRIVATE_IP}" --sport "${PRIVATE_PORT}" \ + -i "${veth_n}" -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A "${fchain}" -p "${PROTOCOL}" -s "${PRIVATE_IP}" --sport "${PRIVATE_PORT}" \ + -i "${veth_n}" -j ACCEPT + + local safe_port + safe_port=$(echo "${PUBLIC_PORT}" | tr ':' '-') + mkdir -p "${vsd}/port-forward" + echo "${PROTOCOL} ${PUBLIC_IP} ${PUBLIC_PORT} ${PRIVATE_IP} ${PRIVATE_PORT}" > \ + "${vsd}/port-forward/${PROTOCOL}_${PUBLIC_IP}_${safe_port}" + + _dump_iptables "${NAMESPACE}" + release_lock + log "add-port-forward: done ${PUBLIC_IP}:${PUBLIC_PORT} -> ${PRIVATE_IP}:${PRIVATE_PORT}" +} + +############################################################################## +# Command: delete-port-forward +############################################################################## + +cmd_delete_port_forward() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + + log "delete-port-forward: network=${NETWORK_ID} ns=${NAMESPACE} ${PUBLIC_IP}:${PUBLIC_PORT} -> ${PRIVATE_IP}:${PRIVATE_PORT}" + [ -z "${PUBLIC_IP}" ] && die "Missing --public-ip" + [ -z "${PUBLIC_PORT}" ] && die "Missing --public-port" + [ -z "${PRIVATE_IP}" ] && die "Missing --private-ip" + [ -z "${PRIVATE_PORT}" ] && die "Missing --private-port" + [ -z "${PROTOCOL}" ] && PROTOCOL="tcp" + + local veth_n nchain_pr fchain + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + nchain_pr="${CHAIN_PREFIX}_${NETWORK_ID}_PR" + fchain=$(filter_chain "${NETWORK_ID}") + + ip netns exec "${NAMESPACE}" iptables -t nat \ + -D "${nchain_pr}" -p "${PROTOCOL}" -d "${PUBLIC_IP}" --dport "${PUBLIC_PORT}" \ + -j DNAT --to-destination "${PRIVATE_IP}:${PRIVATE_PORT}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -p "${PROTOCOL}" -d "${PRIVATE_IP}" --dport "${PRIVATE_PORT}" \ + -o "${veth_n}" -j ACCEPT 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -p "${PROTOCOL}" -s "${PRIVATE_IP}" --sport "${PRIVATE_PORT}" \ + -i "${veth_n}" -j ACCEPT 2>/dev/null || true + + local safe_port + safe_port=$(echo "${PUBLIC_PORT}" | tr ':' '-') + local vsd; vsd=$(_vpc_state_dir) + rm -f "${vsd}/port-forward/${PROTOCOL}_${PUBLIC_IP}_${safe_port}" + + _dump_iptables "${NAMESPACE}" + release_lock + log "delete-port-forward: done" +} + +############################################################################## +# Helpers: path accessors (require NETWORK_ID to be set) +# +# Per-network state → _net_state_dir() = ${STATE_DIR}/network- +# VPC-wide state → _vpc_state_dir() = ${STATE_DIR}/vpc- (or same as net for isolated) +############################################################################## + +_dnsmasq_dir() { echo "$(_net_state_dir)/dnsmasq"; } +_dnsmasq_conf() { echo "$(_net_state_dir)/dnsmasq/dnsmasq.conf"; } +_dnsmasq_pid() { echo "$(_net_state_dir)/dnsmasq/dnsmasq.pid"; } +_dnsmasq_hosts() { echo "$(_net_state_dir)/dnsmasq/hosts"; } +_dnsmasq_dhcp_hosts() { echo "$(_net_state_dir)/dnsmasq/dhcp-hosts"; } +_dnsmasq_dhcp_opts() { echo "$(_net_state_dir)/dnsmasq/dhcp-opts"; } + +_radvd_dir() { echo "$(_net_state_dir)/radvd"; } +_radvd_conf() { echo "$(_net_state_dir)/radvd/radvd.conf"; } +_radvd_pid() { echo "$(_net_state_dir)/radvd/radvd.pid"; } + +_haproxy_dir() { echo "$(_net_state_dir)/haproxy"; } +_haproxy_conf() { echo "$(_net_state_dir)/haproxy/haproxy.cfg"; } +_haproxy_pid() { echo "$(_net_state_dir)/haproxy/haproxy.pid"; } +_haproxy_sock() { echo "$(_net_state_dir)/haproxy/haproxy.sock"; } + +_apache2_dir() { echo "$(_net_state_dir)/apache2"; } +_apache2_conf() { echo "$(_net_state_dir)/apache2/apache2.conf"; } +_apache2_pid() { echo "$(_net_state_dir)/apache2/apache2.pid"; } +_metadata_dir() { echo "$(_net_state_dir)/metadata"; } +_apache2_cgi() { echo "$(_net_state_dir)/apache2/metadata.cgi"; } + +# Password server (VR-compatible, port 8080, DomU_Request protocol) +_passwd_file() { echo "$(_net_state_dir)/passwords"; } +_passwd_server_pid() { echo "$(_net_state_dir)/passwd-server.pid"; } +_passwd_server_script() { echo "$(_net_state_dir)/passwd-server.py"; } + +############################################################################## +# Helpers: binary detection and package checking +############################################################################## + +# _require_binary [ ...] +# Checks that at least one of the given binaries is executable. +# If none are found: +# - network_state is shutdown/destroy/allocated → log and exit 0 (the service +# was never running so there is nothing to clean up). +# - otherwise → die() with a clear error message. +_require_binary() { + local bin + for bin in "$@"; do + if command -v "${bin}" >/dev/null 2>&1 || [ -x "${bin}" ]; then + return 0 + fi + done + case "${NETWORK_STATE:-}" in + shutdown|destroy|allocated) + log "Required binary '$1' not found (network_state=${NETWORK_STATE}) — treating as success" + release_lock + exit 0 ;; + esac + die "Required binary '$1' not found on this host" +} + +_find_apache2_bin() { + for bin in apache2 httpd; do + command -v "${bin}" >/dev/null 2>&1 && echo "${bin}" && return + done + echo "" +} + +_find_apache2_modules_dir() { + for d in /usr/lib/apache2/modules /usr/lib64/apache2/modules \ + /usr/libexec/apache2 /usr/lib/httpd/modules \ + /usr/libexec/httpd /usr/lib64/httpd/modules/; do + [ -d "${d}" ] && echo "${d}" && return + done + echo "/usr/lib/apache2/modules" +} + +_apache2_user() { + id www-data >/dev/null 2>&1 && echo "www-data" && return + id apache >/dev/null 2>&1 && echo "apache" && return + echo "nobody" +} + +# Locate the arping binary; checks PATH first, then common sbin paths. +# Prints the path and returns 0 on success, returns 1 when not found. +_find_arping() { + local bin + for bin in arping /usr/bin/arping /usr/sbin/arping /sbin/arping; do + if command -v "${bin}" >/dev/null 2>&1 || [ -x "${bin}" ]; then + echo "${bin}" + return 0 + fi + done + return 1 +} + +############################################################################## +# Helpers: dnsmasq (DHCP + DNS via the same process) +############################################################################## + +# _cidr_dhcp_range → ",," +_cidr_dhcp_range() { + local ext_ip; ext_ip="${EXTENSION_IP:-${GATEWAY}}" + echo "${ext_ip},static" +} + +# _write_dnsmasq_conf +# Requires: NETWORK_ID, VLAN, CHOSEN_ID, CIDR, GATEWAY, DNS_SERVER +_write_dnsmasq_conf() { + local dns_enabled="${1:-false}" + local dir; dir=$(_dnsmasq_dir) + mkdir -p "${dir}" + + local dhcp_hosts; dhcp_hosts=$(_dnsmasq_dhcp_hosts) + local hosts; hosts=$(_dnsmasq_hosts) + local dhcp_opts; dhcp_opts=$(_dnsmasq_dhcp_opts) + touch "${dhcp_hosts}" "${hosts}" "${dhcp_opts}" + + local veth_n; veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + local dhcp_range; dhcp_range=$(_cidr_dhcp_range "${CIDR}" "${GATEWAY}") + local port_line="port=0" + [ "${dns_enabled}" = "true" ] && port_line="port=53" + + cat > "$(_dnsmasq_conf)" << EOF +# Auto-generated by network-namespace-wrapper.sh — do not edit +${port_line} +interface=${veth_n} +except-interface=lo +#no-hosts +bind-interfaces +localise-queries +pid-file=$(_dnsmasq_pid) +dhcp-range=${dhcp_range},12h +dhcp-option=3,${GATEWAY} +dhcp-hostsfile=${dhcp_hosts} +addn-hosts=${hosts} +dhcp-optsfile=${dhcp_opts} +log-facility=/var/log/cloudstack/extensions/${_WRAPPER_EXT_DIR}/dnsmasq-${NETWORK_ID}.log +EOF + # Add DHCP option 15 (domain-search) when provided by the caller + if [ -n "${DOMAIN}" ]; then + echo "dhcp-option=15,\"${DOMAIN}\"" >> "$(_dnsmasq_conf)" + echo "domain=${DOMAIN}" >> "$(_dnsmasq_conf)" + echo "expand-hosts" >> "$(_dnsmasq_conf)" + fi + local ext_ip; ext_ip="${EXTENSION_IP:-${GATEWAY}}" + [ -n "${DNS_SERVER}" ] && echo "dhcp-option=6,${ext_ip},${DNS_SERVER}" >> "$(_dnsmasq_conf)" + [ -z "${DNS_SERVER}" ] && echo "dhcp-option=6,${ext_ip}" >> "$(_dnsmasq_conf)" + log "dnsmasq: wrote config $(_dnsmasq_conf) (dns_enabled=${dns_enabled})" +} + +_svc_start_or_reload_dnsmasq() { + _require_binary dnsmasq + local pid_f; pid_f=$(_dnsmasq_pid) + if [ -f "${pid_f}" ] && kill -0 "$(cat "${pid_f}")" 2>/dev/null; then + log "dnsmasq: sending SIGHUP to reload (pid=$(cat "${pid_f}"))" + ip netns exec "${NAMESPACE}" kill -HUP "$(cat "${pid_f}")" 2>/dev/null || true + else + log "dnsmasq: starting in namespace ${NAMESPACE}" + if ! ip netns exec "${NAMESPACE}" dnsmasq --conf-file="$(_dnsmasq_conf)"; then + die "dnsmasq failed to start — check $(_dnsmasq_conf) and ${LOG_FILE} for details" + fi + fi +} + +_svc_stop_dnsmasq() { + local pid_f; pid_f=$(_dnsmasq_pid) + if [ -f "${pid_f}" ]; then + local pid; pid=$(cat "${pid_f}") + kill "${pid}" 2>/dev/null || true + rm -f "${pid_f}" + log "dnsmasq: stopped (pid=${pid})" + fi + # Kill any orphaned dnsmasq for this network + pkill -f "dnsmasq.*${NETWORK_ID}" 2>/dev/null || true +} + +############################################################################## +# Helpers: radvd (IPv6 Router Advertisement) +############################################################################## + +# _enable_ipv6_in_namespace +# Enables IPv6 forwarding inside the namespace for the given interface. +# Disables DAD and temporary addresses to avoid tentative/dadfailed states. +_enable_ipv6_in_namespace() { + local veth_n="$1" + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.all.disable_ipv6=0 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.default.disable_ipv6=0 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.all.forwarding=1 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.all.accept_ra=1 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.all.accept_dad=0 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.default.accept_dad=0 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.all.use_tempaddr=0 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf.default.use_tempaddr=0 >/dev/null 2>&1 || true + if [ -n "${veth_n}" ]; then + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf."${veth_n}".accept_dad=0 >/dev/null 2>&1 || true + ip netns exec "${NAMESPACE}" sysctl -w net.ipv6.conf."${veth_n}".use_tempaddr=0 >/dev/null 2>&1 || true + fi +} + +# _write_radvd_conf [] +# Generates /radvd/radvd.conf for the given interface and prefix. +# ip6_cidr must be in CIDR notation (e.g. 2001:db8:1::/64). +# dns6 is an optional comma-separated list of IPv6 DNS server addresses. +_write_radvd_conf() { + local veth_n="$1" ip6_gw="$2" ip6_cidr="$3" dns6="${4:-}" + + local dir; dir=$(_radvd_dir) + mkdir -p "${dir}" + + # Use the network address from ip6_cidr (all interface bits zero) as the + # radvd prefix. Using the gateway address (e.g. ::1/64) causes radvd to + # reject the config on strict versions because host bits are set. + local prefix_addr prefix_size + prefix_addr=$(echo "${ip6_cidr}" | cut -d'/' -f1) + prefix_size=$(echo "${ip6_cidr}" | cut -d'/' -f2) + + cat > "$(_radvd_conf)" << EOF +# Auto-generated by network-namespace-wrapper.sh — do not edit +interface ${veth_n} +{ + AdvSendAdvert on; + MinRtrAdvInterval 5; + MaxRtrAdvInterval 15; + prefix ${prefix_addr}/${prefix_size} + { + AdvOnLink on; + AdvAutonomous on; + }; +EOF + + if [ -n "${dns6}" ]; then + local dns + # shellcheck disable=SC2086 + for dns in $(echo "${dns6}" | tr ',' ' '); do + [ -z "${dns}" ] && continue + cat >> "$(_radvd_conf)" << EOF + RDNSS ${dns} + { + AdvRDNSSLifetime 30; + }; +EOF + done + fi + + echo "};" >> "$(_radvd_conf)" + log "radvd: wrote config $(_radvd_conf)" +} + +_svc_start_or_reload_radvd() { + _require_binary radvd + local pid_f; pid_f=$(_radvd_pid) + if [ -f "${pid_f}" ] && kill -0 "$(cat "${pid_f}")" 2>/dev/null; then + log "radvd: sending SIGHUP to reload (pid=$(cat "${pid_f}"))" + kill -HUP "$(cat "${pid_f}")" 2>/dev/null || true + else + log "radvd: starting in namespace ${NAMESPACE}" + local _ct_out + # Use 'if !' to prevent set -e from killing the script on radvd failure. + # Flag mapping (this radvd build): -C PATH = config file, -c = configtest. + if ! _ct_out=$(ip netns exec "${NAMESPACE}" radvd \ + -C "$(_radvd_conf)" -c 2>&1); then + die "radvd config test failed: ${_ct_out}" + fi + ip netns exec "${NAMESPACE}" radvd \ + -C "$(_radvd_conf)" \ + -p "$(_radvd_pid)" \ + -m syslog + log "radvd: started in namespace ${NAMESPACE}" + fi +} + +_svc_stop_radvd() { + local pid_f; pid_f=$(_radvd_pid) + if [ -f "${pid_f}" ]; then + local pid; pid=$(cat "${pid_f}") + kill "${pid}" 2>/dev/null || true + rm -f "${pid_f}" + log "radvd: stopped (pid=${pid})" + fi +} + +############################################################################## +# Helpers: haproxy (LB via haproxy) +############################################################################## + +# Regenerate haproxy config from all persisted per-rule JSON files. +# Requires: NETWORK_ID +_write_haproxy_conf() { + local lb_dir; lb_dir=$(_haproxy_dir) + local pid_f; pid_f=$(_haproxy_pid) + local sock_f; sock_f=$(_haproxy_sock) + mkdir -p "${lb_dir}" + + python3 - "${lb_dir}" "${pid_f}" "${sock_f}" > "$(_haproxy_conf)" 2>/dev/null << 'PYEOF' +import json, os, sys + +lb_dir = sys.argv[1] +pid_file = sys.argv[2] +sock = sys.argv[3] + +rules = [] +if os.path.isdir(lb_dir): + for fn in sorted(os.listdir(lb_dir)): + if fn.endswith('.json'): + try: + with open(os.path.join(lb_dir, fn)) as f: + r = json.load(f) + if not r.get('revoke', False): + rules.append(r) + except Exception: + pass + +lines = [ + "global", + " daemon", + " maxconn 4096", + " log /dev/log local0", + f" stats socket {sock} mode 660 level admin", + f" pidfile {pid_file}", + "", + "defaults", + " mode tcp", + " timeout connect 5s", + " timeout client 50s", + " timeout server 50s", + " log global", + "", +] + +ALG_MAP = { + 'roundrobin': 'roundrobin', 'leastconn': 'leastconn', + 'source': 'source', 'static-rr': 'static-rr', + 'least_conn': 'leastconn', +} + +for rule in rules: + rid = rule['id'] + pub_ip = rule.get('publicIp', '') + pub_port = rule.get('publicPort', 0) + alg = ALG_MAP.get(rule.get('algorithm', '').lower(), 'roundrobin') + backends = [b for b in rule.get('backends', []) if not b.get('revoked', False)] + if not backends: + continue + lines += [ + f"frontend cs_lb_{rid}_front", + f" bind {pub_ip}:{pub_port}", + f" default_backend cs_lb_{rid}_back", + "", + f"backend cs_lb_{rid}_back", + f" balance {alg}", + ] + for i, b in enumerate(backends): + bip = b.get('ip', '') + bport = b.get('port', pub_port) + lines.append(f" server backend_{rid}_{i} {bip}:{bport} check") + lines.append("") + +print('\n'.join(lines)) +PYEOF + log "haproxy: wrote config $(_haproxy_conf)" +} + +_svc_reload_haproxy() { + _require_binary haproxy + local conf_f; conf_f=$(_haproxy_conf) + local pid_f; pid_f=$(_haproxy_pid) + + if [ -f "${pid_f}" ] && kill -0 "$(cat "${pid_f}")" 2>/dev/null; then + log "haproxy: reloading" + if ! ip netns exec "${NAMESPACE}" haproxy -f "${conf_f}" -p "${pid_f}" \ + -sf "$(cat "${pid_f}")" 2>/dev/null; then + log "WARNING: haproxy reload failed; attempting fresh start" + rm -f "${pid_f}" 2>/dev/null || true + if ! ip netns exec "${NAMESPACE}" haproxy -f "${conf_f}" -p "${pid_f}" 2>/dev/null; then + die "haproxy failed to start — check ${conf_f} and ${LOG_FILE} for details" + fi + fi + else + log "haproxy: starting in namespace ${NAMESPACE}" + if ! ip netns exec "${NAMESPACE}" haproxy -f "${conf_f}" -p "${pid_f}" 2>/dev/null; then + die "haproxy failed to start — check ${conf_f} and ${LOG_FILE} for details" + fi + fi +} + +_svc_stop_haproxy() { + local pid_f; pid_f=$(_haproxy_pid) + if [ -f "${pid_f}" ]; then + local pid; pid=$(cat "${pid_f}") + ip netns exec "${NAMESPACE}" kill "${pid}" 2>/dev/null || kill "${pid}" 2>/dev/null || true + rm -f "${pid_f}" + log "haproxy: stopped (pid=${pid})" + fi +} + +############################################################################## +# Helpers: apache2 (userdata / metadata HTTP service) +# +# apache2 runs inside the namespace, listening on :80. +# EXTENSION_IP equals the network gateway when SourceNat/Gateway is enabled, +# or a dedicated placeholder IP otherwise. Falls back to GATEWAY when absent. +# +# Files served: +# ${STATE_DIR}//metadata//latest/user-data +# ${STATE_DIR}//metadata//latest/meta-data/public-keys +# ${STATE_DIR}//metadata//latest/meta-data/hypervisor-hostname +# ${STATE_DIR}//metadata//latest/meta-data/local-hostname +# +# Apache2 uses a CGI script to dispatch requests to the per-IP subtree. +############################################################################## + +_write_apache2_conf() { + local dir; dir=$(_apache2_dir) + local www; www=$(_metadata_dir) + local cgi; cgi=$(_apache2_cgi) + local mods; mods=$(_find_apache2_modules_dir) + local apuser; apuser=$(_apache2_user) + mkdir -p "${dir}" "${www}" + + # ---- CGI dispatcher script ---- + cat > "${cgi}" << 'CGISCRIPT' +#!/bin/bash +# Metadata/userdata CGI dispatcher – EC2-compatible directory listing support. +# Requests are identified by REMOTE_ADDR (the VM's IP on this network). +CLIENT="${REMOTE_ADDR}" +BASEDIR="$(dirname "$0")/../metadata" +REQ="${PATH_INFO:-${REQUEST_URI}}" +# Strip query string if present +REQ="${REQ%%\?*}" +# Remove leading slash +REQ="${REQ#/}" +# Resolve path, stripping any trailing slash for filesystem lookup +TARGET="${BASEDIR}/${CLIENT}/${REQ%/}" + +if [ -f "${TARGET}" ]; then + # Regular file – serve contents + printf 'Content-Type: text/plain\r\n\r\n' + cat "${TARGET}" +elif [ -d "${TARGET}" ]; then + # Directory – return a newline-delimited list of entries (EC2 API style). + # Sub-directories are listed with a trailing '/'. + printf 'Content-Type: text/plain\r\n\r\n' + for item in "${TARGET}/"*; do + [ -e "${item}" ] || continue # skip empty glob + name=$(basename "${item}") + if [ -d "${item}" ]; then + printf '%s/\n' "${name}" + else + printf '%s\n' "${name}" + fi + done +else + printf 'Status: 404 Not Found\r\nContent-Type: text/plain\r\n\r\nNot found\n' +fi +CGISCRIPT + chmod +x "${cgi}" + + # ---- Detect MPM module ---- + local mpm_mod="mpm_event_module" + local mpm_so="mod_mpm_event.so" + if [ ! -f "${mods}/${mpm_so}" ] && [ -f "${mods}/mod_mpm_prefork.so" ]; then + mpm_mod="mpm_prefork_module"; mpm_so="mod_mpm_prefork.so" + fi + + # ---- Check for authz_core (required in apache2 >= 2.4) ---- + local authz_line="" + [ -f "${mods}/mod_authz_core.so" ] && \ + authz_line="LoadModule authz_core_module ${mods}/mod_authz_core.so" + + local unixd_line="" + [ -f "${mods}/mod_unixd.so" ] && \ + unixd_line="LoadModule unixd_module ${mods}/mod_unixd.so" + + local require_line="Allow from all" + [ -f "${mods}/mod_authz_core.so" ] && require_line="Require all granted" + + # Use EXTENSION_IP as the listen address; fall back to GATEWAY when absent. + local listen_ip; listen_ip="${EXTENSION_IP:-${GATEWAY}}" + + cat > "$(_apache2_conf)" << EOF +# Auto-generated by network-namespace-wrapper.sh — do not edit +ServerRoot /tmp +PidFile $(_apache2_pid) +ServerName metadata-${NETWORK_ID} +Listen ${listen_ip}:80 +#User ${apuser} +#Group ${apuser} + +LoadModule ${mpm_mod} ${mods}/${mpm_so} +LoadModule cgi_module ${mods}/mod_cgi.so +LoadModule alias_module ${mods}/mod_alias.so +${unixd_line} +${authz_line} + +DocumentRoot ${www} +ErrorLog /var/log/cloudstack/extensions/${_WRAPPER_EXT_DIR}/apache2-${NETWORK_ID}.log + + + ServerName metadata + ScriptAlias / ${cgi}/ + + Options +ExecCGI + AllowOverride None + ${require_line} + + +EOF + log "apache2: wrote config $(_apache2_conf) (listen=${listen_ip}:80)" +} + +_svc_start_or_reload_apache2() { + _require_binary apache2 httpd + local bin; bin=$(_find_apache2_bin) + local pid_f; pid_f=$(_apache2_pid) + + if [ -f "${pid_f}" ] && kill -0 "$(cat "${pid_f}")" 2>/dev/null; then + log "apache2: graceful restart (pid=$(cat "${pid_f}"))" + ip netns exec "${NAMESPACE}" "${bin}" -f "$(_apache2_conf)" -k graceful 2>/dev/null || \ + log "WARNING: apache2 graceful restart failed" + else + log "apache2: starting in namespace ${NAMESPACE}" + if ! ip netns exec "${NAMESPACE}" "${bin}" -f "$(_apache2_conf)" -k start 2>/dev/null; then + die "apache2/httpd failed to start — check $(_apache2_conf) and ${LOG_FILE} for details" + fi + fi + + # Allow metadata traffic inbound to the namespace (INPUT) from guest subnet only. + # Skip if namespace is gone (e.g. network already shut down). + if [ -n "${CIDR}" ] && ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C INPUT -p tcp -s "${CIDR}" --dport 80 -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A INPUT -p tcp -s "${CIDR}" --dport 80 -j ACCEPT + fi +} + +_svc_stop_apache2() { + local bin; bin=$(_find_apache2_bin) + local pid_f; pid_f=$(_apache2_pid) + + if [ -f "${pid_f}" ] && kill -0 "$(cat "${pid_f}")" 2>/dev/null; then + local pid; pid=$(cat "${pid_f}") + ip netns exec "${NAMESPACE}" "${bin}" -f "$(_apache2_conf)" -k stop 2>/dev/null || \ + kill "${pid}" 2>/dev/null || true + rm -f "${pid_f}" + log "apache2: stopped (pid=${pid})" + fi +} + +############################################################################## +# Helpers: passwd-server (CloudStack VR-compatible password service) +# +# Listens on :8080 inside the namespace. +# EXTENSION_IP equals the network gateway when SourceNat/Gateway is enabled, +# or a dedicated placeholder IP otherwise. Falls back to GATEWAY when absent. +# Protocol (identical to VR passwd_server_ip.py): +# GET DomU_Request: send_my_password → return password for REMOTE_ADDR +# GET DomU_Request: saved_password → remove password + ack +# +# Passwords are stored in ${STATE_DIR}//passwords as ip=password lines. +############################################################################## + +_svc_start_or_reload_passwd_server() { + local script_f; script_f=$(_passwd_server_script) + local pid_f; pid_f=$(_passwd_server_pid) + local passwd_f; passwd_f=$(_passwd_file) + local log_f; log_f="/var/log/cloudstack/extensions/${_WRAPPER_EXT_DIR}/passwd-${NETWORK_ID}.log" + + mkdir -p "$(dirname "${script_f}")" + touch "${passwd_f}" + + # Write the embedded Python password server (same protocol as VR passwd_server_ip.py) + cat > "${script_f}" << 'PYEOF' +#!/usr/bin/env python3 +import os, sys, threading, syslog +from http.server import BaseHTTPRequestHandler, HTTPServer +from socketserver import ThreadingMixIn + +gateway = sys.argv[1] if len(sys.argv) > 1 else '0.0.0.0' +passwd_file = sys.argv[2] if len(sys.argv) > 2 else '/tmp/passwords' +pid_file = sys.argv[3] if len(sys.argv) > 3 else '/tmp/passwd-server.pid' + +lock = threading.RLock() + +# Write PID so the shell wrapper can manage us +with open(pid_file, 'w') as _f: + _f.write(str(os.getpid())) + +def get_password(ip): + """Read password for ip from the passwords file (re-read on every call).""" + try: + with open(passwd_file) as f: + for line in f: + line = line.strip() + if '=' in line: + k, v = line.split('=', 1) + if k == ip: + return v + except Exception: + pass + return None + +def remove_password(ip): + """Remove the password entry for ip from the passwords file.""" + with lock: + try: + lines = [] + with open(passwd_file) as f: + for line in f: + if '=' not in line or line.strip().split('=', 1)[0] != ip: + lines.append(line) + with open(passwd_file, 'w') as f: + f.writelines(lines) + except Exception: + pass + +class PasswordRequestHandler(BaseHTTPRequestHandler): + server_version = 'CloudStack Password Server' + sys_version = '4.x' + + def do_GET(self): + req_type = self.headers.get('DomU_Request', '') + client_ip = self.client_address[0] + self.send_response(200) + self.send_header('Content-Type', 'text/plain') + self.end_headers() + if req_type == 'send_my_password': + pw = get_password(client_ip) + if pw: + self.wfile.write(pw.encode()) + syslog.syslog(f'passwd-server: password sent to {client_ip}') + else: + self.wfile.write(b'saved_password') + syslog.syslog(f'passwd-server: no password for {client_ip}') + elif req_type == 'saved_password': + remove_password(client_ip) + self.wfile.write(b'saved_password') + syslog.syslog(f'passwd-server: saved_password ack from {client_ip}') + else: + self.wfile.write(b'bad_request') + + def log_message(self, fmt, *args): + pass # silence access log; syslog used above + +class ThreadedHTTPServer(ThreadingMixIn, HTTPServer): + allow_reuse_address = True + +server = ThreadedHTTPServer((gateway, 8080), PasswordRequestHandler) +syslog.syslog(f'passwd-server: listening on {gateway}:8080 (pid={os.getpid()})') +server.serve_forever() +PYEOF + chmod +x "${script_f}" + + # Only start if not already running; the iptables rule is (re-)applied regardless + # so that it is always present in the current namespace after a cleanup restart. + if [ -f "${pid_f}" ] && kill -0 "$(cat "${pid_f}")" 2>/dev/null; then + log "passwd-server: already running (pid=$(cat "${pid_f}"))" + else + # Use EXTENSION_IP as the listen address; fall back to GATEWAY when absent. + local listen_ip; listen_ip="${EXTENSION_IP:-${GATEWAY}}" + log "passwd-server: starting in namespace ${NAMESPACE} on ${listen_ip}:8080" + ip netns exec "${NAMESPACE}" python3 "${script_f}" \ + "${listen_ip}" "${passwd_f}" "${pid_f}" \ + >> "${log_f}" 2>&1 & + # Brief pause to let the server write its PID + sleep 0.3 + fi + + # Always ensure the iptables INPUT rule is present (idempotent). + if [ -n "${CIDR}" ]; then + ip netns exec "${NAMESPACE}" iptables -t filter \ + -C INPUT -p tcp -s "${CIDR}" --dport 8080 -j ACCEPT 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t filter \ + -A INPUT -p tcp -s "${CIDR}" --dport 8080 -j ACCEPT + fi +} + +_svc_stop_passwd_server() { + local pid_f; pid_f=$(_passwd_server_pid) + if [ -f "${pid_f}" ]; then + local pid; pid=$(cat "${pid_f}") + kill "${pid}" 2>/dev/null || true + rm -f "${pid_f}" + log "passwd-server: stopped (pid=${pid})" + fi + # Kill any orphaned instance for this network + pkill -f "python3.*passwd-server.*${NETWORK_ID}" 2>/dev/null || true +} + +############################################################################## +# Command: config-dhcp-subnet +# Configure dnsmasq for DHCP (DNS disabled at port 53). +############################################################################## + +cmd_config_dhcp_subnet() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "config-dhcp-subnet: network=${NETWORK_ID} ns=${NAMESPACE} gw=${GATEWAY} cidr=${CIDR}" + [ -z "${GATEWAY}" ] && die "config-dhcp-subnet: missing --gateway" + [ -z "${CIDR}" ] && die "config-dhcp-subnet: missing --cidr" + _write_dnsmasq_conf false + _svc_start_or_reload_dnsmasq + if [ -n "${NETWORK_IP6_GATEWAY}" ] && [ -n "${NETWORK_IP6_CIDR}" ]; then + local veth_n; veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + # implement-network may have run before CloudStack assigned the IPv6 + # gateway to this network. Ensure the namespace is configured for IPv6, + # the gateway address is on the veth, and state files are persisted so + # that subsequent commands and the discovery path can find them. + _enable_ipv6_in_namespace "${veth_n}" + local ip6_prefix; ip6_prefix=$(echo "${NETWORK_IP6_CIDR}" | cut -d'/' -f2) + ip netns exec "${NAMESPACE}" ip -6 addr show "${veth_n}" 2>/dev/null | \ + grep -q "${NETWORK_IP6_GATEWAY}/" || \ + ip netns exec "${NAMESPACE}" ip -6 addr add \ + "${NETWORK_IP6_GATEWAY}/${ip6_prefix}" dev "${veth_n}" 2>/dev/null || true + local nsd; nsd=$(_net_state_dir) + [ -f "${nsd}/ip6-gateway" ] || echo "${NETWORK_IP6_GATEWAY}" > "${nsd}/ip6-gateway" + [ -f "${nsd}/ip6-cidr" ] || echo "${NETWORK_IP6_CIDR}" > "${nsd}/ip6-cidr" + [ -n "${DNS6_SERVER}" ] && { [ -f "${nsd}/dns6" ] || echo "${DNS6_SERVER}" > "${nsd}/dns6"; } + _write_radvd_conf "${veth_n}" "${NETWORK_IP6_GATEWAY}" "${NETWORK_IP6_CIDR}" "${DNS6_SERVER}" + _svc_start_or_reload_radvd + fi + release_lock + log "config-dhcp-subnet: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: config-dns-subnet +# Configure dnsmasq for DNS (also enables DHCP; DNS on port 53). +############################################################################## + +cmd_config_dns_subnet() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "config-dns-subnet: network=${NETWORK_ID} ns=${NAMESPACE} gw=${GATEWAY} cidr=${CIDR}" + [ -z "${GATEWAY}" ] && die "config-dns-subnet: missing --gateway" + [ -z "${CIDR}" ] && die "config-dns-subnet: missing --cidr" + # Ensure the per-network hosts file contains an entry for the namespace + # extension IP named 'data-server' (idempotent). + # Use EXTENSION_IP when provided; fall back to GATEWAY. + local data_server_ip; data_server_ip="${EXTENSION_IP:-${GATEWAY}}" + local hosts_f; hosts_f=$(_dnsmasq_hosts) + mkdir -p "$(dirname "${hosts_f}")" + touch "${hosts_f}" + # Remove any existing data-server lines, then append the desired mapping + grep -v -E "\sdata-server(\s|$)" "${hosts_f}" > "${hosts_f}.tmp" 2>/dev/null || true + mv "${hosts_f}.tmp" "${hosts_f}" + # Add the mapping: data-server + echo "${data_server_ip} data-server" >> "${hosts_f}" + + _write_dnsmasq_conf true + _svc_start_or_reload_dnsmasq + if [ -n "${NETWORK_IP6_GATEWAY}" ] && [ -n "${NETWORK_IP6_CIDR}" ]; then + local veth_n; veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + _enable_ipv6_in_namespace "${veth_n}" + local ip6_prefix; ip6_prefix=$(echo "${NETWORK_IP6_CIDR}" | cut -d'/' -f2) + ip netns exec "${NAMESPACE}" ip -6 addr show "${veth_n}" 2>/dev/null | \ + grep -q "${NETWORK_IP6_GATEWAY}/" || \ + ip netns exec "${NAMESPACE}" ip -6 addr add \ + "${NETWORK_IP6_GATEWAY}/${ip6_prefix}" dev "${veth_n}" 2>/dev/null || true + local nsd; nsd=$(_net_state_dir) + [ -f "${nsd}/ip6-gateway" ] || echo "${NETWORK_IP6_GATEWAY}" > "${nsd}/ip6-gateway" + [ -f "${nsd}/ip6-cidr" ] || echo "${NETWORK_IP6_CIDR}" > "${nsd}/ip6-cidr" + [ -n "${DNS6_SERVER}" ] && { [ -f "${nsd}/dns6" ] || echo "${DNS6_SERVER}" > "${nsd}/dns6"; } + _write_radvd_conf "${veth_n}" "${NETWORK_IP6_GATEWAY}" "${NETWORK_IP6_CIDR}" "${DNS6_SERVER}" + _svc_start_or_reload_radvd + fi + release_lock + log "config-dns-subnet: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: remove-dhcp-subnet +# Tear down dnsmasq DHCP for this network. +############################################################################## + +cmd_remove_dhcp_subnet() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + log "remove-dhcp-subnet: network=${NETWORK_ID}" + _svc_stop_dnsmasq + _svc_stop_radvd + rm -rf "$(_dnsmasq_dir)" "$(_radvd_dir)" + release_lock + log "remove-dhcp-subnet: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: remove-dns-subnet +# Disable DNS (port 53) but keep DHCP running if configured. +############################################################################## + +cmd_remove_dns_subnet() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "remove-dns-subnet: network=${NETWORK_ID}" + if [ -f "$(_dnsmasq_conf)" ]; then + _write_dnsmasq_conf false + _svc_start_or_reload_dnsmasq + fi + release_lock + log "remove-dns-subnet: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: add-dhcp-entry +# Add a static DHCP host entry (mac→ip) to dnsmasq. +############################################################################## + +cmd_add_dhcp_entry() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "add-dhcp-entry: network=${NETWORK_ID} mac=${MAC} ip=${VM_IP} hostname=${HOSTNAME}" + [ -z "${MAC}" ] && die "add-dhcp-entry: missing --mac" + [ -z "${VM_IP}" ] && die "add-dhcp-entry: missing --ip" + + local dhcp_hosts; dhcp_hosts=$(_dnsmasq_dhcp_hosts) + mkdir -p "$(_dnsmasq_dir)" + touch "${dhcp_hosts}" + + # Normalize MAC for use as a dnsmasq tag (colons → underscores, lowercase) + local mac_tag; mac_tag=$(echo "${MAC}" | tr ':' '_' | tr '[:upper:]' '[:lower:]') + + # Remove any existing entry for this MAC (with or without a tag prefix) + grep -v "${MAC}" "${dhcp_hosts}" > "${dhcp_hosts}.tmp" 2>/dev/null || true + mv "${dhcp_hosts}.tmp" "${dhcp_hosts}" + + if [ "${DEFAULT_NIC}" = "false" ]; then + # Non-default NIC: tag the host so we can suppress the gateway option. + # dnsmasq set: in dhcp-hosts assigns the tag for this MAC. + if [ -n "${HOSTNAME}" ]; then + echo "set:norouter_${mac_tag},${MAC},${VM_IP},${HOSTNAME},infinite" >> "${dhcp_hosts}" + else + echo "set:norouter_${mac_tag},${MAC},${VM_IP},infinite" >> "${dhcp_hosts}" + fi + # Suppress option 3 (default gateway) for this specific MAC so the VM + # does not get a competing default route via this secondary NIC. + local dhcp_opts; dhcp_opts=$(_dnsmasq_dhcp_opts) + touch "${dhcp_opts}" + grep -v "tag:norouter_${mac_tag}" "${dhcp_opts}" > "${dhcp_opts}.tmp" 2>/dev/null || true + mv "${dhcp_opts}.tmp" "${dhcp_opts}" + echo "dhcp-option=tag:norouter_${mac_tag},option:router,0.0.0.0" >> "${dhcp_opts}" + log "add-dhcp-entry: non-default NIC ${MAC} (${VM_IP}) — gateway suppressed for this NIC" + else + if [ -n "${HOSTNAME}" ]; then + echo "${MAC},${VM_IP},${HOSTNAME},infinite" >> "${dhcp_hosts}" + else + echo "${MAC},${VM_IP},infinite" >> "${dhcp_hosts}" + fi + fi + + _svc_start_or_reload_dnsmasq + release_lock + log "add-dhcp-entry: done mac=${MAC} ip=${VM_IP}" +} + +############################################################################## +# Command: remove-dhcp-entry +# Remove a static DHCP host entry from dnsmasq. +############################################################################## + +cmd_remove_dhcp_entry() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "remove-dhcp-entry: network=${NETWORK_ID} mac=${MAC}" + [ -z "${MAC}" ] && die "remove-dhcp-entry: missing --mac" + + local dhcp_hosts; dhcp_hosts=$(_dnsmasq_dhcp_hosts) + if [ -f "${dhcp_hosts}" ]; then + grep -v "${MAC}" "${dhcp_hosts}" > "${dhcp_hosts}.tmp" 2>/dev/null || true + mv "${dhcp_hosts}.tmp" "${dhcp_hosts}" + # Also remove any per-MAC gateway-suppression option + local mac_tag; mac_tag=$(echo "${MAC}" | tr ':' '_' | tr '[:upper:]' '[:lower:]') + local dhcp_opts; dhcp_opts=$(_dnsmasq_dhcp_opts) + if [ -f "${dhcp_opts}" ]; then + grep -v "tag:norouter_${mac_tag}" "${dhcp_opts}" > "${dhcp_opts}.tmp" 2>/dev/null || true + mv "${dhcp_opts}.tmp" "${dhcp_opts}" + fi + _svc_start_or_reload_dnsmasq + fi + release_lock + log "remove-dhcp-entry: done mac=${MAC}" +} + +############################################################################## +# Command: add-dns-entry +# Add a hostname→IP mapping to dnsmasq. +############################################################################## + +cmd_add_dns_entry() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "add-dns-entry: network=${NETWORK_ID} hostname=${HOSTNAME} ip=${VM_IP}" + [ -z "${VM_IP}" ] && die "add-dns-entry: missing --ip" + [ -z "${HOSTNAME}" ] && die "add-dns-entry: missing --hostname" + + local hosts; hosts=$(_dnsmasq_hosts) + mkdir -p "$(_dnsmasq_dir)" + touch "${hosts}" + + # Remove existing entry for this IP then append fresh + grep -v "^${VM_IP}[[:space:]]" "${hosts}" > "${hosts}.tmp" 2>/dev/null || true + mv "${hosts}.tmp" "${hosts}" + echo "${VM_IP} ${HOSTNAME}" >> "${hosts}" + + # Also add AAAA record when an IPv6 address is provided for this NIC + if [ -n "${NIC_IP6_ADDRESS}" ]; then + grep -v "^${NIC_IP6_ADDRESS}[[:space:]]" "${hosts}" > "${hosts}.tmp" 2>/dev/null || true + mv "${hosts}.tmp" "${hosts}" + echo "${NIC_IP6_ADDRESS} ${HOSTNAME}" >> "${hosts}" + fi + + _svc_start_or_reload_dnsmasq + release_lock + log "add-dns-entry: done ${VM_IP} ${HOSTNAME}" +} + +############################################################################## +# Command: remove-dns-entry +# Remove a hostname→IP mapping from dnsmasq. +############################################################################## + +cmd_remove_dns_entry() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "remove-dns-entry: network=${NETWORK_ID} ip=${VM_IP}" + [ -z "${VM_IP}" ] && die "remove-dns-entry: missing --ip" + + local hosts; hosts=$(_dnsmasq_hosts) + if [ -f "${hosts}" ]; then + grep -v "^${VM_IP}[[:space:]]" "${hosts}" > "${hosts}.tmp" 2>/dev/null || true + mv "${hosts}.tmp" "${hosts}" + # Also remove the AAAA record when an IPv6 address was associated + if [ -n "${NIC_IP6_ADDRESS}" ]; then + grep -v "^${NIC_IP6_ADDRESS}[[:space:]]" "${hosts}" > "${hosts}.tmp" 2>/dev/null || true + mv "${hosts}.tmp" "${hosts}" + fi + _svc_start_or_reload_dnsmasq + fi + release_lock + log "remove-dns-entry: done ${VM_IP}" +} + +############################################################################## +# Command: prepare-nic +# Called when a VM NIC is being attached to the network (before VM boots). +# Idempotently adds DHCP and DNS entries so the NIC is reachable as soon +# as the VM starts. A no-op when dnsmasq is not yet configured. +############################################################################## + +cmd_prepare_nic() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "prepare-nic: network=${NETWORK_ID} ns=${NAMESPACE} guest_type=${GUEST_TYPE} mac=${MAC} ip=${VM_IP} hostname=${HOSTNAME}" + + # ---- Shared network: lazily implement on first NIC attach ---- + # For Shared networks implement-network is not called at deploy time + # (no dedicated gateway / no NAT), so the namespace + bridge + veth + # must be created here the first time a VM NIC is attached. + if [ "${GUEST_TYPE}" = "shared" ]; then + local nsd; nsd=$(_net_state_dir) + if [ ! -f "${nsd}/vlan" ]; then + log "prepare-nic: shared network — running implement-network for network=${NETWORK_ID}" + release_lock + cmd_implement_network "$@" + acquire_lock "${NETWORK_ID}" + _load_state + fi + fi + + local dnsmasq_reloaded="false" + + # ---- DHCP entry ---- + local dhcp_hosts; dhcp_hosts=$(_dnsmasq_dhcp_hosts) + if [ -n "${MAC}" ] && [ -n "${VM_IP}" ] && [ -f "${dhcp_hosts}" ]; then + local mac_tag; mac_tag=$(echo "${MAC}" | tr ':' '_' | tr '[:upper:]' '[:lower:]') + grep -v "${MAC}" "${dhcp_hosts}" > "${dhcp_hosts}.tmp" 2>/dev/null || true + mv "${dhcp_hosts}.tmp" "${dhcp_hosts}" + if [ "${DEFAULT_NIC}" = "false" ]; then + if [ -n "${HOSTNAME}" ]; then + echo "set:norouter_${mac_tag},${MAC},${VM_IP},${HOSTNAME},infinite" >> "${dhcp_hosts}" + else + echo "set:norouter_${mac_tag},${MAC},${VM_IP},infinite" >> "${dhcp_hosts}" + fi + local dhcp_opts; dhcp_opts=$(_dnsmasq_dhcp_opts) + touch "${dhcp_opts}" + grep -v "tag:norouter_${mac_tag}" "${dhcp_opts}" > "${dhcp_opts}.tmp" 2>/dev/null || true + mv "${dhcp_opts}.tmp" "${dhcp_opts}" + echo "dhcp-option=tag:norouter_${mac_tag},option:router,0.0.0.0" >> "${dhcp_opts}" + log "prepare-nic: non-default NIC ${MAC} (${VM_IP}) — gateway suppressed" + else + if [ -n "${HOSTNAME}" ]; then + echo "${MAC},${VM_IP},${HOSTNAME},infinite" >> "${dhcp_hosts}" + else + echo "${MAC},${VM_IP},infinite" >> "${dhcp_hosts}" + fi + fi + dnsmasq_reloaded="true" + fi + + # ---- DNS entry ---- + local hosts; hosts=$(_dnsmasq_hosts) + if [ -n "${VM_IP}" ] && [ -n "${HOSTNAME}" ] && [ -f "${hosts}" ]; then + grep -v "^${VM_IP}[[:space:]]" "${hosts}" > "${hosts}.tmp" 2>/dev/null || true + mv "${hosts}.tmp" "${hosts}" + echo "${VM_IP} ${HOSTNAME}" >> "${hosts}" + dnsmasq_reloaded="true" + fi + + [ "${dnsmasq_reloaded}" = "true" ] && _svc_start_or_reload_dnsmasq + + release_lock + log "prepare-nic: done network=${NETWORK_ID} mac=${MAC} ip=${VM_IP}" +} + +############################################################################## +# Command: release-nic +# Called when a VM NIC is being detached from the network (after VM stops). +# Removes DHCP and DNS entries and cleans up per-VM metadata files. +############################################################################## + +cmd_release_nic() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + log "release-nic: network=${NETWORK_ID} ns=${NAMESPACE} mac=${MAC} ip=${VM_IP}" + + local dnsmasq_reloaded="false" + + # ---- Remove DHCP entry ---- + local dhcp_hosts; dhcp_hosts=$(_dnsmasq_dhcp_hosts) + if [ -n "${MAC}" ] && [ -f "${dhcp_hosts}" ]; then + local mac_tag; mac_tag=$(echo "${MAC}" | tr ':' '_' | tr '[:upper:]' '[:lower:]') + grep -v "${MAC}" "${dhcp_hosts}" > "${dhcp_hosts}.tmp" 2>/dev/null || true + mv "${dhcp_hosts}.tmp" "${dhcp_hosts}" + local dhcp_opts; dhcp_opts=$(_dnsmasq_dhcp_opts) + if [ -f "${dhcp_opts}" ]; then + grep -v "tag:norouter_${mac_tag}" "${dhcp_opts}" > "${dhcp_opts}.tmp" 2>/dev/null || true + mv "${dhcp_opts}.tmp" "${dhcp_opts}" + fi + dnsmasq_reloaded="true" + fi + + # ---- Remove DNS entry ---- + local hosts; hosts=$(_dnsmasq_hosts) + if [ -n "${VM_IP}" ] && [ -f "${hosts}" ]; then + grep -v "^${VM_IP}[[:space:]]" "${hosts}" > "${hosts}.tmp" 2>/dev/null || true + mv "${hosts}.tmp" "${hosts}" + dnsmasq_reloaded="true" + fi + + [ "${dnsmasq_reloaded}" = "true" ] && _svc_start_or_reload_dnsmasq + + # ---- Remove per-VM metadata ---- + if [ -n "${VM_IP}" ]; then + local vm_meta_dir; vm_meta_dir="$(_metadata_dir)/${VM_IP}" + if [ -d "${vm_meta_dir}" ]; then + rm -rf "${vm_meta_dir}" + log "release-nic: removed metadata for ${VM_IP}" + fi + # Remove password entry + local passwd_f; passwd_f=$(_passwd_file) + if [ -f "${passwd_f}" ]; then + grep -v "^${VM_IP}=" "${passwd_f}" > "${passwd_f}.tmp" 2>/dev/null || true + mv "${passwd_f}.tmp" "${passwd_f}" + fi + fi + + release_lock + log "release-nic: done network=${NETWORK_ID} mac=${MAC} ip=${VM_IP}" +} + +############################################################################## +# Command: save-userdata +# Write user-data for a VM; start/reload apache2. +############################################################################## + +cmd_save_userdata() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary apache2 httpd + log "save-userdata: network=${NETWORK_ID} ip=${VM_IP}" + [ -z "${VM_IP}" ] && die "save-userdata: missing --ip" + + local vm_dir; vm_dir="$(_metadata_dir)/${VM_IP}/latest" + mkdir -p "${vm_dir}" + + if [ -n "${USERDATA}" ]; then + printf '%s' "${USERDATA}" > "${vm_dir}/user-data" + else + # Create empty user-data file if USERDATA is empty + rm -rf "${vm_dir}/user-data" && touch "${vm_dir}/user-data" + fi + + _write_apache2_conf + _svc_start_or_reload_apache2 + release_lock + log "save-userdata: done ${VM_IP}" +} + +############################################################################## +# Command: save-password +# Write a VM password served via the metadata HTTP service. +############################################################################## + +cmd_save_password() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary apache2 httpd + log "save-password: network=${NETWORK_ID} ip=${VM_IP}" + [ -z "${VM_IP}" ] && die "save-password: missing --ip" + + # Write to the passwords file for the VR-compatible password server only. + # Format: ip=password (same as /var/cache/cloud/passwords- on VR) + local passwd_f; passwd_f=$(_passwd_file) + if [ -n "${PASSWORD}" ]; then + touch "${passwd_f}" + grep -v "^${VM_IP}=" "${passwd_f}" > "${passwd_f}.tmp" 2>/dev/null || true + mv "${passwd_f}.tmp" "${passwd_f}" + echo "${VM_IP}=${PASSWORD}" >> "${passwd_f}" + fi + + _write_apache2_conf + _svc_start_or_reload_apache2 + _svc_start_or_reload_passwd_server + release_lock + log "save-password: done ${VM_IP}" +} + +############################################################################## +# Command: save-sshkey +# Write an SSH public key for a VM. +############################################################################## + +cmd_save_sshkey() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary apache2 httpd + log "save-sshkey: network=${NETWORK_ID} ip=${VM_IP}" + [ -z "${VM_IP}" ] && die "save-sshkey: missing --ip" + + local meta_dir; meta_dir="$(_metadata_dir)/${VM_IP}/latest/meta-data" + mkdir -p "${meta_dir}" + # All public keys for the VM are stored together in a single flat file + # (one key per line) at latest/meta-data/public-keys. + printf '%s' "${SSH_KEY}" > "${meta_dir}/public-keys" + + _write_apache2_conf + _svc_start_or_reload_apache2 + release_lock + log "save-sshkey: done ${VM_IP}" +} + +############################################################################## +# Command: save-hypervisor-hostname +# Write the hypervisor hostname into the VM's meta-data. +############################################################################## + +cmd_save_hypervisor_hostname() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary apache2 httpd + log "save-hypervisor-hostname: network=${NETWORK_ID} ip=${VM_IP} host=${HYPERVISOR_HOSTNAME}" + [ -z "${VM_IP}" ] && die "save-hypervisor-hostname: missing --ip" + + local meta_dir; meta_dir="$(_metadata_dir)/${VM_IP}/latest/meta-data" + mkdir -p "${meta_dir}" + printf '%s' "${HYPERVISOR_HOSTNAME}" > "${meta_dir}/hypervisor-hostname" + + _write_apache2_conf + _svc_start_or_reload_apache2 + release_lock + log "save-hypervisor-hostname: done ${VM_IP}" +} + +############################################################################## +# Command: save-vm-data +# Write the full VM metadata/userdata/password set in one call. +# payload.vm_data JSON array of {dir,file,content} entries. +# Each 'content' value is a plain UTF-8 string. +# +# Path mapping from generateVmData() output: +# [userdata, user_data, ] → latest/user-data +# [metadata, public-keys, ] → latest/meta-data/public-keys (flat file, one key per line) +# [metadata, , ] → latest/meta-data/ +# [password, vm_password, ] → passwords file (VR-compat passwd-server only) +# [password, vm-password-md5checksum, ] → latest/meta-data/password-checksum +# +# After writing all files the apache2 metadata server and the VR-compatible +# password server on port 8080 are both started / reloaded. +############################################################################## + +cmd_save_vm_data() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary apache2 httpd + log "save-vm-data: network=${NETWORK_ID} ip=${VM_IP}" + [ -z "${VM_IP}" ] && die "save-vm-data: missing --ip" + + local vm_data_file="" + local cleanup_vm_data_file="false" + if [ -z "${vm_data_file}" ]; then + [ -z "${VM_DATA}" ] && die "save-vm-data: missing payload.vm_data" + vm_data_file=$(mktemp /tmp/cs-extnet-vm-data-XXXXXX) + cleanup_vm_data_file="true" + printf '%s' "${VM_DATA}" > "${vm_data_file}" + fi + [ -f "${vm_data_file}" ] || die "save-vm-data: payload file not found: ${vm_data_file}" + + local meta_dir; meta_dir=$(_metadata_dir) + local passwd_f; passwd_f=$(_passwd_file) + mkdir -p "${meta_dir}" "$(dirname "${passwd_f}")" + touch "${passwd_f}" + + python3 - "${VM_IP}" "${meta_dir}" "${passwd_f}" "${vm_data_file}" << 'PYEOF' +import json, os, sys + +vm_ip = sys.argv[1] +meta_dir = sys.argv[2] +passwd_f = sys.argv[3] +data_file = sys.argv[4] + +try: + with open(data_file, 'r', encoding='utf-8') as f: + data_json = f.read().strip() +except Exception as e: + print(f"save-vm-data: failed to read vm-data file: {e}", file=sys.stderr) + sys.exit(1) + +# Parse the JSON array directly (content is plain text, not base64-encoded) +try: + entries = json.loads(data_json) +except Exception as e: + print(f"save-vm-data: failed to parse vm-data: {e}", file=sys.stderr) + sys.exit(1) + +password_written = None +pub_keys = [] # accumulate all public-key entries; written as one flat file after the loop + +for entry in entries: + d = entry.get('dir', '') + f = entry.get('file', '') + c = entry.get('content', '') + if not c: + continue + content = c.encode('utf-8') if isinstance(c, str) else c + + if not content: + continue + + # ---- path mapping ---- + if d == 'userdata' and f == 'user_data': + path = os.path.join(meta_dir, vm_ip, 'latest', 'user-data') + elif d == 'metadata' and f == 'public-keys': + # All public keys are collected and written together as a single + # flat file (one key per line) after the loop. + pub_keys.append(content.rstrip(b'\n') + b'\n') + continue + elif d == 'metadata': + path = os.path.join(meta_dir, vm_ip, 'latest', 'meta-data', f) + elif d == 'password' and f == 'vm_password': + # Only write to the VR-compatible passwords file; do NOT save to + # latest/password inside the metadata tree. + password_written = content.decode('utf-8', errors='replace').strip() + continue + elif d == 'password' and f == 'vm-password-md5checksum': + path = os.path.join(meta_dir, vm_ip, 'latest', 'meta-data', 'password-checksum') + else: + # Fallback: store under latest// + path = os.path.join(meta_dir, vm_ip, 'latest', d, f) + + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, 'wb') as fp: + fp.write(content if isinstance(content, bytes) else content.encode('utf-8')) + +# Write all collected public keys into a single flat file. +if pub_keys: + pk_path = os.path.join(meta_dir, vm_ip, 'latest', 'meta-data', 'public-keys') + os.makedirs(os.path.dirname(pk_path), exist_ok=True) + with open(pk_path, 'wb') as fp: + fp.writelines(pub_keys) + +# Update the passwords file (ip=password format, same as VR) +if password_written: + try: + lines = [] + try: + with open(passwd_f) as pf: + lines = [l for l in pf if l.strip() and not l.startswith(vm_ip + '=')] + except FileNotFoundError: + pass + lines.append(f'{vm_ip}={password_written}\n') + with open(passwd_f, 'w') as pf: + pf.writelines(lines) + except Exception as e: + print(f"save-vm-data: could not update passwords file: {e}", file=sys.stderr) + +print(f"save-vm-data: wrote {len(entries)} entries for {vm_ip}") +PYEOF + + if [ "${cleanup_vm_data_file}" = "true" ]; then + rm -f "${vm_data_file}" 2>/dev/null || true + fi + + _write_apache2_conf + _svc_start_or_reload_apache2 + _svc_start_or_reload_passwd_server + release_lock + log "save-vm-data: done network=${NETWORK_ID} ip=${VM_IP}" +} + + +############################################################################## +# Command: apply-fw-rules +# +# Rebuilds the per-network firewall chain CS_EXTNET_FWRULES_ from +# scratch using the JSON object supplied in payload.fw_rules. +# +# JSON payload structure (payload.fw_rules): +# { +# "default_egress_allow": true|false, +# "cidr": "10.0.0.0/24", +# "rules": [ +# { +# "id": 1, +# "type": "ingress"|"egress", +# "protocol": "tcp"|"udp"|"icmp"|"all", +# "portStart": , // optional +# "portEnd": , // optional +# "icmpType": , // optional +# "icmpCode": , // optional +# "publicIp": "1.2.3.4", // ingress only +# "sourceCidrs": ["0.0.0.0/0", ...], +# "destCidrs": ["0.0.0.0/0", ...] // egress only, optional +# }, ... +# ] +# } +# +# iptables design (two-part, mirroring the VR FIREWALL_ pattern) +# ---------------------------------------------------------------------- +# +# PART 1 – Ingress: mangle table, PREROUTING hook (BEFORE nat DNAT) +# +# Per-public-IP chains CS_EXTNET_FWI_ in the mangle table: +# -A PREROUTING -d /32 -j CS_EXTNET_FWI_ +# -A CS_EXTNET_FWI_ [explicit allow rules] -j RETURN +# -A CS_EXTNET_FWI_ -m state --state RELATED,ESTABLISHED -j RETURN +# -A CS_EXTNET_FWI_ -j DROP +# +# Checking in PREROUTING mangle (before DNAT) lets us match directly on the +# public destination IP — works for static-NAT, port-forwarding, and LB alike. +# +# PART 2 – Egress: filter table, FORWARD hook (CS_EXTNET_FWRULES_) +# +# CS_EXTNET_FWD_ (existing "fchain") +# └─[pos 1]─> CS_EXTNET_FWRULES_ (fw_chain, egress only) +# 1. RELATED,ESTABLISHED → ACCEPT +# 2. explicit egress rules (-i ... -j ACCEPT|DROP) +# 3. default egress policy (-i ... -j ACCEPT|DROP) +# 4. -j RETURN +# +# Default egress policy: +# default_egress_allow=true → ALLOW by default; explicit egress rules are DROP +# default_egress_allow=false → DENY by default; explicit egress rules are ACCEPT +############################################################################## + +cmd_apply_fw_rules() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _guard_ns_teardown "apply-fw-rules" + log "apply-fw-rules: network=${NETWORK_ID} ns=${NAMESPACE}" + + local fw_rules_file="" + local cleanup_fw_rules_file="false" + if [ -z "${fw_rules_file}" ]; then + fw_rules_file=$(mktemp /tmp/cs-extnet-fw-rules-XXXXXX) + cleanup_fw_rules_file="true" + printf '%s' "${FW_RULES_JSON:-}" > "${fw_rules_file}" + fi + [ -f "${fw_rules_file}" ] || die "apply-fw-rules: payload file not found: ${fw_rules_file}" + + local veth_n fchain fw_chain + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + fchain=$(filter_chain "${NETWORK_ID}") + fw_chain=$(firewall_chain "${NETWORK_ID}") + + # ---- 1. Remove existing jump from fchain to fw_chain (idempotent) ---- + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -j "${fw_chain}" 2>/dev/null || true + + # ---- 2. Flush and delete old fw chain ---- + ip netns exec "${NAMESPACE}" iptables -t filter -F "${fw_chain}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter -X "${fw_chain}" 2>/dev/null || true + + # ---- 3. Create fresh fw chain ---- + ip netns exec "${NAMESPACE}" iptables -t filter -N "${fw_chain}" + + # ---- 4. Build iptables rules via Python ---- + python3 - "${NAMESPACE}" "${fw_rules_file}" "${veth_n}" \ + "${fw_chain}" "$(_vpc_state_dir)" << 'PYEOF' +import json, os, re, subprocess, sys + +namespace = sys.argv[1] +rules_file = sys.argv[2] +veth_n = sys.argv[3] +fw_chain = sys.argv[4] # filter table egress chain (CS_EXTNET_FWRULES_) +state_dir = sys.argv[5] # vpc-or-network state directory for static-nat entries + +try: + with open(rules_file, 'r', encoding='utf-8') as f: + rules_json = f.read().strip() +except Exception as e: + print(f"apply-fw-rules: failed to read rules file: {e}", file=sys.stderr) + sys.exit(1) + +# Prefix for per-public-IP ingress chains in the mangle table. +# e.g. CS_EXTNET_FWI_10.0.56.20 +FW_INGRESS_PREFIX = 'CS_EXTNET_FWI_' + +def _run(table, *args): + cmd = ['ip', 'netns', 'exec', namespace, 'iptables', '-t', table] + list(args) + r = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if r.returncode != 0: + print(f"iptables ({table}): {r.stderr.decode().strip()}", file=sys.stderr) + return r + +def iptf(*args): + """iptables filter table.""" + _run('filter', *args) + +def iptm(*args): + """iptables mangle table.""" + _run('mangle', *args) + +# --------------------------------------------------------------------------- +# Parse payload +# --------------------------------------------------------------------------- +if rules_json: + try: + data = json.loads(rules_json) + except Exception as e: + print(f"apply-fw-rules: failed to parse fw-rules: {e}", file=sys.stderr) + sys.exit(1) +else: + data = {} + +default_egress_allow = data.get('default_egress_allow', True) +rules = data.get('rules', []) + +ingress_rules = [r for r in rules if r.get('type') == 'ingress'] +egress_rules = [r for r in rules if r.get('type') == 'egress'] + +# --------------------------------------------------------------------------- +# PART 1 – Egress rules in filter table (CS_EXTNET_FWRULES_) +# +# Handles only outbound guest-VM traffic (-i ). +# --------------------------------------------------------------------------- + +# Explicit egress rules. +# default_egress_allow=true → explicit rules are DROP (deny specific traffic) +# default_egress_allow=false → explicit rules are ACCEPT (allow specific traffic) +rule_target = 'DROP' if default_egress_allow else 'ACCEPT' + +for rule in egress_rules: + protocol = (rule.get('protocol') or 'all').lower() + port_start = rule.get('portStart') + port_end = rule.get('portEnd') + icmp_type = rule.get('icmpType') + icmp_code = rule.get('icmpCode') + src_cidrs = rule.get('sourceCidrs') or ['0.0.0.0/0'] + dest_cidrs = rule.get('destCidrs') or [] + + for src_cidr in src_cidrs: + a = ['-i', veth_n] + if src_cidr and src_cidr not in ('0.0.0.0/0', '::/0', ''): + a += ['-s', src_cidr] + if protocol not in ('all', ''): + a += ['-p', protocol] + if protocol in ('tcp', 'udp') and port_start is not None: + port_spec = str(port_start) + if port_end is not None and port_end != port_start: + port_spec = f"{port_start}:{port_end}" + a += ['--dport', port_spec] + elif protocol == 'icmp' and icmp_type is not None and icmp_type != -1: + icmp_spec = str(icmp_type) + if icmp_code is not None and icmp_code != -1: + icmp_spec += f"/{icmp_code}" + a += ['--icmp-type', icmp_spec] + + if dest_cidrs: + for dest_cidr in dest_cidrs: + if dest_cidr and dest_cidr not in ('0.0.0.0/0', '::/0', ''): + iptf('-A', fw_chain, *a, '-d', dest_cidr, '-j', rule_target) + else: + iptf('-A', fw_chain, *a, '-j', rule_target) + else: + iptf('-A', fw_chain, *a, '-j', rule_target) + +# Default egress policy (NEW connections only — RELATED/ESTABLISHED already ACCEPTed). +if not default_egress_allow: + iptf('-A', fw_chain, '-i', veth_n, '-m', 'state', '--state', 'NEW', '-j', 'DROP') +else: + iptf('-A', fw_chain, '-i', veth_n, '-j', 'ACCEPT') + +# RETURN so that fchain catch-all rules (static-NAT, PF, etc.) remain active +# for any non-VM traffic that reaches this chain. +iptf('-A', fw_chain, '-j', 'RETURN') + +# Insert established/related last so it lands at position 1 regardless of what +# was appended above — ongoing sessions must never be re-evaluated against the +# explicit egress rules/default policy, no matter how this chain is built. +iptf('-I', fw_chain, '1', '-m', 'state', '--state', 'RELATED,ESTABLISHED', '-j', 'ACCEPT') + +# --------------------------------------------------------------------------- +# PART 2 – Ingress firewall in mangle table, PREROUTING hook (before DNAT) +# +# Mirrors the VR's FIREWALL_ chains in the mangle table: +# -A PREROUTING -d /32 -j CS_EXTNET_FWI_ +# -A CS_EXTNET_FWI_ [explicit allow rules] -j RETURN +# -A CS_EXTNET_FWI_ -m state --state RELATED,ESTABLISHED -j RETURN +# -A CS_EXTNET_FWI_ -j DROP +# +# Running in PREROUTING mangle (before nat PREROUTING DNAT) means we match +# on the real public destination IP directly — no conntrack tricks needed. +# Works uniformly for static-NAT, port-forwarding, and LB public IPs. +# --------------------------------------------------------------------------- + +# Step 1: discover and flush all existing CS_EXTNET_FWI_* chains (full rebuild). +ls_out = _run('mangle', '-n', '-L').stdout.decode('utf-8', errors='replace') +old_chains = re.findall( + r'^Chain (' + re.escape(FW_INGRESS_PREFIX) + r'[\d.]+) ', + ls_out, re.MULTILINE) +for old_chain in old_chains: + old_ip = old_chain[len(FW_INGRESS_PREFIX):] + iptm('-D', 'PREROUTING', '-d', f'{old_ip}/32', '-j', old_chain) + iptm('-F', old_chain) + iptm('-X', old_chain) + +# Step 2: build a per-IP chain for every public IP that has ingress rules. +pub_ip_rules = {} +for rule in ingress_rules: + pub_ip = rule.get('publicIp', '') + if not pub_ip: + continue + pub_ip_rules.setdefault(pub_ip, []).append(rule) + +for pub_ip, ip_rules in pub_ip_rules.items(): + chain_name = FW_INGRESS_PREFIX + pub_ip # e.g. CS_EXTNET_FWI_10.0.56.20 + + # Create chain and register it in mangle PREROUTING. + iptm('-N', chain_name) + iptm('-A', 'PREROUTING', '-d', f'{pub_ip}/32', '-j', chain_name) + + # Explicit ALLOW rules — RETURN lets the packet proceed to DNAT and FORWARD. + for rule in ip_rules: + protocol = (rule.get('protocol') or 'all').lower() + port_start = rule.get('portStart') + port_end = rule.get('portEnd') + icmp_type = rule.get('icmpType') + icmp_code = rule.get('icmpCode') + src_cidrs = rule.get('sourceCidrs') or ['0.0.0.0/0'] + + for src_cidr in src_cidrs: + a = ['-A', chain_name] + if src_cidr and src_cidr not in ('0.0.0.0/0', '::/0', ''): + a += ['-s', src_cidr] + if protocol not in ('all', ''): + a += ['-p', protocol] + if protocol in ('tcp', 'udp') and port_start is not None: + port_spec = str(port_start) + if port_end is not None and port_end != port_start: + port_spec = f"{port_start}:{port_end}" + a += ['--dport', port_spec] + elif protocol == 'icmp' and icmp_type is not None and icmp_type != -1: + icmp_spec = str(icmp_type) + if icmp_code is not None and icmp_code != -1: + icmp_spec += f"/{icmp_code}" + a += ['--icmp-type', icmp_spec] + a += ['-j', 'RETURN'] + iptm(*a) + + # Default: drop new connections not matched by any explicit rule above. + iptm('-A', chain_name, '-j', 'DROP') + # Insert established/related last so it lands at position 1 regardless of + # what was appended above — active sessions must never be re-evaluated + # against the explicit rules/deny-all, no matter how this chain is built. + iptm('-I', chain_name, '1', '-m', 'state', '--state', 'RELATED,ESTABLISHED', '-j', 'RETURN') + +# Step 3: create default-DROP chains for static-NAT IPs that have no ingress +# rules in this invocation. Without this, removing all firewall rules for a +# static-NAT IP leaves no mangle chain for that IP, so inbound traffic is never +# filtered and the VM remains reachable despite having no allowed firewall rules. +static_nat_dir = os.path.join(state_dir, 'static-nat') +n_protected = 0 +if os.path.isdir(static_nat_dir): + for fn in os.listdir(static_nat_dir): + if not re.match(r'^\d+\.\d+\.\d+\.\d+$', fn): + continue + pub_ip = fn + if pub_ip in pub_ip_rules: + continue # already has an explicit chain built above + chain_name = FW_INGRESS_PREFIX + pub_ip + iptm('-N', chain_name) + iptm('-A', 'PREROUTING', '-d', f'{pub_ip}/32', '-j', chain_name) + iptm('-A', chain_name, '-j', 'DROP') + # Insert established/related last so it lands at position 1 (see the + # main pub_ip_rules loop above for why). + iptm('-I', chain_name, '1', '-m', 'state', '--state', 'RELATED,ESTABLISHED', '-j', 'RETURN') + n_protected += 1 + +n_in = sum(len(v) for v in pub_ip_rules.values()) +n_eg = len(egress_rules) +policy = 'ALLOW' if default_egress_allow else 'DENY' +print(f"apply-fw-rules: built {n_in} ingress rule(s) across {len(pub_ip_rules)} public IP(s), " + f"{n_eg} egress rule(s), default_egress={policy}, " + f"default-DROP chains for {n_protected} static-NAT IP(s) with no rules") +PYEOF + + local py_exit=$? + + if [ "${cleanup_fw_rules_file}" = "true" ]; then + rm -f "${fw_rules_file}" 2>/dev/null || true + fi + + if [ ${py_exit} -ne 0 ]; then + # Python script failed — leave chain empty but continue so that the + # fchain catch-all rules remain effective (fail-open for existing traffic). + log "apply-fw-rules: Python rule builder exited ${py_exit}; firewall chain may be incomplete" + fi + + # ---- 5. Insert jump from fchain to fw_chain at position 1 ---- + # Runs fw_chain BEFORE the fchain catch-all ACCEPT rules so the firewall + # policy takes precedence. Skipped when fchain does not exist yet (i.e. + # implement has not been called — the jump will be inserted on next + # apply-fw-rules invocation after implement). + if ip netns exec "${NAMESPACE}" iptables -t filter -n -L "${fchain}" >/dev/null 2>&1; then + ip netns exec "${NAMESPACE}" iptables -t filter \ + -I "${fchain}" 1 -j "${fw_chain}" 2>/dev/null || true + log "apply-fw-rules: jump ${fchain} -> ${fw_chain} inserted at position 1" + else + log "apply-fw-rules: WARNING — ${fchain} not found; jump will be inserted on next implement" + fi + + _dump_iptables "${NAMESPACE}" + release_lock + log "apply-fw-rules: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: apply-lb-rules +# Apply/revoke load balancing rules via haproxy inside the namespace. +# --lb-rules — array of LB rule objects (see Java side for schema) +############################################################################## + +cmd_apply_lb_rules() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary haproxy + log "apply-lb-rules: network=${NETWORK_ID} ns=${NAMESPACE}" + + # Normalise empty input + [ -z "${LB_RULES_JSON}" ] && LB_RULES_JSON="[]" + + local lb_dir; lb_dir=$(_haproxy_dir) + mkdir -p "${lb_dir}" + + # Persist/remove per-rule state files + python3 - "${lb_dir}" "${LB_RULES_JSON}" << 'PYEOF' +import json, os, sys + +lb_dir = sys.argv[1] +rules = json.loads(sys.argv[2]) + +for rule in rules: + rid = str(rule['id']) + fn = os.path.join(lb_dir, f"{rid}.json") + if rule.get('revoke', False): + try: + os.remove(fn) + except FileNotFoundError: + pass + else: + with open(fn, 'w') as f: + json.dump(rule, f) +PYEOF + + # Regenerate haproxy config + _write_haproxy_conf + + # Count active rules (json files in lb_dir, excluding haproxy.cfg / haproxy.pid / etc.) + local active_rules + active_rules=$(find "${lb_dir}" -maxdepth 1 -name '*.json' 2>/dev/null | wc -l) + + if [ "${active_rules}" -gt 0 ]; then + _svc_reload_haproxy + else + log "apply-lb-rules: no active rules; stopping haproxy" + _svc_stop_haproxy + fi + + _dump_iptables "${NAMESPACE}" + release_lock + log "apply-lb-rules: done network=${NETWORK_ID}" +} + +############################################################################## +# Command: custom-action + +_pbr_param() { + # Return the first non-empty key from ACTION_PARAMS_JSON. + local _k _v + for _k in "$@"; do + _v=$(_json_get "${ACTION_PARAMS_JSON}" "${_k}") + if [ -n "${_v}" ]; then + echo "${_v}" + return 0 + fi + done + echo "" +} + +_pbr_table_file() { + [ -z "${NAMESPACE}" ] && die "pbr: namespace not resolved" + echo "/etc/netns/${NAMESPACE}/iproute2/rt_tables" +} + +# Ensure the per-namespace rt_tables file exists; seed from the system default. +# Works with ip netns exec which auto-bind-mounts /etc/netns//* over /etc/*. +_pbr_ensure_table_file() { + local tf + tf="$(_pbr_table_file)" + if [ ! -f "${tf}" ]; then + mkdir -p "$(dirname "${tf}")" + echo "255 local" > "${tf}" + echo "254 main" >> "${tf}" + echo "253 default" >> "${tf}" + echo "0 unspec" >> "${tf}" + log "pbr: created per-namespace table file ${tf}" + fi + if [ ! -d "/etc/iproute2" ]; then + mkdir -p /etc/iproute2/ + fi +} + +_pbr_create_table() { + local tid tname tf tmp + tid="$(_pbr_param table-id table_id id tableid)" + tname="$(_pbr_param table-name table_name name tablename table)" + [ -z "${tid}" ] && die "pbr-create-table: missing table id" + [ -z "${tname}" ] && die "pbr-create-table: missing table name" + + _pbr_ensure_table_file + tf="$(_pbr_table_file)" + grep -Eq "^[[:space:]]*${tid}[[:space:]]+${tname}([[:space:]]|$)" "${tf}" 2>/dev/null && { + echo "pbr-create-table: exists ${tid} ${tname}" + return 0 + } + + tmp=$(mktemp /tmp/cs-extnet-rt-tables-XXXXXX) + awk -v tid="${tid}" -v tname="${tname}" ' + BEGIN { done = 0 } + { + if ($0 ~ "^[[:space:]]*#" || $0 ~ "^[[:space:]]*$") { print; next } + if ($1 == tid || $2 == tname) { + if (!done) { + print tid " " tname + done = 1 + } + next + } + print + } + END { + if (!done) print tid " " tname + } + ' "${tf}" > "${tmp}" + cat "${tmp}" > "${tf}" + rm -f "${tmp}" 2>/dev/null || true + echo "pbr-create-table: OK ${tid} ${tname}" +} + +_pbr_delete_table() { + local tid tname tf tmp + tid="$(_pbr_param table-id table_id id tableid)" + tname="$(_pbr_param table-name table_name name tablename table)" + [ -z "${tid}" ] && [ -z "${tname}" ] && die "pbr-delete-table: missing table id/name" + + _pbr_ensure_table_file + tf="$(_pbr_table_file)" + tmp=$(mktemp /tmp/cs-extnet-rt-tables-XXXXXX) + awk -v tid="${tid}" -v tname="${tname}" ' + { + if ($0 ~ "^[[:space:]]*#" || $0 ~ "^[[:space:]]*$") { print; next } + if ((tid != "" && $1 == tid) || (tname != "" && $2 == tname)) { + next + } + print + } + ' "${tf}" > "${tmp}" + cat "${tmp}" > "${tf}" + rm -f "${tmp}" 2>/dev/null || true + echo "pbr-delete-table: OK id=${tid:-n/a} name=${tname:-n/a}" +} + +_pbr_list_tables() { + _pbr_ensure_table_file + awk ' + { + if ($0 ~ "^[[:space:]]*#" || $0 ~ "^[[:space:]]*$") next + print + } + ' "$(_pbr_table_file)" +} + +_pbr_add_route() { + local table route + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + route="$(_pbr_param route route-spec route_spec)" + [ -z "${table}" ] && die "pbr-add-route: missing table" + [ -z "${route}" ] && die "pbr-add-route: missing route spec" + [ -z "${NAMESPACE}" ] && die "pbr-add-route: namespace not resolved" + + # replace is idempotent and avoids duplicate route errors. + ip netns exec "${NAMESPACE}" sh -c "ip route replace ${route} table ${table}" + echo "pbr-add-route: OK table=${table} route=${route}" +} + +_pbr_delete_route() { + local table route + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + route="$(_pbr_param route route-spec route_spec)" + [ -z "${table}" ] && die "pbr-delete-route: missing table" + [ -z "${route}" ] && die "pbr-delete-route: missing route spec" + [ -z "${NAMESPACE}" ] && die "pbr-delete-route: namespace not resolved" + + ip netns exec "${NAMESPACE}" sh -c "ip route del ${route} table ${table}" 2>/dev/null || true + echo "pbr-delete-route: OK table=${table} route=${route}" +} + +_pbr_list_routes() { + local table + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + [ -z "${NAMESPACE}" ] && die "pbr-list-routes: namespace not resolved" + if [ -n "${table}" ]; then + ip netns exec "${NAMESPACE}" ip route show table "${table}" + else + ip netns exec "${NAMESPACE}" ip route show table all + fi +} + +_pbr_add_rule() { + local table rule + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + rule="$(_pbr_param rule rule-spec rule_spec)" + [ -z "${table}" ] && die "pbr-add-rule: missing table" + [ -z "${rule}" ] && die "pbr-add-rule: missing rule spec" + [ -z "${NAMESPACE}" ] && die "pbr-add-rule: namespace not resolved" + + ip netns exec "${NAMESPACE}" sh -c "ip rule add ${rule} table ${table}" 2>/dev/null || true + echo "pbr-add-rule: OK table=${table} rule=${rule}" +} + +_pbr_delete_rule() { + local table rule + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + rule="$(_pbr_param rule rule-spec rule_spec)" + [ -z "${table}" ] && die "pbr-delete-rule: missing table" + [ -z "${rule}" ] && die "pbr-delete-rule: missing rule spec" + [ -z "${NAMESPACE}" ] && die "pbr-delete-rule: namespace not resolved" + + ip netns exec "${NAMESPACE}" sh -c "ip rule del ${rule} table ${table}" 2>/dev/null || true + echo "pbr-delete-rule: OK table=${table} rule=${rule}" +} + +_pbr_list_rules() { + local table + table="$(_pbr_param table table-name table_name tablename table-id table_id id tableid)" + [ -z "${NAMESPACE}" ] && die "pbr-list-rules: namespace not resolved" + if [ -n "${table}" ]; then + ip netns exec "${NAMESPACE}" ip rule show | grep -E "[[:space:]]lookup[[:space:]]+${table}([[:space:]]|$)" || true + else + ip netns exec "${NAMESPACE}" ip rule show + fi +} + +_fw_list_rules() { + [ -z "${NAMESPACE}" ] && die "list-firewall-rules: namespace not resolved" + ip netns exec "${NAMESPACE}" iptables-save 2>/dev/null || true +} + +_pbr_emit_custom_action_result() { + local action="$1" + shift + local raw_output + raw_output="$("$@")" + + RAW_OUTPUT="${raw_output}" python3 - "${action}" << 'PYEOF' +import json +import os +import sys + +action = sys.argv[1] +raw = os.environ.get("RAW_OUTPUT", "") +rows = [line.rstrip() for line in raw.splitlines() if line.strip()] + +if action == "pbr-list-tables": + data = [] + for row in rows: + parts = row.split(None, 1) + if len(parts) == 2 and parts[0].isdigit(): + data.append({"id": parts[0], "name": parts[1]}) + else: + data.append({"result": row}) + print(json.dumps({"status": "success", "printmessage": "true", "message": data})) +elif action == "pbr-list-routes": + data = [{"route": row} for row in rows] + print(json.dumps({"status": "success", "printmessage": "true", "message": data})) +elif action == "pbr-list-rules": + data = [{"rule": row} for row in rows] + print(json.dumps({"status": "success", "printmessage": "true", "message": data})) +elif action == "list-firewall-rules": + tables, cur_table, cur_lines = [], None, [] + for line in raw.splitlines(): + line = line.rstrip() + if line.startswith('*'): + cur_table, cur_lines = line[1:], [] + elif line == 'COMMIT': + if cur_table: + tables.append({"table": cur_table, "rules": "\n".join(cur_lines)}) + cur_table = None + elif cur_table and not line.startswith('#'): + cur_lines.append(line) + print(json.dumps({"status": "success", "printmessage": "true", "message": tables})) +else: + msg = rows[0] if rows else f"{action}: OK" + print(json.dumps({"status": "success", "printmessage": "true", "message": msg})) +PYEOF +} + +cmd_custom_action() { + NETWORK_ID="" + VPC_ID="" + ACTION_NAME="" + ACTION_PARAMS_JSON="{}" + if [ $# -ge 1 ] && [ -f "$1" ]; then + local payload_file="$1" + NETWORK_ID=$(_payload_json_get "${payload_file}" "network_id") + VPC_ID=$(_payload_json_get "${payload_file}" "vpc_id") + ACTION_NAME=$(_payload_json_get "${payload_file}" "action") + ACTION_PARAMS_JSON=$(_payload_json_get "${payload_file}" "action-params") + [ -z "${ACTION_PARAMS_JSON}" ] && ACTION_PARAMS_JSON=$(_payload_json_get "${payload_file}" "action_params") + [ -z "${ACTION_PARAMS_JSON}" ] && ACTION_PARAMS_JSON="{}" + else + while [ $# -gt 0 ]; do + case "$1" in + --network-id) NETWORK_ID="$2"; shift 2 ;; + --vpc-id) VPC_ID="$2"; shift 2 ;; + --action) ACTION_NAME="$2"; shift 2 ;; + --action-params) ACTION_PARAMS_JSON="${2:-{}}"; shift 2 ;; + --physical-network-extension-details|--network-extension-details) + shift 2 ;; + *) shift ;; + esac + done + fi + [ -z "${NETWORK_ID}" ] && [ -z "${VPC_ID}" ] && die "custom-action: missing --network-id or --vpc-id" + [ -z "${ACTION_NAME}" ] && die "custom-action: missing --action" + + # Set NAMESPACE/CHOSEN_ID similar to parse_args + if [ -z "${NAMESPACE}" ]; then + if [ -n "${VPC_ID}" ]; then + NAMESPACE="cs-vpc-${VPC_ID}" + else + local NS_FROM_DETAILS + NS_FROM_DETAILS=$(_json_get "${EXTENSION_DETAILS}" "namespace") + NAMESPACE="${NS_FROM_DETAILS:-cs-net-${NETWORK_ID}}" + fi + fi + CHOSEN_ID="${VPC_ID:-${NETWORK_ID}}" + + # Ensure the namespace exists when running VPC custom actions + if [ -n "${VPC_ID}" ] && ! ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + log "custom-action: creating namespace ${NAMESPACE} for VPC ${VPC_ID}" + ip netns add "${NAMESPACE}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set lo up 2>/dev/null || true + _pbr_ensure_table_file + fi + + _load_state + acquire_lock "${CHOSEN_ID}" + + log "custom-action: network=${NETWORK_ID} ns=${NAMESPACE} action=${ACTION_NAME} params=${ACTION_PARAMS_JSON}" + + case "${ACTION_NAME}" in + reboot-device) + local veth_h veth_n + veth_h=$(veth_host_name "${VLAN}" "${CHOSEN_ID}") + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + ip link set "${veth_h}" down 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set "${veth_n}" down 2>/dev/null || true + sleep 1 + ip link set "${veth_h}" up 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set "${veth_n}" up 2>/dev/null || true + echo "reboot-device: OK (namespace=${NAMESPACE})" + ;; + dump-config) + echo "=== Namespace: ${NAMESPACE} ===" + ip netns exec "${NAMESPACE}" ip addr 2>/dev/null || echo "(no namespace)" + echo "=== Host bridge: $(host_bridge_name "${GUEST_ETH}" "${VLAN}") ===" + ip link show "$(host_bridge_name "${GUEST_ETH}" "${VLAN}")" 2>/dev/null || echo "(not found)" + echo "=== NAT table ===" + ip netns exec "${NAMESPACE}" iptables -t nat -L -n -v 2>/dev/null || echo "(unavailable)" + echo "=== FILTER table ===" + ip netns exec "${NAMESPACE}" iptables -t filter -L -n -v 2>/dev/null || echo "(unavailable)" + echo "=== Per-network state ($(_net_state_dir)) ===" + ls -la "$(_net_state_dir)/" 2>/dev/null || echo "(no network state)" + echo "=== VPC/shared state ($(_vpc_state_dir)) ===" + ls -la "$(_vpc_state_dir)/" 2>/dev/null || echo "(no vpc state)" + ;; + pbr-create-table) + _pbr_emit_custom_action_result "pbr-create-table" _pbr_create_table + ;; + pbr-delete-table) + _pbr_emit_custom_action_result "pbr-delete-table" _pbr_delete_table + ;; + pbr-list-tables) + _pbr_emit_custom_action_result "pbr-list-tables" _pbr_list_tables + ;; + pbr-add-route) + _pbr_emit_custom_action_result "pbr-add-route" _pbr_add_route + ;; + pbr-delete-route) + _pbr_emit_custom_action_result "pbr-delete-route" _pbr_delete_route + ;; + pbr-list-routes) + _pbr_emit_custom_action_result "pbr-list-routes" _pbr_list_routes + ;; + pbr-add-rule) + _pbr_emit_custom_action_result "pbr-add-rule" _pbr_add_rule + ;; + pbr-delete-rule) + _pbr_emit_custom_action_result "pbr-delete-rule" _pbr_delete_rule + ;; + pbr-list-rules) + _pbr_emit_custom_action_result "pbr-list-rules" _pbr_list_rules + ;; + list-firewall-rules) + _pbr_emit_custom_action_result "list-firewall-rules" _fw_list_rules + ;; + *) + local hook="${STATE_DIR}/hooks/custom-action-${ACTION_NAME}.sh" + if [ -x "${hook}" ]; then + exec "${hook}" --network-id "${NETWORK_ID}" --action "${ACTION_NAME}" \ + --action-params "${ACTION_PARAMS_JSON}" + else + die "Unknown action '${ACTION_NAME}'. Built-ins: reboot-device, dump-config, list-firewall-rules, pbr-*" + fi + ;; + esac + + release_lock +} + +############################################################################## +# Command: restore-network +# Batch-restore DHCP/DNS/metadata for all VMs on a network in a single call. +# Called by AggregatedCommandExecutor.completeAggregatedExecution() on network +# restart so that we rebuild all state in one shot instead of N per-VM calls. +# +# Required arguments: +# --network-id +# payload.restore_data JSON object: see buildRestoreNetworkData() in Java +# +# Optional (for dnsmasq reconfiguration): +# --gateway --cidr --dns --domain --extension-ip +############################################################################## + +cmd_restore_network() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _require_binary dnsmasq + _require_binary apache2 httpd + _require_binary haproxy + log "restore-network: network=${NETWORK_ID} ns=${NAMESPACE}" + + local restore_data_file="" + local cleanup_restore_data_file="false" + if [ -z "${restore_data_file}" ]; then + [ -z "${RESTORE_DATA}" ] && die "restore-network: missing payload.restore_data" + restore_data_file=$(mktemp /tmp/cs-extnet-restore-data-XXXXXX) + cleanup_restore_data_file="true" + printf '%s' "${RESTORE_DATA}" > "${restore_data_file}" + fi + [ -f "${restore_data_file}" ] || die "restore-network: payload file not found: ${restore_data_file}" + + local dhcp_hosts; dhcp_hosts=$(_dnsmasq_dhcp_hosts) + local dhcp_opts; dhcp_opts=$(_dnsmasq_dhcp_opts) + local dns_hosts; dns_hosts=$(_dnsmasq_hosts) + local meta_dir; meta_dir=$(_metadata_dir) + local passwd_f; passwd_f=$(_passwd_file) + + mkdir -p "$(_dnsmasq_dir)" "${meta_dir}" "$(dirname "${passwd_f}")" + touch "${dhcp_hosts}" "${dhcp_opts}" "${dns_hosts}" "${passwd_f}" + + python3 - \ + "${restore_data_file}" \ + "${dhcp_hosts}" "${dhcp_opts}" "${dns_hosts}" \ + "${meta_dir}" "${passwd_f}" \ + << 'PYEOF' +import json, os, sys + +restore_file = sys.argv[1] +dhcp_hosts_f = sys.argv[2] +dhcp_opts_f = sys.argv[3] +dns_hosts_f = sys.argv[4] +meta_dir = sys.argv[5] +passwd_f = sys.argv[6] + +try: + with open(restore_file, 'r', encoding='utf-8') as f: + restore_json = f.read().strip() +except Exception as e: + print(f"restore-network: failed to read restore-data file: {e}", file=sys.stderr) + sys.exit(1) + +# ---- Parse JSON directly ---- +try: + data = json.loads(restore_json) +except Exception as e: + print(f"restore-network: failed to parse restore-data: {e}", file=sys.stderr) + sys.exit(1) + +dhcp_enabled = data.get('dhcp_enabled', False) +dns_enabled = data.get('dns_enabled', False) +userdata_enabled = data.get('userdata_enabled', False) +vms = data.get('vms', []) + +print(f"restore-network: dhcp={dhcp_enabled} dns={dns_enabled} " + f"userdata={userdata_enabled} vms={len(vms)}") + +# ---- Rebuild DHCP hosts file ---- +if dhcp_enabled: + new_hosts_lines = [] + new_opts_lines = [] + # Keep existing custom dhcp-option lines not managed by us + try: + with open(dhcp_opts_f) as f: + existing_opts = [l for l in f if not l.startswith('set:norouter_') and + 'norouter_' not in l] + except FileNotFoundError: + existing_opts = [] + + for vm in vms: + ip = vm.get('ip', '') + mac = vm.get('mac', '') + hostname = vm.get('hostname', '') + default_nic = str(vm.get('default_nic', True)).lower() not in ('false', '0', 'no') + if not ip or not mac: + continue + mac_tag = mac.replace(':', '_').lower() + if not default_nic: + if hostname: + new_hosts_lines.append(f"set:norouter_{mac_tag},{mac},{ip},{hostname},infinite\n") + else: + new_hosts_lines.append(f"set:norouter_{mac_tag},{mac},{ip},infinite\n") + new_opts_lines.append(f"dhcp-option=tag:norouter_{mac_tag},option:router,0.0.0.0\n") + else: + if hostname: + new_hosts_lines.append(f"{mac},{ip},{hostname},infinite\n") + else: + new_hosts_lines.append(f"{mac},{ip},infinite\n") + + with open(dhcp_hosts_f, 'w') as f: + f.writelines(new_hosts_lines) + with open(dhcp_opts_f, 'w') as f: + f.writelines(existing_opts + new_opts_lines) + print(f"restore-network: wrote {len(new_hosts_lines)} DHCP entries") + +# ---- Rebuild DNS hosts file ---- +if dns_enabled: + new_dns_lines = [] + for vm in vms: + ip = vm.get('ip', '') + hostname = vm.get('hostname', '') + if not ip or not hostname: + continue + new_dns_lines.append(f"{ip} {hostname}\n") + with open(dns_hosts_f, 'w') as f: + f.writelines(new_dns_lines) + print(f"restore-network: wrote {len(new_dns_lines)} DNS entries") + +# ---- Restore per-VM metadata / userdata ---- +if userdata_enabled: + passwords = {} + # Load existing passwords to preserve non-restored entries + try: + with open(passwd_f) as pf: + for line in pf: + line = line.strip() + if '=' in line: + k, v = line.split('=', 1) + passwords[k] = v + except FileNotFoundError: + pass + + vm_count = 0 + for vm in vms: + vm_ip = vm.get('ip', '') + vm_data = vm.get('vm_data', []) + if not vm_ip or not vm_data: + continue + vm_count += 1 + pub_keys = [] # accumulate public keys; written as a single flat file after inner loop + for entry in vm_data: + d = entry.get('dir', '') + f_ = entry.get('file', '') + c = entry.get('content', '') + if not c: + continue + content = c.encode('utf-8') if isinstance(c, str) else c + + if not content: + continue + # ---- path mapping (same as cmd_save_vm_data) ---- + if d == 'userdata' and f_ == 'user_data': + path = os.path.join(meta_dir, vm_ip, 'latest', 'user-data') + elif d == 'metadata' and f_ == 'public-keys': + # Accumulate; written as a single flat file after the inner loop. + pub_keys.append(content.rstrip(b'\n') + b'\n') + continue + elif d == 'metadata': + path = os.path.join(meta_dir, vm_ip, 'latest', 'meta-data', f_) + elif d == 'password' and f_ == 'vm_password': + # Only write to the VR-compatible passwords file; do NOT save + # to latest/password inside the metadata tree. + passwords[vm_ip] = content.decode('utf-8', errors='replace').strip() + continue + elif d == 'password' and f_ == 'vm-password-md5checksum': + path = os.path.join(meta_dir, vm_ip, 'latest', 'meta-data', 'password-checksum') + else: + path = os.path.join(meta_dir, vm_ip, 'latest', d, f_) + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, 'wb') as fp: + fp.write(content if isinstance(content, bytes) + else content.encode('utf-8')) + + # Write all collected public keys as a single flat file. + if pub_keys: + pk_path = os.path.join(meta_dir, vm_ip, 'latest', 'meta-data', 'public-keys') + os.makedirs(os.path.dirname(pk_path), exist_ok=True) + with open(pk_path, 'wb') as fp: + fp.writelines(pub_keys) + + # Rewrite the passwords file atomically + try: + with open(passwd_f, 'w') as pf: + for k, v in passwords.items(): + pf.write(f'{k}={v}\n') + except Exception as e: + print(f"restore-network: could not write passwords file: {e}", file=sys.stderr) + + print(f"restore-network: restored metadata for {vm_count} VMs") + +print("restore-network: done") +PYEOF + + # ------------------------------------------------------------------ + # Decode dhcp_enabled / dns_enabled from restore_data_file so we can + # reconfigure dnsmasq correctly even when the namespace (and therefore + # dnsmasq.conf) was deleted and recreated. + # ------------------------------------------------------------------ + local _r_flags + _r_flags=$(python3 - "${restore_data_file}" 2>/dev/null << 'PYFLAGSEOF' +import json, sys, pathlib +try: + json_data = pathlib.Path(sys.argv[1]).read_text(encoding='utf-8').strip() + d = json.loads(json_data) + dhcp = 'true' if d.get('dhcp_enabled', False) else 'false' + dns = 'true' if d.get('dns_enabled', False) else 'false' + print(dhcp + ' ' + dns) +except Exception: + print('false false') +PYFLAGSEOF +) + + if [ "${cleanup_restore_data_file}" = "true" ]; then + rm -f "${restore_data_file}" 2>/dev/null || true + fi + local _r_dhcp _r_dns + _r_dhcp="${_r_flags%% *}"; _r_dhcp="${_r_dhcp:-false}" + _r_dns="${_r_flags##* }"; _r_dns="${_r_dns:-false}" + + # Re-add the data-server → extension-IP mapping when DNS is enabled. + # This entry is normally written by cmd_config_dns_subnet; we must + # reproduce it here so DNS resolution for the metadata/userdata service + # keeps working after a namespace restart with cleanup. + if [ "${_r_dns}" = "true" ]; then + local _ds_ip; _ds_ip="${EXTENSION_IP:-${GATEWAY}}" + grep -v -E "\sdata-server(\s|$)" "${dns_hosts}" > "${dns_hosts}.tmp" 2>/dev/null || true + mv "${dns_hosts}.tmp" "${dns_hosts}" + echo "${_ds_ip} data-server" >> "${dns_hosts}" + log "restore-network: added data-server (${_ds_ip}) to DNS hosts" + fi + + # Re-write dnsmasq.conf. The conf file is NOT part of the persisted + # state directory, so it is lost whenever the namespace is deleted. + # Without this call _svc_start_or_reload_dnsmasq has nothing to read + # and dnsmasq fails to start. + if [ "${_r_dhcp}" = "true" ] || [ "${_r_dns}" = "true" ]; then + _write_dnsmasq_conf "${_r_dns}" + fi + + # Reload services once for the whole batch + _svc_start_or_reload_dnsmasq + _write_apache2_conf + _svc_start_or_reload_apache2 + _svc_start_or_reload_passwd_server + + release_lock + log "restore-network: done network=${NETWORK_ID}" +} + +############################################################################## +# Helpers: parse VPC-level args (no --network-id required) +############################################################################## + +parse_vpc_args() { + local payload_file="$1" + + VPC_ID=$(_payload_json_get "${payload_file}" "payload.vpc_id") + VPC_CIDR=$(_payload_json_get "${payload_file}" "payload.vpc_cidr") + PUBLIC_IP=$(_payload_json_get "${payload_file}" "payload.public_ip") + PUBLIC_VLAN=$(_payload_json_get "${payload_file}" "payload.public_vlan") + PUBLIC_GATEWAY=$(_payload_json_get "${payload_file}" "payload.public_gateway") + PUBLIC_CIDR=$(_payload_json_get "${payload_file}" "payload.public_cidr") + SOURCE_NAT=$(_payload_json_get "${payload_file}" "payload.source_nat") + + [ -z "${SOURCE_NAT}" ] && SOURCE_NAT="false" + + [ -z "${VPC_ID}" ] && die "Missing payload.vpc_id" + + local NS_FROM_DETAILS + NS_FROM_DETAILS=$(_json_get "${EXTENSION_DETAILS}" "namespace") + NAMESPACE="${NS_FROM_DETAILS:-cs-vpc-${VPC_ID}}" + + # Normalise VLANs + if [ -n "${PUBLIC_VLAN}" ]; then + PUBLIC_VLAN=$(normalize_vlan "${PUBLIC_VLAN}") + fi +} + +############################################################################## +# Command: implement-vpc +# Creates the VPC namespace, enables IP forwarding, and optionally sets up +# VPC-level source NAT (--public-ip / --public-vlan / --source-nat true). +# State is persisted to ${STATE_DIR}/vpc-/ +############################################################################## + +cmd_implement_vpc() { + parse_vpc_args "$@" + log "implement-vpc: vpc=${VPC_ID} ns=${NAMESPACE} cidr=${VPC_CIDR}" + + # ---- 1. Create (or ensure) VPC namespace ---- + if ! ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns add "${NAMESPACE}" + log "implement-vpc: created namespace ${NAMESPACE}" + fi + ip netns exec "${NAMESPACE}" ip link set lo up 2>/dev/null || true + # Ensure per-namespace iproute2 rt_tables for PBR isolation + _pbr_ensure_table_file + + # IPv6 is managed per-tier by implement-network: each VPC tier enables or + # disables IPv6 on its own guest veth without touching sibling tiers. + + # ---- 2. IP forwarding ---- + ip netns exec "${NAMESPACE}" sysctl -w net.ipv4.ip_forward=1 >/dev/null 2>&1 || true + + # ---- 3. VPC-level source NAT (if source NAT IP provided) ---- + # Creates the public veth pair and SNAT rule for the entire VPC CIDR. + if [ -n "${PUBLIC_IP}" ] && [ -n "${PUBLIC_VLAN}" ] && [ "${SOURCE_NAT}" = "true" ] && [ -n "${VPC_CIDR}" ]; then + local pveth_h pveth_n pub_br + pveth_h=$(pub_veth_host_name "${PUBLIC_VLAN}" "${VPC_ID}") + pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${VPC_ID}") + ensure_host_bridge "${PUB_ETH}" "${PUBLIC_VLAN}" + pub_br=$(host_bridge_name "${PUB_ETH}" "${PUBLIC_VLAN}") + + if ! ip link show "${pveth_h}" >/dev/null 2>&1; then + ip link add "${pveth_h}" type veth peer name "${pveth_n}" + ip link set "${pveth_n}" netns "${NAMESPACE}" + ip link set "${pveth_h}" master "${pub_br}" + ip link set "${pveth_h}" up + ip netns exec "${NAMESPACE}" ip link set "${pveth_n}" up + log "implement-vpc: created public veth ${pveth_h} <-> ${pveth_n}" + else + ip link set "${pveth_h}" up 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set "${pveth_n}" up 2>/dev/null || true + fi + + # Assign public IP + local ADDR_SPEC + if [ -n "${PUBLIC_CIDR}" ] && echo "${PUBLIC_CIDR}" | grep -q '/'; then + local PREFIX + PREFIX=$(echo "${PUBLIC_CIDR}" | cut -d'/' -f2) + ADDR_SPEC="${PUBLIC_IP}/${PREFIX}" + else + ADDR_SPEC="${PUBLIC_IP}/32" + fi + ip netns exec "${NAMESPACE}" ip addr show "${pveth_n}" 2>/dev/null | \ + grep -q "${PUBLIC_IP}/" || \ + ip netns exec "${NAMESPACE}" ip addr add "${ADDR_SPEC}" dev "${pveth_n}" + + # Host route + ip route show | grep -q "^${PUBLIC_IP}" || \ + ip route add "${PUBLIC_IP}/32" dev "${pveth_h}" 2>/dev/null || true + + # Default route inside namespace toward upstream gateway + if [ -n "${PUBLIC_GATEWAY}" ]; then + ip netns exec "${NAMESPACE}" ip route replace default \ + via "${PUBLIC_GATEWAY}" dev "${pveth_n}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" ip route add default \ + via "${PUBLIC_GATEWAY}" dev "${pveth_n}" 2>/dev/null || true + log "implement-vpc: default route via ${PUBLIC_GATEWAY} dev ${pveth_n}" + fi + + # VPC SNAT rule — covers the entire VPC CIDR (all tiers) + # Use a VPC-level POSTROUTING chain: CS_EXTNET__VPC_POST + local vpc_post_chain="${CHAIN_PREFIX}_${VPC_ID}_VPC_POST" + ensure_chain nat "${vpc_post_chain}" + ensure_jump nat POSTROUTING "${vpc_post_chain}" + + ip netns exec "${NAMESPACE}" iptables -t nat \ + -C "${vpc_post_chain}" -s "${VPC_CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" iptables -t nat \ + -A "${vpc_post_chain}" -s "${VPC_CIDR}" -o "${pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" + log "implement-vpc: VPC SNAT ${VPC_CIDR} -> ${PUBLIC_IP} via ${pveth_n}" + + # Persist public IP state + local vsd="${STATE_DIR}/vpc-${VPC_ID}" + mkdir -p "${vsd}/ips" + echo "true" > "${vsd}/ips/${PUBLIC_IP}" + echo "${PUBLIC_VLAN}" > "${vsd}/ips/${PUBLIC_IP}.pvlan" + fi + + # ---- 4. Persist VPC state ---- + local vsd="${STATE_DIR}/vpc-${VPC_ID}" + mkdir -p "${vsd}" + echo "${NAMESPACE}" > "${vsd}/namespace" + [ -n "${VPC_CIDR}" ] && echo "${VPC_CIDR}" > "${vsd}/cidr" + + log "implement-vpc: done vpc=${VPC_ID} namespace=${NAMESPACE}" +} + +############################################################################## +# Command: update-vpc-source-nat-ip +# Updates VPC source NAT egress to a new public IP without restarting tiers. +# Reconciles public veth/IP state, default route, VPC SNAT iptables chain, +# and source NAT markers under ${STATE_DIR}/vpc-/ips/. +############################################################################## + +cmd_update_vpc_source_nat_ip() { + parse_vpc_args "$@" + acquire_lock "vpc-${VPC_ID}" + + [ -z "${PUBLIC_IP}" ] && die "update-vpc-source-nat-ip: missing --public-ip" + + local vsd="${STATE_DIR}/vpc-${VPC_ID}" + mkdir -p "${vsd}/ips" + + # Load persisted values when omitted by the caller. + if [ -z "${VPC_CIDR}" ] && [ -f "${vsd}/cidr" ]; then + VPC_CIDR=$(cat "${vsd}/cidr" 2>/dev/null || true) + fi + if [ -z "${PUBLIC_VLAN}" ] && [ -f "${vsd}/ips/${PUBLIC_IP}.pvlan" ]; then + PUBLIC_VLAN=$(cat "${vsd}/ips/${PUBLIC_IP}.pvlan" 2>/dev/null || true) + fi + + [ -z "${VPC_CIDR}" ] && die "update-vpc-source-nat-ip: missing --vpc-cidr (or persisted vpc cidr)" + [ -z "${PUBLIC_VLAN}" ] && die "update-vpc-source-nat-ip: missing --public-vlan" + + log "update-vpc-source-nat-ip: vpc=${VPC_ID} ns=${NAMESPACE} old=? new=${PUBLIC_IP} pvlan=${PUBLIC_VLAN} cidr=${VPC_CIDR}" + + local old_source_nat_ip="" + local old_public_vlan="" + local f ip flag + for f in "${vsd}/ips/"*; do + [ -f "${f}" ] || continue + ip=$(basename "${f}") + case "${ip}" in + *.pvlan|*.tier) continue ;; + esac + flag=$(cat "${f}" 2>/dev/null || true) + if [ "${flag}" = "true" ]; then + old_source_nat_ip="${ip}" + break + fi + done + + if [ -n "${old_source_nat_ip}" ] && [ -f "${vsd}/ips/${old_source_nat_ip}.pvlan" ]; then + old_public_vlan=$(cat "${vsd}/ips/${old_source_nat_ip}.pvlan" 2>/dev/null || true) + fi + + local new_pveth_h new_pveth_n pub_br + new_pveth_h=$(pub_veth_host_name "${PUBLIC_VLAN}" "${VPC_ID}") + new_pveth_n=$(pub_veth_ns_name "${PUBLIC_VLAN}" "${VPC_ID}") + ensure_host_bridge "${PUB_ETH}" "${PUBLIC_VLAN}" + pub_br=$(host_bridge_name "${PUB_ETH}" "${PUBLIC_VLAN}") + + if ! ip link show "${new_pveth_h}" >/dev/null 2>&1; then + ip link add "${new_pveth_h}" type veth peer name "${new_pveth_n}" + ip link set "${new_pveth_n}" netns "${NAMESPACE}" + ip link set "${new_pveth_h}" master "${pub_br}" + ip link set "${new_pveth_h}" up + ip netns exec "${NAMESPACE}" ip link set "${new_pveth_n}" up + log "update-vpc-source-nat-ip: created public veth ${new_pveth_h} <-> ${new_pveth_n}" + else + ip link set "${new_pveth_h}" up 2>/dev/null || true + ip netns exec "${NAMESPACE}" ip link set "${new_pveth_n}" up 2>/dev/null || true + fi + + ensure_public_ip_on_namespace "${PUBLIC_IP}" "${PUBLIC_CIDR}" "${new_pveth_n}" "${new_pveth_h}" + ip route replace "${PUBLIC_IP}/32" dev "${new_pveth_h}" 2>/dev/null || true + + if [ -n "${old_source_nat_ip}" ] && [ "${old_source_nat_ip}" != "${PUBLIC_IP}" ] && [ -n "${old_public_vlan}" ]; then + local old_pveth_n + old_pveth_n=$(pub_veth_ns_name "${old_public_vlan}" "${VPC_ID}") + if [ "${old_pveth_n}" != "${new_pveth_n}" ]; then + ip netns exec "${NAMESPACE}" ip route show default 2>/dev/null | \ + grep " dev ${old_pveth_n}\b" | \ + while read -r route; do + ip netns exec "${NAMESPACE}" ip route del ${route} 2>/dev/null || true + done + fi + fi + + if [ -n "${PUBLIC_GATEWAY}" ]; then + ip netns exec "${NAMESPACE}" ip route replace default \ + via "${PUBLIC_GATEWAY}" dev "${new_pveth_n}" 2>/dev/null || \ + ip netns exec "${NAMESPACE}" ip route add default \ + via "${PUBLIC_GATEWAY}" dev "${new_pveth_n}" 2>/dev/null || true + log "update-vpc-source-nat-ip: default route via ${PUBLIC_GATEWAY} dev ${new_pveth_n}" + fi + + local vpc_post_chain="${CHAIN_PREFIX}_${VPC_ID}_VPC_POST" + ensure_chain nat "${vpc_post_chain}" + ensure_jump nat POSTROUTING "${vpc_post_chain}" + + # This chain is dedicated to VPC source NAT egress; rebuild to a single rule. + ip netns exec "${NAMESPACE}" iptables -t nat -F "${vpc_post_chain}" + ip netns exec "${NAMESPACE}" iptables -t nat \ + -A "${vpc_post_chain}" -s "${VPC_CIDR}" -o "${new_pveth_n}" -j SNAT --to-source "${PUBLIC_IP}" + + # Keep exactly one source-NAT marker: new public IP=true, all others=false. + for f in "${vsd}/ips/"*; do + [ -f "${f}" ] || continue + ip=$(basename "${f}") + case "${ip}" in + *.pvlan|*.tier) continue ;; + esac + echo "false" > "${f}" + done + echo "true" > "${vsd}/ips/${PUBLIC_IP}" + echo "${PUBLIC_VLAN}" > "${vsd}/ips/${PUBLIC_IP}.pvlan" + + local _arping_bin + _arping_bin=$(_find_arping) || true + if [ -n "${_arping_bin}" ]; then + ip netns exec "${NAMESPACE}" "${_arping_bin}" -c 3 -U -I "${new_pveth_n}" "${PUBLIC_IP}" \ + >/dev/null 2>&1 || true + fi + + _dump_iptables "${NAMESPACE}" + release_lock + log "update-vpc-source-nat-ip: done vpc=${VPC_ID} old=${old_source_nat_ip:-none} new=${PUBLIC_IP}" +} + +############################################################################## +# Command: shutdown-vpc +# Removes the VPC namespace after all tiers have been shut down. +# Called by shutdownVpc() in NetworkExtensionElement after all tiers are gone. +############################################################################## + +cmd_shutdown_vpc() { + parse_vpc_args "$@" + log "shutdown-vpc: vpc=${VPC_ID} ns=${NAMESPACE}" + + if ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns del "${NAMESPACE}" + rm -rf "/etc/netns/${NAMESPACE}" 2>/dev/null || true + log "shutdown-vpc: deleted namespace ${NAMESPACE}" + else + log "shutdown-vpc: namespace ${NAMESPACE} not found (already removed?)" + fi + + log "shutdown-vpc: done vpc=${VPC_ID}" +} + +############################################################################## +# Command: destroy-vpc +# Destroys the VPC namespace and removes all VPC state. +############################################################################## + +cmd_destroy_vpc() { + parse_vpc_args "$@" + log "destroy-vpc: vpc=${VPC_ID} ns=${NAMESPACE}" + + if ip netns list 2>/dev/null | grep -q "^${NAMESPACE}\b"; then + ip netns del "${NAMESPACE}" + rm -rf "/etc/netns/${NAMESPACE}" 2>/dev/null || true + log "destroy-vpc: deleted namespace ${NAMESPACE}" + fi + + local vsd="${STATE_DIR}/vpc-${VPC_ID}" + rm -rf "${vsd}" + log "destroy-vpc: removed VPC state dir ${vsd}" + + log "destroy-vpc: done vpc=${VPC_ID}" +} + +############################################################################## +# Command: apply-network-acl +# Applies VPC network ACL rules to the FORWARD chain inside the namespace. +# Rules are passed as a JSON array in payload.acl_rules. +# Each rule has: +# number, action (allow|deny), trafficType (ingress|egress), +# protocol, portStart, portEnd, icmpType, icmpCode, sourceCidrs[] +############################################################################## + +cmd_apply_network_acl() { + parse_args "$@" + _load_state + acquire_lock "${NETWORK_ID}" + _guard_ns_teardown "apply-network-acl" + log "apply-network-acl: network=${NETWORK_ID} ns=${NAMESPACE} cidr=${CIDR}" + + local acl_rules_file="" + local cleanup_acl_file="false" + if [ -z "${acl_rules_file}" ]; then + acl_rules_file=$(mktemp /tmp/cs-extnet-acl-rules-XXXXXX) + cleanup_acl_file="true" + printf '%s' "${ACL_RULES_JSON:-}" > "${acl_rules_file}" + fi + [ -f "${acl_rules_file}" ] || die "apply-network-acl: payload file not found: ${acl_rules_file}" + + local veth_n acl_chain_name fchain + veth_n=$(veth_ns_name "${VLAN}" "${CHOSEN_ID}") + acl_chain_name=$(acl_chain "${NETWORK_ID}") + fchain=$(filter_chain "${NETWORK_ID}") + + # ---- 1. Remove existing jump(s) from fchain to acl chain (idempotent) ---- + # Jumps are scoped by interface (-i/-o, see step 5 below) so both forms + # must be removed here — an unscoped jump from an older run of this + # script, if one is still present, is cleared out too. If any reference + # to acl_chain_name survives this step, the -X delete in step 2 fails + # silently (chain still in use) and the -N create in step 3 then fails + # with "Chain already exists" on the next call. + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -i "${veth_n}" -j "${acl_chain_name}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -o "${veth_n}" -j "${acl_chain_name}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -D "${fchain}" -j "${acl_chain_name}" 2>/dev/null || true + + # ---- 2. Flush and delete old ACL chain ---- + ip netns exec "${NAMESPACE}" iptables -t filter -F "${acl_chain_name}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter -X "${acl_chain_name}" 2>/dev/null || true + + # ---- 3. Create fresh ACL chain ---- + ip netns exec "${NAMESPACE}" iptables -t filter -N "${acl_chain_name}" + + # ---- 4. Build iptables ACL rules via Python ---- + python3 - "${NAMESPACE}" "${acl_rules_file}" "${veth_n}" \ + "${acl_chain_name}" "${CIDR:-}" << 'PYEOF' +import json, subprocess, sys + +namespace = sys.argv[1] +rules_file = sys.argv[2] +veth_n = sys.argv[3] +acl_chain = sys.argv[4] +net_cidr = sys.argv[5] # tier CIDR (may be empty) + +try: + with open(rules_file, 'r', encoding='utf-8') as f: + rules_json = f.read().strip() +except Exception as e: + print(f"apply-network-acl: failed to read rules file: {e}", file=sys.stderr) + sys.exit(1) + +def _run(table, *args): + cmd = ['ip', 'netns', 'exec', namespace, 'iptables', '-t', table] + list(args) + r = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if r.returncode != 0: + print(f"iptables ({table}): {r.stderr.decode().strip()}", file=sys.stderr) + return r + +def iptf(*args): + _run('filter', *args) + +if rules_json: + try: + rules = json.loads(rules_json) + except Exception as e: + print(f"apply-network-acl: failed to parse rules: {e}", file=sys.stderr) + sys.exit(1) +else: + rules = [] + + +for rule in sorted(rules, key=lambda r: r.get('number', 999)): + direction = rule.get('trafficType', 'ingress').lower() + protocol = (rule.get('protocol') or 'all').lower() + port_start = rule.get('portStart') + port_end = rule.get('portEnd') + icmp_type = rule.get('icmpType') + icmp_code = rule.get('icmpCode') + src_cidrs = rule.get('sourceCidrs') or ['0.0.0.0/0'] + action = 'ACCEPT' if rule.get('action', 'deny').lower() == 'allow' else 'DROP' + + for src_cidr in src_cidrs: + a = [] + if direction == 'ingress': + # Traffic toward VMs (going OUT on guest veth_n into the tier subnet) + a = ['-o', veth_n] + if net_cidr: + a += ['-d', net_cidr] + if src_cidr and src_cidr not in ('0.0.0.0/0', '::/0', ''): + a += ['-s', src_cidr] + else: + # Traffic FROM VMs (coming IN on guest veth_n from the tier subnet) + a = ['-i', veth_n] + if net_cidr: + a += ['-s', net_cidr] + # For egress rules sourceCidrs is used as destination filter + if src_cidr and src_cidr not in ('0.0.0.0/0', '::/0', ''): + a += ['-d', src_cidr] + + if protocol not in ('all', ''): + a += ['-p', protocol] + if protocol in ('tcp', 'udp') and port_start is not None: + port_spec = str(port_start) + if port_end is not None and port_end != port_start: + port_spec = f"{port_start}:{port_end}" + a += ['--dport', port_spec] + elif protocol == 'icmp' and icmp_type is not None and icmp_type != -1: + icmp_spec = str(icmp_type) + if icmp_code is not None and icmp_code != -1: + icmp_spec += f"/{icmp_code}" + a += ['--icmp-type', icmp_spec] + + iptf('-A', acl_chain, *a, '-j', action) + +# Default: DROP all unmatched traffic (implicit deny at end of ACL) +iptf('-A', acl_chain, '-j', 'DROP') + +# Insert RELATED,ESTABLISHED last so it lands at position 1 regardless of what +# was appended above — active sessions must never be re-evaluated against the +# explicit rules/deny-all, no matter how this chain is built. +iptf('-I', acl_chain, '1', '-m', 'state', '--state', 'RELATED,ESTABLISHED', '-j', 'ACCEPT') + +print(f"apply-network-acl: applied {len(rules)} ACL rule(s) to chain {acl_chain}") +PYEOF + + local py_exit=$? + + if [ "${cleanup_acl_file}" = "true" ]; then + rm -f "${acl_rules_file}" 2>/dev/null || true + fi + + if [ ${py_exit} -ne 0 ]; then + log "apply-network-acl: Python rule builder exited ${py_exit}; ACL chain may be incomplete" + fi + + # ---- 5. Insert jump from fchain to acl chain at position 1 ---- + # ACL rules take precedence over the catch-all ACCEPT rules in fchain. + # + # The jump MUST be scoped to this network's own veth (-o for traffic + # heading to the guest, -i for traffic coming from it). All tiers in a + # VPC share one namespace and one top-level FORWARD chain, so an + # unconditional jump here would hand every OTHER tier's traffic to this + # ACL chain too — and since the chain ends in an unconditional catch-all + # DROP (the implicit deny), it would silently swallow packets for tiers + # evaluated after this one, before they ever reach their own (correct) + # ACL chain. Scoping by interface keeps each tier's implicit deny from + # catching anything but its own traffic. + if ip netns exec "${NAMESPACE}" iptables -t filter -n -L "${fchain}" >/dev/null 2>&1; then + ip netns exec "${NAMESPACE}" iptables -t filter \ + -I "${fchain}" 1 -i "${veth_n}" -j "${acl_chain_name}" 2>/dev/null || true + ip netns exec "${NAMESPACE}" iptables -t filter \ + -I "${fchain}" 1 -o "${veth_n}" -j "${acl_chain_name}" 2>/dev/null || true + log "apply-network-acl: inserted ACL jump in ${fchain} (scoped to ${veth_n})" + fi + + release_lock + log "apply-network-acl: done network=${NETWORK_ID}" +} + +############################################################################## +# Main dispatcher +############################################################################## + +ensure_dirs + +COMMAND="${1:-}" +shift || true + +case "${COMMAND}" in + implement-network) cmd_implement_network "$@" ;; + shutdown-network) cmd_shutdown_network "$@" ;; + destroy-network) cmd_destroy_network "$@" ;; + # VPC lifecycle + implement-vpc) cmd_implement_vpc "$@" ;; + update-vpc-source-nat-ip) cmd_update_vpc_source_nat_ip "$@" ;; + shutdown-vpc) cmd_shutdown_vpc "$@" ;; + destroy-vpc) cmd_destroy_vpc "$@" ;; + assign-ip) cmd_assign_ip "$@" ;; + release-ip) cmd_release_ip "$@" ;; + add-static-nat) cmd_add_static_nat "$@" ;; + delete-static-nat) cmd_delete_static_nat "$@" ;; + add-port-forward) cmd_add_port_forward "$@" ;; + delete-port-forward) cmd_delete_port_forward "$@" ;; + # NIC lifecycle + prepare-nic) cmd_prepare_nic "$@" ;; + release-nic) cmd_release_nic "$@" ;; + # DHCP / DNS (dnsmasq) + config-dhcp-subnet) cmd_config_dhcp_subnet "$@" ;; + remove-dhcp-subnet) cmd_remove_dhcp_subnet "$@" ;; + add-dhcp-entry) cmd_add_dhcp_entry "$@" ;; + remove-dhcp-entry) cmd_remove_dhcp_entry "$@" ;; + config-dns-subnet) cmd_config_dns_subnet "$@" ;; + remove-dns-subnet) cmd_remove_dns_subnet "$@" ;; + add-dns-entry) cmd_add_dns_entry "$@" ;; + remove-dns-entry) cmd_remove_dns_entry "$@" ;; + # UserData / metadata (apache2) + save-userdata) cmd_save_userdata "$@" ;; + save-password) cmd_save_password "$@" ;; + save-sshkey) cmd_save_sshkey "$@" ;; + save-hypervisor-hostname) cmd_save_hypervisor_hostname "$@" ;; + save-vm-data) cmd_save_vm_data "$@" ;; + restore-network) cmd_restore_network "$@" ;; + # Load balancing (haproxy) + apply-fw-rules) cmd_apply_fw_rules "$@" ;; + apply-lb-rules) cmd_apply_lb_rules "$@" ;; + # ACL rules (VPC network ACLs) + apply-network-acl) cmd_apply_network_acl "$@" ;; + # Custom actions + custom-action) cmd_custom_action "$@" ;; + "") + echo "Usage: $0 {implement-network|shutdown-network|destroy-network|" \ + "implement-vpc|update-vpc-source-nat-ip|shutdown-vpc|destroy-vpc|" \ + "assign-ip|release-ip|" \ + "add-static-nat|delete-static-nat|add-port-forward|delete-port-forward|" \ + "prepare-nic|release-nic|" \ + "config-dhcp-subnet|remove-dhcp-subnet|add-dhcp-entry|remove-dhcp-entry|set-dhcp-options|" \ + "config-dns-subnet|remove-dns-subnet|add-dns-entry|remove-dns-entry|" \ + "save-userdata|save-password|save-sshkey|save-hypervisor-hostname|save-vm-data|restore-network|" \ + "apply-fw-rules|apply-lb-rules|apply-network-acl|custom-action} [options]" >&2 + exit 1 ;; + *) + echo "Unknown command: ${COMMAND}" >&2 + exit 1 ;; +esac + +exit 0 + diff --git a/Network-Namespace/network-namespace.sh b/Network-Namespace/network-namespace.sh new file mode 100755 index 0000000..9e11e6b --- /dev/null +++ b/Network-Namespace/network-namespace.sh @@ -0,0 +1,469 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +############################################################################## +# network-namespace.sh (network-namespace) +# +# Proxy script for the network-namespace CloudStack extension. +# Runs on the CloudStack management server. +# +# Invocation model: +# network-namespace.sh +# +# The payload JSON includes top-level extension details: +# physical-network-extension-details +# network-extension-details +# +# For standard commands, command-specific keys are nested under payload.{...}. +# For custom-action, command-specific keys are top-level (flat payload). +# +# Two runtime modes: +# 1) ensure-network-device (local, no SSH): selects/revalidates host and emits +# a single-line JSON object like: +# {"host":"192.168.1.10","namespace":"cs-net-42"} +# 2) all other commands: forwards the payload file to the selected host and +# executes network-namespace-wrapper.sh remotely. +# +# Common extension-detail keys (inside physical-network-extension-details): +# hosts, host, port, username, password, sshkey +# +# ---- SSH authentication priority ---- +# 1. sshkey field in --physical-network-extension-details → PEM key +# 2. password field → sshpass(1) +# 3. No credentials → relies on SSH agent / host keys on mgmt server +# +# Exit codes: +# 0 – success +# 1 – usage / configuration error +# 2 – SSH connection / authentication error +# 3 – remote command returned non-zero +############################################################################## + +set -euo pipefail + +DEFAULT_SSH_PORT=22 +DEFAULT_SSH_USER=root + +# --------------------------------------------------------------------------- +# Resolve this entry-point's absolute path so we can derive both the KVM +# wrapper path and the log file name from the extension directory name. +# +# Layout: +# management server: /usr/share/cloudstack-management/extensions//.sh +# KVM host (wrapper): /etc/cloudstack/extensions//-wrapper.sh +# +# _EXT_DIR_NAME is the basename of the directory containing this script, +# which equals the extension name assigned by CloudStack (e.g. +# "extnet-isolated-gk3yys"). Both the wrapper path and the log file are +# derived from it so that renamed deployments work automatically. +# +# Callers may still override the remote path via CS_NET_SCRIPT_PATH: +# CS_NET_SCRIPT_PATH=/custom/path/wrapper.sh network-namespace.sh ... +# --------------------------------------------------------------------------- +_SELF="$(readlink -f "$0" 2>/dev/null \ + || realpath "$0" 2>/dev/null \ + || echo "$0")" +_SCRIPT_BASENAME="$(basename "${_SELF}" .sh)" +_EXT_DIR_NAME="$(basename "$(dirname "${_SELF}")")" + +# Remote wrapper path on each KVM host. +DEFAULT_SCRIPT_PATH="/etc/cloudstack/extensions/${_EXT_DIR_NAME}/${_SCRIPT_BASENAME}-wrapper.sh" + +# Log file — under /var/log/cloudstack/extensions/ named after the extension. +LOG_FILE="/tmp/cloudstack-extensions/${_EXT_DIR_NAME}.log" +mkdir -p "$(dirname "${LOG_FILE}")" 2>/dev/null || true +TMPDIR_BASE=/tmp + +# --------------------------------------------------------------------------- +# Logging +# --------------------------------------------------------------------------- + +log() { + local ts + ts=$(date '+%Y-%m-%d %H:%M:%S') + printf '[%s] %s\n' "${ts}" "$*" >> "${LOG_FILE}" 2>/dev/null || true +} + +die() { + log "ERROR: $*" + exit "${2:-1}" +} + +# --------------------------------------------------------------------------- +# JSON helpers (no jq dependency) +# --------------------------------------------------------------------------- + +json_get() { + # json_get → unquoted string value or empty + printf '%s' "$1" | grep -o "\"$2\":\"[^\"]*\"" | cut -d'"' -f4 || true +} + +# --------------------------------------------------------------------------- +# Validate input and parse command +# --------------------------------------------------------------------------- + +if [ $# -lt 1 ]; then + die "Usage: network-namespace.sh " 1 +fi + +COMMAND="$1" +shift || true + +PAYLOAD_FILE="" +TIMEOUT_SECONDS="" +PAYLOAD_MODE="false" + +if [ $# -ge 1 ] && [ -f "${1}" ]; then + PAYLOAD_FILE="$1" + TIMEOUT_SECONDS="${2:-60}" + PAYLOAD_MODE="true" + shift || true + [ $# -gt 0 ] && shift || true +fi + +payload_json_get() { + # payload_json_get where path is dot-separated JSON path + python3 - "$1" "$2" <<'PY' +import json, sys +with open(sys.argv[1], encoding='utf-8') as fh: + data = json.load(fh) +cur = data +for part in sys.argv[2].split('.'): + if isinstance(cur, dict): + cur = cur.get(part) + else: + cur = None + if cur is None: + break +if cur is None: + print("") +elif isinstance(cur, (dict, list)): + print(json.dumps(cur, separators=(",", ":"))) +else: + print(str(cur)) +PY +} + +# --------------------------------------------------------------------------- +# Parse CLI arguments: extract known flags, collect the rest as FORWARD_ARGS +# --------------------------------------------------------------------------- + +PHYS_DETAILS="{}" +EXTENSION_DETAILS="{}" +NETWORK_ID="" +CURRENT_DETAILS="{}" +VPC_ID="" +FORWARD_ARGS=() + +if [ "${PAYLOAD_MODE}" = "true" ]; then + PHYS_DETAILS=$(payload_json_get "${PAYLOAD_FILE}" "physical-network-extension-details") + EXTENSION_DETAILS=$(payload_json_get "${PAYLOAD_FILE}" "network-extension-details") + + if [ "${COMMAND}" = "custom-action" ]; then + NETWORK_ID=$(payload_json_get "${PAYLOAD_FILE}" "network_id") + VPC_ID=$(payload_json_get "${PAYLOAD_FILE}" "vpc_id") + else + NETWORK_ID=$(payload_json_get "${PAYLOAD_FILE}" "payload.network_id") + VPC_ID=$(payload_json_get "${PAYLOAD_FILE}" "payload.vpc_id") + CURRENT_DETAILS=$(payload_json_get "${PAYLOAD_FILE}" "payload.current_details") + [ -z "${CURRENT_DETAILS}" ] && CURRENT_DETAILS="{}" + fi +else + while [ $# -gt 0 ]; do + case "$1" in + --physical-network-extension-details) + PHYS_DETAILS="${2:-{}}" + shift 2 ;; + --network-extension-details) + EXTENSION_DETAILS="${2:-{}}" + shift 2 ;; + --network-id) + NETWORK_ID="${2:-}" + FORWARD_ARGS+=("$1" "$2") + shift 2 ;; + --vpc-id) + VPC_ID="${2:-}" + FORWARD_ARGS+=("$1" "$2") + shift 2 ;; + --current-details) + CURRENT_DETAILS="${2:-{}}" + shift 2 ;; + *) + FORWARD_ARGS+=("$1") + shift ;; + esac + done +fi + +REMOTE_SCRIPT="${CS_NET_SCRIPT_PATH:-${DEFAULT_SCRIPT_PATH}}" + +REMOTE_PORT=$(json_get "${PHYS_DETAILS}" "port") +REMOTE_USER=$(json_get "${PHYS_DETAILS}" "username") +REMOTE_PASS=$(json_get "${PHYS_DETAILS}" "password") +REMOTE_SSHKEY=$(json_get "${PHYS_DETAILS}" "sshkey") +HOSTS_CSV=$(json_get "${PHYS_DETAILS}" "hosts") +SINGLE_HOST=$(json_get "${PHYS_DETAILS}" "host") + +REMOTE_PORT="${REMOTE_PORT:-${DEFAULT_SSH_PORT}}" +REMOTE_USER="${REMOTE_USER:-${DEFAULT_SSH_USER}}" + +# Build the candidate host list +if [ -n "${HOSTS_CSV}" ]; then + IFS=',' read -ra HOST_LIST <<< "${HOSTS_CSV}" +elif [ -n "${SINGLE_HOST}" ]; then + HOST_LIST=("${SINGLE_HOST}") +else + HOST_LIST=() +fi + +# --------------------------------------------------------------------------- +# SSH helpers +# --------------------------------------------------------------------------- + +KEY_TMPFILE="" +KEY_TMPDIR="" + +cleanup() { + local rc=$? + if [ -n "${KEY_TMPDIR}" ] && [ -d "${KEY_TMPDIR}" ]; then + rm -rf "${KEY_TMPDIR}" 2>/dev/null || true + fi + exit ${rc} +} +trap cleanup EXIT INT TERM + +setup_ssh_key() { + if [ -n "${REMOTE_SSHKEY}" ] && [ -z "${KEY_TMPFILE}" ]; then + KEY_TMPDIR=$(mktemp -d "${TMPDIR_BASE}/.cs-extnet-key-XXXXXX") + chmod 700 "${KEY_TMPDIR}" + KEY_TMPFILE="${KEY_TMPDIR}/id_extnet" + printf '%s\n' "${REMOTE_SSHKEY}" > "${KEY_TMPFILE}" + chmod 600 "${KEY_TMPFILE}" + fi +} + +ssh_opts() { + local opts=( + -o StrictHostKeyChecking=no + -o UserKnownHostsFile=/dev/null + -o LogLevel=ERROR + -o ConnectTimeout=10 + -p "${REMOTE_PORT}" + ) + if [ -n "${KEY_TMPFILE}" ]; then + opts+=(-i "${KEY_TMPFILE}" -o IdentitiesOnly=yes -o BatchMode=yes) + elif [ -n "${REMOTE_PASS}" ]; then + # When using password-based auth we should not force an IdentityFile of /dev/null + # because recent OpenSSH may attempt to parse it and emit libcrypto errors + # (seen as: Load key "/dev/null": error in libcrypto). Just rely on sshpass + # (SSHPASS) to provide the password if needed. + opts+=(-o IdentitiesOnly=yes) + fi + printf '%s\n' "${opts[@]}" +} + +host_reachable() { + local host="$1" + setup_ssh_key + local opts + mapfile -t opts < <(ssh_opts) + if [ -n "${REMOTE_SSHKEY}" ]; then + ssh "${opts[@]}" "${REMOTE_USER}@${host}" "echo ok" >/dev/null 2>&1 + elif [ -n "${REMOTE_PASS}" ]; then + command -v sshpass >/dev/null 2>&1 || return 1 + SSHPASS="${REMOTE_PASS}" sshpass -e \ + ssh "${opts[@]}" "${REMOTE_USER}@${host}" "echo ok" >/dev/null 2>&1 + else + ssh "${opts[@]}" "${REMOTE_USER}@${host}" "echo ok" >/dev/null 2>&1 + fi +} + +ssh_exec() { + local host="$1" + local remote_cmd="$2" + setup_ssh_key + local opts + mapfile -t opts < <(ssh_opts) + if [ -n "${REMOTE_SSHKEY}" ]; then + ssh "${opts[@]}" "${REMOTE_USER}@${host}" "${remote_cmd}" + elif [ -n "${REMOTE_PASS}" ]; then + command -v sshpass >/dev/null 2>&1 || \ + die "password set but sshpass not installed. Use sshkey instead." 2 + SSHPASS="${REMOTE_PASS}" sshpass -e \ + ssh "${opts[@]}" "${REMOTE_USER}@${host}" "${remote_cmd}" + else + ssh "${opts[@]}" "${REMOTE_USER}@${host}" "${remote_cmd}" + fi +} + +upload_file_to_remote() { + local host="$1" local_file="$2" tag="$3" + [ -f "${local_file}" ] || die "Missing local payload file: ${local_file}" 1 + + local remote_tmp + remote_tmp=$(ssh_exec "${host}" "mktemp /tmp/cs-extnet-${tag}-XXXXXX") || \ + die "Failed to create remote temp file for ${tag}" 2 + remote_tmp=$(printf '%s' "${remote_tmp}" | tr -d '\r\n') + [ -n "${remote_tmp}" ] || die "Failed to resolve remote temp file for ${tag}" 2 + + cat "${local_file}" | ssh_exec "${host}" "cat > '${remote_tmp}' && chmod 600 '${remote_tmp}'" || \ + die "Failed to upload payload file for ${tag}" 2 + + printf '%s' "${remote_tmp}" +} + +# --------------------------------------------------------------------------- +# ensure-network-device +# --------------------------------------------------------------------------- + +if [ "${COMMAND}" = "ensure-network-device" ]; then + [ -z "${NETWORK_ID}" ] && [ -z "${VPC_ID}" ] && die "ensure-network-device: missing --network-id or --vpc-id" 1 + + if [ ${#HOST_LIST[@]} -eq 0 ]; then + die "ensure-network-device: no hosts configured. Set 'hosts' in registerExtension details." 1 + fi + + # Namespace names must match those used by the wrapper on the KVM host. + # VPC networks share one namespace per VPC (cs-vpc-); + # standalone networks (Isolated and Shared) each get their own namespace (cs-net-). + if [ -n "${VPC_ID}" ]; then + NAMESPACE="cs-vpc-${VPC_ID}" + else + NAMESPACE="cs-net-${NETWORK_ID}" + fi + + # ---- Step 1: honour the previously selected host (sticky assignment) ---- + # This preserves the host–namespace binding across API calls once a network + # has been implemented on a particular KVM host. + CURRENT_HOST=$(json_get "${CURRENT_DETAILS}" "host") + [ -z "${CURRENT_HOST}" ] && CURRENT_HOST=$(json_get "${EXTENSION_DETAILS}" "host") + + if [ -n "${CURRENT_HOST}" ]; then + for h in "${HOST_LIST[@]}"; do + h="${h// /}" + if [ "${h}" = "${CURRENT_HOST}" ]; then + if host_reachable "${CURRENT_HOST}"; then + log "ensure-network-device: ${NETWORK_ID:+network=${NETWORK_ID} }${VPC_ID:+vpc=${VPC_ID} }keeping current host=${CURRENT_HOST}" + if [ -n "${VPC_ID}" ]; then + printf '{"host":"%s","namespace":"%s","vpc_id":"%s"}\n' \ + "${CURRENT_HOST}" "${NAMESPACE}" "${VPC_ID}" + else + printf '{"host":"%s","namespace":"%s"}\n' \ + "${CURRENT_HOST}" "${NAMESPACE}" + fi + exit 0 + else + log "ensure-network-device: current host ${CURRENT_HOST} not reachable — failover" + fi + break + fi + done + fi + + # ---- Step 2: stable hash-based host selection for new / failed-over networks ---- + # + # For VPC networks ALL tiers must land on the same KVM host (they share one + # namespace). Using VPC_ID as the hash key guarantees every tier in a VPC + # hashes to the same preferred index even when its own details are not yet + # stored. For isolated networks the NETWORK_ID is used. + # + # Algorithm: CRC32 of the routing key (via cksum) modulo the host count + # gives a stable preferred index. We probe hosts starting from that index, + # wrapping around, until a reachable one is found. This distributes + # different networks evenly across KVM hosts while remaining deterministic. + _ROUTE_KEY="${VPC_ID:-${NETWORK_ID}}" + _HOST_COUNT="${#HOST_LIST[@]}" + _PREFERRED_IDX=$(printf '%s' "${_ROUTE_KEY}" | cksum | awk -v n="${_HOST_COUNT}" '{print ($1 % n)}') + + _SELECTED_HOST="" + _PROBE=0 + while [ "${_PROBE}" -lt "${_HOST_COUNT}" ]; do + _IDX=$(( (_PREFERRED_IDX + _PROBE) % _HOST_COUNT )) + _H="${HOST_LIST[$_IDX]// /}" + if host_reachable "${_H}"; then + _SELECTED_HOST="${_H}" + log "ensure-network-device: ${NETWORK_ID:+network=${NETWORK_ID} }${VPC_ID:+vpc=${VPC_ID} }hash-selected host=${_SELECTED_HOST} (key=${_ROUTE_KEY}, idx=${_IDX})" + break + fi + log "ensure-network-device: host ${_H} not reachable, trying next" + _PROBE=$(( _PROBE + 1 )) + done + + [ -z "${_SELECTED_HOST}" ] && \ + die "ensure-network-device: no reachable host found in list: ${HOSTS_CSV:-${SINGLE_HOST}}" 1 + + if [ -n "${VPC_ID}" ]; then + printf '{"host":"%s","namespace":"%s","vpc_id":"%s"}\n' \ + "${_SELECTED_HOST}" "${NAMESPACE}" "${VPC_ID}" + else + printf '{"host":"%s","namespace":"%s"}\n' "${_SELECTED_HOST}" "${NAMESPACE}" + fi + exit 0 +fi + +# --------------------------------------------------------------------------- +# All other commands: forward via SSH to the selected network device +# --------------------------------------------------------------------------- + +REMOTE_HOST=$(json_get "${EXTENSION_DETAILS}" "host") +if [ -z "${REMOTE_HOST}" ]; then + REMOTE_HOST="${SINGLE_HOST:-}" + [ -z "${REMOTE_HOST}" ] && [ ${#HOST_LIST[@]} -gt 0 ] && REMOTE_HOST="${HOST_LIST[0]// /}" +fi +[ -z "${REMOTE_HOST}" ] && die "No target host available. Run ensure-network-device first." 1 + +# Build and execute remote command +REMOTE_PAYLOAD_FILES=() +if [ "${PAYLOAD_MODE}" = "true" ]; then + REMOTE_PAYLOAD_FILE=$(upload_file_to_remote "${REMOTE_HOST}" "${PAYLOAD_FILE}" "payload") + REMOTE_PAYLOAD_FILES+=("${REMOTE_PAYLOAD_FILE}") + REMOTE_CMD="'${REMOTE_SCRIPT}' '${COMMAND}' '${REMOTE_PAYLOAD_FILE//"'"/"'\\''"}' '${TIMEOUT_SECONDS}'" +else + remote_args=() + for arg in "${FORWARD_ARGS[@]}"; do + remote_args+=("'${arg//"'"/"'\\''"}'" ) + done + + PHYS_ESCAPED="${PHYS_DETAILS//\'/\'\\\'\'}" + EXT_ESCAPED="${EXTENSION_DETAILS//\'/\'\\\'\'}" + REMOTE_CMD="'${REMOTE_SCRIPT}' '${COMMAND}' ${remote_args[*]} --physical-network-extension-details '${PHYS_ESCAPED}' --network-extension-details '${EXT_ESCAPED}'" +fi + +log "Remote: ${REMOTE_USER}@${REMOTE_HOST}:${REMOTE_PORT} cmd=${COMMAND}" + +RC=0 +ssh_exec "${REMOTE_HOST}" "${REMOTE_CMD}" || RC=$? + +if [ ${#REMOTE_PAYLOAD_FILES[@]} -gt 0 ]; then + for _rf in "${REMOTE_PAYLOAD_FILES[@]}"; do + ssh_exec "${REMOTE_HOST}" "rm -f '${_rf}'" >/dev/null 2>&1 || true + done +fi + +if [ ${RC} -ne 0 ]; then + if [ ${RC} -eq 255 ]; then + log "SSH connection failed (rc=255): host=${REMOTE_HOST}:${REMOTE_PORT} user=${REMOTE_USER}" + exit 2 + fi + log "Remote script returned rc=${RC}" + exit 3 +fi + +log "Command '${COMMAND}' completed successfully on ${REMOTE_HOST}" +exit 0 +