Direct Routed (L3) guest networks: route public IPv4/IPv6 directly to Instances

Adds a new guest network type in which the hypervisor performs L3 routing
for the Instance: no Virtual Router, no NAT and no DHCP. Each Instance
receives a public IPv4 address as a /32 and/or an IPv6 address as a /128,
with a shared, host-independent gateway (169.254.0.1 and fe80::1) that
every hypervisor carries on the network's bridge. All addressing reaches
the Instance exclusively via ConfigDrive/cloud-init; a routing daemon on
the host (FRR, BIRD, ...) advertises the addresses to the fabric and is
deliberately out of scope for CloudStack.

Management server:
- GuestType.L3; the guest_type column is char(32), so no schema change.
- Offering validation: UserData via ConfigDrive is mandatory, Dns optional
  but ConfigDrive-only, SecurityGroup permitted (now allowed for L3
  alongside Shared), Dhcp rejected as not supported and not needed.
  Network mode, specifyVlan and VPC use are rejected.
- DirectRoutedNetworkGuru subclasses DirectNetworkGuru, inheriting the
  Shared-network address lifecycle. canHandle() selects on the offering's
  guest type alone; design() produces a Native broadcast domain with no
  isolation id. After allocation the NicProfile is forced into host-route
  form, which is also the signature by which the agent and ConfigDrive
  recognise these NICs.
- createNetwork treats L3 like Shared for the subnet: explicit IP range
  mandatory, vlan/IP-range row created at network creation, IPv6 accepted
  without the /64 restriction, aclType Account.
- Zone-wide IPv4 overlap validation for L3 ranges: all L3 subnets share
  one host routing table and one fabric, so an overlap is an address
  conflict. The IPv6 vlan check was already zone-wide.

ConfigDrive:
- Network data is always generated for a direct routed NIC; the historical
  gate (Dhcp or Dns supported) held while ConfigDrive supplemented a VR
  but would leave these NICs with no addressing at all. Route generation
  itself is unchanged: cloud-init detects an IPv4 gateway inside
  169.254.0.0/16 and sets on-link on the rendered route by itself.

KVM agent:
- One uplink-less bridge per network, brdr-<network id>, created and
  removed by the new modifybrdr.sh (flock'd, idempotent, refuses to remove
  a bridge still in use). The bridge carries the gateway addresses,
  forwarding and strict rp_filter; separate bridges make isolation between
  networks topological rather than a filtering concern.
- BridgeVifDriver plugs direct routed NICs into their brdr bridge and runs
  the existing modifymacip.sh hook per NIC to install the static neighbour
  entry and host route, regardless of the host-wide EVPN property, whose
  meaning is unchanged.

The design document, including the decision log and the verification notes
behind each choice, is added under docs/design/.

Author: wido

api/src/main/java/com/cloud/network/Network.java

@@ -43,7 +43,7 @@
 public interface Network extends ControlledEntity, StateObject<Network.State>, InternalIdentity, Identity, Serializable, Displayable {
 
     enum GuestType {
-        Shared, Isolated, L2;
+        Shared, Isolated, L2, L3;
 
         public static GuestType fromValue(String type) {
             if (StringUtils.isBlank(type)) {
@@ -54,6 +54,8 @@ public static GuestType fromValue(String type) {
                 return Isolated;
             } else if (type.equalsIgnoreCase("L2")) {
                 return L2;
+            } else if (type.equalsIgnoreCase("L3")) {
+                return L3;
             } else {
                 throw new InvalidParameterValueException("Unexpected Guest type : " + type);
             }

api/src/main/java/org/apache/cloudstack/api/command/user/network/CreateNetworkCmd.java

@@ -340,10 +340,10 @@ public Long getPhysicalNetworkId() {
             }
         }
         if (physicalNetworkId != null) {
-            if ((offering.getGuestType() == GuestType.Shared) || (offering.getGuestType() == GuestType.L2)) {
+            if ((offering.getGuestType() == GuestType.Shared) || (offering.getGuestType() == GuestType.L2) || (offering.getGuestType() == GuestType.L3)) {
                 return physicalNetworkId;
             } else {
-                throw new InvalidParameterValueException("Physical network ID can be specified for networks of guest IP type " + GuestType.Shared + " or " + GuestType.L2 + " only.");
+                throw new InvalidParameterValueException(String.format("Physical network ID can be specified for networks of guest IP type %s, %s or %s only.", GuestType.Shared, GuestType.L2, GuestType.L3));
             }
         } else {
             if (zoneId == null) {

api/src/main/java/org/apache/cloudstack/api/command/user/vm/RemoveIpFromVmNicCmd.java

@@ -124,6 +124,16 @@ public NetworkType getNetworkType() {
     }
 
 
+    /**
+     * A Direct Routed (L3) network needs the agent told when a secondary IP goes away, so the
+     * host route and neighbour entry are removed - otherwise the host keeps routing an address
+     * the Instance no longer owns, and the routing daemon keeps advertising it.
+     */
+    private boolean isDirectRoutedNetwork() {
+        Network ntwk = _entityMgr.findById(Network.class, getNetworkId());
+        return ntwk != null && Network.GuestType.L3.equals(ntwk.getGuestType());
+    }
+
     private boolean isZoneSGEnabled() {
         Network ntwk = _entityMgr.findById(Network.class, getNetworkId());
         DataCenter dc = _entityMgr.findById(DataCenter.class, ntwk.getDataCenterId());
@@ -144,7 +154,7 @@ public void execute() throws InvalidParameterValueException {
             secIp = nicSecIp.getIp6Address();
         }
 
-        if (isZoneSGEnabled()) {
+        if (isZoneSGEnabled() || isDirectRoutedNetwork()) {
             //remove the security group rules for this secondary ip
             boolean success = false;
             success = _securityGroupService.securityGroupRulesForVmSecIp(nicSecIp.getNicId(), secIp, false);

core/src/main/java/com/cloud/agent/api/NetworkRulesVmSecondaryIpCommand.java

@@ -28,12 +28,28 @@ public class NetworkRulesVmSecondaryIpCommand extends Command {
     private String vmSecIp;
     private String vmMac;
     private String action;
+    private boolean directRouted;
+    private boolean applySecurityGroupRules = true;
 
     public NetworkRulesVmSecondaryIpCommand(String vmName, VirtualMachine.Type type) {
         this.vmName = vmName;
         this.type = type;
     }
 
+    public NetworkRulesVmSecondaryIpCommand(String vmName, String vmMac, String secondaryIp, boolean action, boolean directRouted, boolean applySecurityGroupRules) {
+        this(vmName, vmMac, secondaryIp, action);
+        this.directRouted = directRouted;
+        this.applySecurityGroupRules = applySecurityGroupRules;
+    }
+
+    public boolean isDirectRouted() {
+        return directRouted;
+    }
+
+    public boolean isApplySecurityGroupRules() {
+        return applySecurityGroupRules;
+    }
+
     public NetworkRulesVmSecondaryIpCommand(String vmName, String vmMac, String secondaryIp, boolean action) {
         this.vmName = vmName;
         this.vmMac = vmMac;

docs/design/direct-routed-networks.md

@@ -48,6 +48,7 @@
 
 import com.cloud.network.NetworkModel;
 import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.net.NetUtils;
 import com.cloud.utils.script.Script;
 import com.google.gson.JsonArray;
 import com.google.gson.JsonElement;
@@ -249,8 +250,15 @@ static void writeVmMetadata(List<String[]> vmData, String tempDirName, File open
      */
     static void writeNetworkData(List<NicProfile> nics, Map<Long, List<Network.Service>> supportedServices, File openStackFolder) {
         JsonObject finalNetworkData = new JsonObject();
-        if (needForGeneratingNetworkData(supportedServices)) {
+        // A direct routed NIC always needs its network data written: ConfigDrive is the only
+        // channel that carries its addressing, whatever services the offering does or does not
+        // have. For every other NIC the historical gate (Dhcp or Dns supported) is unchanged.
+        boolean generateForAllNics = needForGeneratingNetworkData(supportedServices);
+        if (generateForAllNics || nics.stream().anyMatch(ConfigDriveBuilder::isDirectRoutedNic)) {
             for (NicProfile nic : nics) {
+                if (!generateForAllNics && !isDirectRoutedNic(nic)) {
+                    continue;
+                }
                 List<Network.Service> supportedService = supportedServices.get(nic.getId());
                 JsonObject networkData = getNetworkDataJsonObjectForNic(nic, supportedService);
 
@@ -267,6 +275,24 @@ static boolean needForGeneratingNetworkData(Map<Long, List<Network.Service>> sup
         return supportedServices.values().stream().anyMatch(services -> services.contains(Network.Service.Dhcp) || services.contains(Network.Service.Dns));
     }
 
+    /**
+     * A NIC on a Direct Routed (L3) network is recognised by the form of its addressing, not by a
+     * flag: an IPv4 host netmask with a link-local gateway, or an IPv6 /128 with the fixed
+     * link-local gateway. No other network type produces this combination. ConfigDrive is the
+     * only channel that carries such a NIC's network configuration (there is no DHCP and no RA),
+     * so network data must always be generated for it, whatever services the offering carries.
+     */
+    static boolean isDirectRoutedNic(NicProfile nic) {
+        if (nic == null) {
+            return false;
+        }
+        boolean directRoutedIpv4 = StringUtils.isNotBlank(nic.getIPv4Address()) && NetUtils.IPV4_HOST_NETMASK.equals(nic.getIPv4Netmask())
+                && StringUtils.isNotBlank(nic.getIPv4Gateway()) && NetUtils.isIpWithInCidrRange(nic.getIPv4Gateway(), NetUtils.getLinkLocalCIDR());
+        boolean directRoutedIpv6 = StringUtils.isNotBlank(nic.getIPv6Address()) && StringUtils.isNotBlank(nic.getIPv6Cidr())
+                && nic.getIPv6Cidr().endsWith("/" + NetUtils.IPV6_HOST_PREFIX_LENGTH) && NetUtils.getIpv6LinkLocalGateway().equals(nic.getIPv6Gateway());
+        return directRoutedIpv4 || directRoutedIpv6;
+    }
+
     /**
      *  Writes an empty JSON file named vendor_data.json in openStackFolder
      *

engine/storage/configdrive/src/main/java/org/apache/cloudstack/storage/configdrive/ConfigDriveBuilder.java

@@ -52,6 +52,7 @@
 
 import com.cloud.utils.exception.CloudRuntimeException;
 import com.cloud.utils.script.Script;
+import com.google.gson.JsonArray;
 import com.google.gson.JsonObject;
 
 @RunWith(MockitoJUnitRunner.class)
@@ -659,4 +660,96 @@ public void testWriteNetworkDataEmptyJson() throws Exception {
         Assert.assertEquals(expectedJsonObject, actualJson);
         folder.delete();
     }
-}
+
+    private NicProfile directRoutedNicProfile() {
+        NicProfile nic = new NicProfile();
+        nic.setId(1L);
+        nic.setDeviceId(0);
+        nic.setMacAddress("02:00:4c:5f:00:01");
+        nic.setIPv4Address("203.0.113.55");
+        nic.setIPv4Netmask("255.255.255.255");
+        nic.setIPv4Gateway("169.254.0.1");
+        nic.setIPv6Address("2001:db8:1::55");
+        nic.setIPv6Cidr("2001:db8:1::55/128");
+        nic.setIPv6Gateway("fe80::1");
+        nic.setIPv4Dns1("8.8.8.8");
+        return nic;
+    }
+
+    private NicProfile sharedNicProfile() {
+        NicProfile nic = new NicProfile();
+        nic.setId(1L);
+        nic.setDeviceId(0);
+        nic.setMacAddress("02:00:4c:5f:00:02");
+        nic.setIPv4Address("10.1.1.55");
+        nic.setIPv4Netmask("255.255.255.0");
+        nic.setIPv4Gateway("10.1.1.1");
+        return nic;
+    }
+
+    @Test
+    public void isDirectRoutedNicRecognisesHostRouteForm() {
+        Assert.assertTrue(ConfigDriveBuilder.isDirectRoutedNic(directRoutedNicProfile()));
+    }
+
+    @Test
+    public void isDirectRoutedNicRecognisesIpv6OnlyForm() {
+        NicProfile nic = directRoutedNicProfile();
+        nic.setIPv4Address(null);
+        nic.setIPv4Netmask(null);
+        nic.setIPv4Gateway(null);
+        Assert.assertTrue(ConfigDriveBuilder.isDirectRoutedNic(nic));
+    }
+
+    @Test
+    public void isDirectRoutedNicRejectsOrdinaryNics() {
+        Assert.assertFalse(ConfigDriveBuilder.isDirectRoutedNic(sharedNicProfile()));
+        Assert.assertFalse(ConfigDriveBuilder.isDirectRoutedNic(null));
+        // a /32 with an ordinary gateway is not direct routed
+        NicProfile hostMaskOnly = sharedNicProfile();
+        hostMaskOnly.setIPv4Netmask("255.255.255.255");
+        Assert.assertFalse(ConfigDriveBuilder.isDirectRoutedNic(hostMaskOnly));
+    }
+
+    @Test
+    public void ordinaryNicRouteGenerationIsUnchanged() {
+        JsonArray networks = ConfigDriveBuilder.getNetworksJsonArrayForNic(sharedNicProfile());
+        JsonObject ipv4Network = networks.get(0).getAsJsonObject();
+        JsonArray routes = ipv4Network.getAsJsonArray("routes");
+        Assert.assertEquals(1, routes.size());
+        JsonObject defaultRoute = routes.get(0).getAsJsonObject();
+        Assert.assertEquals("0.0.0.0", defaultRoute.get("network").getAsString());
+        Assert.assertEquals("0.0.0.0", defaultRoute.get("netmask").getAsString());
+        Assert.assertEquals("10.1.1.1", defaultRoute.get("gateway").getAsString());
+    }
+
+    @Test
+    public void networkDataIsGeneratedForDirectRoutedNicWithoutDhcpOrDns() throws Exception {
+        TemporaryFolder folder = new TemporaryFolder();
+        folder.create();
+        try {
+            Map<Long, List<Network.Service>> userDataOnly = Map.of(1L, List.of(Network.Service.UserData));
+            ConfigDriveBuilder.writeNetworkData(List.of(directRoutedNicProfile()), userDataOnly, folder.getRoot());
+            String json = FileUtils.readFileToString(new File(folder.getRoot(), "network_data.json"), com.cloud.utils.StringUtils.getPreferredCharset());
+            Assert.assertTrue("direct routed nic must appear in network_data.json", json.contains("203.0.113.55"));
+            Assert.assertTrue(json.contains("169.254.0.1"));
+        } finally {
+            folder.delete();
+        }
+    }
+
+    @Test
+    public void networkDataStaysEmptyForOrdinaryNicWithoutDhcpOrDns() throws Exception {
+        TemporaryFolder folder = new TemporaryFolder();
+        folder.create();
+        try {
+            Map<Long, List<Network.Service>> userDataOnly = Map.of(1L, List.of(Network.Service.UserData));
+            ConfigDriveBuilder.writeNetworkData(List.of(sharedNicProfile()), userDataOnly, folder.getRoot());
+            String json = FileUtils.readFileToString(new File(folder.getRoot(), "network_data.json"), com.cloud.utils.StringUtils.getPreferredCharset());
+            Assert.assertEquals("historical gate must be preserved for ordinary nics", "{}", json);
+        } finally {
+            folder.delete();
+        }
+    }
+
+}