Reuse VXLAN carriers across encrypted ingress groups

CHANGELOG.md

@@ -13,6 +13,7 @@ All Nullnet releases with the relative changes are documented in this file.
 - Per-service egress/ingress traffic filters: arbitrary AND/OR/group combinations of Country, Organization, Src IP (ingress), and Dst IP (egress) conditions, evaluated via `rpn-predicate-interpreter` — replaces the country-only egress/ingress policy ([#171](https://github.com/NullNet-ai/nullnet/pull/171) — fixes [#143](https://github.com/NullNet-ai/nullnet/issues/143))
 - Persist ingress and egress sessions to SQLite and show the full history on the Sessions page, filterable by status, service, direction, and policy verdict, with its own retention window ([#170](https://github.com/NullNet-ai/nullnet/pull/170) — fixes [#156](https://github.com/NullNet-ai/nullnet/issues/156))
 ### Changed
+- Use independently keyed client groups for all encrypted ingress and balance multiplexed groups across reusable standard VXLAN carriers ([#216](https://github.com/NullNet-ai/nullnet/pull/216)).
 - Keep routine daemon diagnostics debug-only to avoid release logging overhead under multiplexed load ([#215](https://github.com/NullNet-ai/nullnet/pull/215)).
 - Give each multiplexed cross-host ingress group its own IPsec key while retaining the shared VXLAN transport ([#215](https://github.com/NullNet-ai/nullnet/pull/215)).
 - Allow GeoIP/ASN enrichment and fallback database downloads to be disabled with `IPINFO_DISABLED` ([#215](https://github.com/NullNet-ai/nullnet/pull/215)).

README.md

@@ -40,9 +40,9 @@ Three binaries coordinate over a gRPC control plane:
 
 ## What you get
 
-- **On-demand tunnels** — point-to-point VXLAN links, built atomically across a whole
-  dependency chain, encrypted by default (MACsec / XFRM), and reclaimed once the connections
-  they carry are gone.
+- **On-demand tunnels** — encrypted links built atomically across a whole dependency chain and
+  reclaimed once the connections they carry are gone. Encrypted ingress uses isolated client
+  groups that can share a reusable VXLAN carrier without sharing keys.
 - **Default-deny host firewall** — an eBPF classifier on every node's uplink NIC, with one
   global allowlist decided server-side so there's a single point of configuration.
 - **Ingress that knows what it's serving** — Let's Encrypt certificates with automatic

SETUP.md

@@ -29,6 +29,10 @@ plus the shared gRPC interface.
   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   ```
 
+Encrypted VXLAN ingress requires Linux VXLAN VNI filtering (`vnifilter`, available in mainline
+Linux since 5.18). Nullnet uses it to run several metadata VXLAN carriers on UDP 4791 and assign
+each live client group to exactly one carrier.
+
 The repository should be cloned under `/root` so the provided `setup-*.sh` scripts and
 `.service` units work without changes.
 Run the setup scripts as root. The supplied systemd services also run as root; privileged commands execute directly without sudo.
@@ -260,17 +264,17 @@ Run the setup scripts as root. The supplied systemd services also run as root; p
 - `timeout` controls proxy-reachability: when present the service is a proxy-reachable entry point
   with that per-client idle timeout in seconds (`0` disables the timeout); omit it to keep the
   service off the proxy (backend-only)
-- `max_networks` is an optional hard cap on dedicated ingress networks for a proxy-reachable
+- `max_networks` is an optional hard cap on ingress carrier VXLANs for a proxy-reachable
   service. Independently of that setting, encrypted VXLAN ingress switches new clients to
   multiplexing when the proxy's RPC backlog and slot-wait time grow for consecutive windows. A
   host-scoped controller first pauses dedicated setups, then additively admits one more probe per
   healthy window; renewed congestion immediately returns the rate to zero. Stable queues do not
   activate it, and only sustained successful probes under traffic remove the limit (idle time does
-  not count as recovery). Multiplexed clients get one policy
-  generation and encryption key per client IP over a shared VXLAN. Cross-host groups use marked
-  per-group IPsec associations over the shared VXLAN; packet metadata and endpoint maps preserve
-  per-client isolation. Same-host multiplexing uses the same isolation policy with
-  direct eBPF redirection and no VXLAN hairpin.
+  not count as recovery). Every client IP gets its own group, generation, encryption key, and
+  marked IPsec associations. A standard group seeds a VNI-filtered carrier; multiplexed groups are
+  assigned to the least-loaded live carrier, so both paths use the same dataplane and UDP 4791.
+  Packet metadata and endpoint maps preserve per-client isolation. Same-host groups use the same
+  isolation policy with direct eBPF redirection and no VXLAN hairpin.
 - `proxy_dependencies` is a list of independent dep chains walked when the service is reached via a
   `Proxy` RPC from nullnet-proxy; each inner array is one linear branch and all branches are brought
   up in parallel
@@ -441,14 +445,15 @@ details. Stop observation to restore the saved config and review added/unused
 backends. Applying updates only backend lists. Completed observations remain
 available; results cannot overwrite a configuration edited since activation.
 
-VXLAN edges create fresh dedicated interfaces and use fixed TC redirects;
-Nullnet shares UDP 47890, separate from Docker Swarm’s UDP 4789. Egress policy rules use priorities 1000–1015 and protocol 242; reserve these for Nullnet.
+Dependency, backend-trigger, and egress VXLAN edges create dedicated interfaces on UDP 47890.
+Encrypted ingress uses VNI-filtered group carriers on UDP 4791; each carrier can host multiple
+independently keyed client groups. Both ports are separate from Docker Swarm's UDP 4789. Egress
+policy rules use priorities 1000–1015 and protocol 242; reserve these for Nullnet.
 Reserve TC blocks 7801–7832 and interface names `nn-tc-0` through `nn-tc-15`
-for Nullnet's shared classifiers. These down dummy interfaces carry no traffic;
-each edge still has dedicated interfaces and encryption keys. Startup removes
+for Nullnet's shared classifiers. These down dummy interfaces carry no traffic. Startup removes
 stale holders using their reserved name, group, and device type before rebuilding the classifiers.
 `br_<id>_<side>` is the gateway interface name, not a Linux bridge. Startup
 keeps routed overlay traffic allowed by owned interface groups while preserving
-the host's existing FORWARD policy. Pooling and policy multiplexing are deferred.
+the host's existing FORWARD policy.
 
 The client and proxy services set `LimitNOFILE=524288`: fresh endpoints retain netlink sockets, and each proxied connection needs downstream and upstream sockets. Update both installed units and run `systemctl daemon-reload` when upgrading.

docs/architecture.md

@@ -17,7 +17,7 @@ NullNet architecture is made of three components that coordinate over a gRPC con
 
 Each time a service is requested — through the proxy or by another service — the server walks the whole chain that request will travel. <br>
 If service A needs B, and B in turn needs C, that one request opens every link along the way at the same time, so the full path is ready atomically. <br>
-Each link is a dedicated VXLAN tunnel, private between the two machines, and only stays open while it's being used; once detected as idle via a configurable timeout, the server tears it down.
+Each link is isolated and only stays open while it is being used; once detected as idle via a configurable timeout, the server tears it down. Encrypted ingress groups have independent keys and policy state while standard and multiplexed groups share the same reusable VXLAN carrier implementation; dependency, backend-trigger, and egress links retain dedicated VXLAN interfaces.
 
 The result: there is no broad "internal network" for an attacker to roam around inside if they break in somewhere.<br>
 The only paths that exist are the narrow, temporary ones carrying legitimate traffic as determined by the current network activity and defined topology. <br>

ebpf/src/policy.rs

@@ -1,4 +1,4 @@
-//! Permission-gated IPv4 forwarding over shared policy transports.
+//! Permission-gated IPv4 forwarding over reusable group carriers.
 use aya_ebpf::{
     EbpfContext,
     bindings::{TC_ACT_SHOT, bpf_tunnel_key},
@@ -125,7 +125,7 @@ fn outgoing(ctx: &TcContext) -> Result<i32, i64> {
     }
 
     let mut tunnel: bpf_tunnel_key = unsafe { zeroed() };
-    tunnel.tunnel_id = (rule.generation & 0xff_ffff) as u32;
+    tunnel.tunnel_id = rule.vni;
     tunnel.tunnel_ttl = 64;
     tunnel.__bindgen_anon_1.remote_ipv4 = rule.underlay_remote;
     tunnel.__bindgen_anon_3.local_ipv4 = rule.underlay_local;
@@ -166,7 +166,8 @@ fn incoming(ctx: &TcContext) -> Result<i32, i64> {
         return Err(-1);
     }
     if unsafe { tunnel.__bindgen_anon_1.remote_ipv4 } != rule.underlay_remote
-        || (u64::from(option & 0xffff) << 24 | u64::from(tunnel.tunnel_id)) != rule.generation
+        || tunnel.tunnel_id != rule.vni
+        || u64::from(option & 0xffff) != rule.generation >> 24
     {
         return Err(-1);
     }

members/nullnet-client/src/commands/mod.rs

@@ -12,6 +12,7 @@ pub(crate) mod endpoint_namespace;
 mod endpoint_tc;
 mod native_macsec;
 mod native_netlink;
+mod native_vxlan;
 mod native_xfrm;
 mod netlink;
 pub(crate) mod nfqueue;

members/nullnet-client/src/commands/native_vxlan.rs

@@ -0,0 +1,85 @@
+//! VXLAN VNI-filter membership through route Netlink.
+use super::native_netlink::{NESTED, NativeSocket, REQUEST, attr, blocking};
+use std::io;
+
+const RTM_NEWTUNNEL: u16 = 120;
+const RTM_DELTUNNEL: u16 = 121;
+const RTM_DELLINK: u16 = 17;
+const VXLAN_VNIFILTER_ENTRY: u16 = 1;
+const VXLAN_VNIFILTER_ENTRY_START: u16 = 1;
+
+fn message(ifindex: u32, vni: u32) -> Vec<u8> {
+    let mut body = vec![libc::AF_BRIDGE as u8, 0, 0, 0];
+    body.extend(ifindex.to_ne_bytes());
+    let entry = attr(VXLAN_VNIFILTER_ENTRY_START, &vni.to_ne_bytes());
+    body.extend(attr(VXLAN_VNIFILTER_ENTRY | NESTED, &entry));
+    body
+}
+
+async fn modify(ifindex: u32, vni: u32, add: bool) -> io::Result<()> {
+    blocking(move || {
+        let kind = if add { RTM_NEWTUNNEL } else { RTM_DELTUNNEL };
+        let result = NativeSocket::new(libc::NETLINK_ROUTE as _)?.request(
+            kind,
+            REQUEST,
+            &message(ifindex, vni),
+        );
+        match result {
+            Ok(_) => Ok(()),
+            Err(error)
+                if !add && matches!(error.raw_os_error(), Some(libc::ENOENT | libc::ESRCH)) =>
+            {
+                Ok(())
+            }
+            Err(error) => Err(error),
+        }
+    })
+    .await
+}
+
+pub(super) async fn add_vni(ifindex: u32, vni: u32) -> io::Result<()> {
+    modify(ifindex, vni, true).await
+}
+
+pub(super) async fn remove_vni(ifindex: u32, vni: u32) -> io::Result<()> {
+    modify(ifindex, vni, false).await
+}
+
+pub(super) async fn remove_link(ifindex: u32) -> io::Result<()> {
+    blocking(move || {
+        let mut body = vec![libc::AF_UNSPEC as u8, 0, 0, 0];
+        body.extend(ifindex.to_ne_bytes());
+        body.extend([0; 8]);
+        let result =
+            NativeSocket::new(libc::NETLINK_ROUTE as _)?.request(RTM_DELLINK, REQUEST, &body);
+        match result {
+            Ok(_) => Ok(()),
+            Err(error) if matches!(error.raw_os_error(), Some(libc::ENODEV | libc::ENOENT)) => {
+                Ok(())
+            }
+            Err(error) => Err(error),
+        }
+    })
+    .await
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use crate::commands::native_netlink::attrs;
+
+    #[test]
+    fn vni_filter_message_matches_tunnel_uapi() {
+        let body = message(17, 0x12_3456);
+        assert_eq!(body[..4], [libc::AF_BRIDGE as u8, 0, 0, 0]);
+        assert_eq!(u32::from_ne_bytes(body[4..8].try_into().unwrap()), 17);
+        let outer = attrs(&body[8..]).unwrap();
+        assert_eq!(outer[0].0, VXLAN_VNIFILTER_ENTRY);
+        let inner = attrs(outer[0].1).unwrap();
+        assert_eq!(inner[0].0, VXLAN_VNIFILTER_ENTRY_START);
+        assert_eq!(
+            u32::from_ne_bytes(inner[0].1.try_into().unwrap()),
+            0x12_3456
+        );
+    }
+}