NSX: add native route-based Site-to-Site VPN support

[Dogface2k]

Description

This PR implements native, route-based Site-to-Site VPN for NSX NAT-mode VPCs. The VPN terminates on the VPC's NSX Tier-1 gateway; the CloudStack virtual router remains outside the IPsec data path.

The implementation makes the existing built-in Vpn/Nsx service mapping functional and supersedes the capability-removal approach in the closed #13765. It does not change seeded offerings or the database schema.

The main lifecycle is provider-dispatched through Site2SiteVpnServiceProvider instead of adding another provider-specific branch to Site2SiteVpnManagerImpl:

• gateway creation validates or acquires a dedicated public IP, records whether NSX owns that allocation, and creates a deterministic Tier-1 IPSecVpnService and local endpoint;
• connection start validates CloudStack crypto policy, creates NSX IKE/tunnel/DPD profiles, a RouteBasedIPSecVpnSession, and a VTI named default-tunnel-interface;
• each remote CIDR receives a Tier-1 static route and a NO_SNAT rule so traffic is not rewritten by the VPC's catch-all source NAT before entering the tunnel;
• stop disables the session and removes connection routes/NAT exemptions; restart recreates them; permanent deletion also removes the session and profiles;
• gateway and VPC teardown remove VPN objects before the Tier-1 gateway is deleted;
• a scheduled status poll maps NSX detailed session status back to CloudStack connection state without allowing a transient query failure to corrupt the last known state.

The public endpoint is intentionally separate from the VPC source-NAT address. NSX requires the IPsec local endpoint to differ from the Tier-1 uplink address. Auto-acquired endpoint addresses are released on gateway deletion; operator-supplied addresses are detached but not released.

Compatibility and failure handling are explicit:

• the new provider SPI methods have defaults, so existing virtual-router and Netris implementations retain their behavior;
• legacy gateway rows without an ownership marker retain the historical virtual-router fallback;
• persisted provider ownership prevents an offering change from redirecting an existing gateway to a different implementation;
• gateway creation refuses to adopt a pre-existing deterministic NSX service and reports ambiguous rollback state instead of silently releasing an address that may already be active;
• Tier-1 VPN mutations share a per-gateway lock with Tier-1 teardown;
• PSKs are excluded from CloudStack command logging while remaining present on the agent wire payload;
• unsupported algorithms, invalid lifetimes, oversized PSKs, missing VPC/gateway rows, and deterministic VTI collisions fail with specific errors.

The Tier-0 must redistribute TIER1_IPSEC_LOCAL_ENDPOINT routes for the dedicated gateway address to be externally reachable.

Related context: #13764 and the superseded closed PR #13765.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

Not applicable.

How Has This Been Tested?

The exact upstream-clean commit was built and tested from current main at 4f117071c9397b1e4714c8fb61c384883c872842 with Java 17:

NSX plugin suite:
Tests run: 179, Failures: 0, Errors: 0, Skipped: 0

Full server suite:
Tests run: 3625, Failures: 0, Errors: 0, Skipped: 9

Site2SiteVpnManagerImplTest (included in the server total):
Tests run: 51, Failures: 0, Errors: 0, Skipped: 0

The deployed 4.22.1.0 focused NSX run after the lifecycle hardening reported 206/206 tests passing, including testAddVpnConnectionRoutesRetriesMarkedForDeletion. This is an additional backport result; the local checkout used for review has no JDK/Maven executable, so repository CI remains the authoritative build check.\n\nThe changed api, server, and NSX plugin modules also passed:

Checkstyle: 0 violations in all three modules
Apache RAT: BUILD SUCCESS for all three modules
Packaging: BUILD SUCCESS for all three modules
git diff --check: clean

The NSX suite covers all six management-server/agent command paths, including a real CloudStack Request/Gson round trip proving that the PSK remains on the wire while @LogLevel(Off) excludes it from log serialization.

An earlier 4.22.1 backport/prototype was exercised against NSX 4.2.4. Live validation exposed five integration defects that are addressed in this branch: virtual-router command leakage, the virtual-router monitor overwriting NSX state, NSX marked-for-deletion races, idle tunnels being removed by periodic DPD, and catch-all SNAT rewriting VPN traffic.

Post-deploy live control-plane validation was run on 2026-08-02 against an isolated test VPC on 4.22.1.0/NSX 4.2.4: a fresh NSX VPN gateway, customer gateway, and connection were created, then reset immediately; the reset job succeeded. The disposable connection, customer gateway, VPN gateway, and VPC were then deleted successfully. Production connection 8 remained Connected before and after the run. This validates the management-plane lifecycle only: no tunnel-traffic success is claimed from the intentionally unreachable RFC 5737 peer, and no production connection was modified. The earlier marked-for-deletion 431 occurred on a pre-deploy build and is not presented as a current-build failure. Full peer traffic and Tier-0 endpoint-route redistribution remain environment-dependent validation prerequisites.

How did you try to break this feature and the system with this change?

The negative and compatibility coverage exercises the failure boundaries rather than only the happy path:

• requested endpoint IPs are rejected if they are outside the VPC, inactive, source-NAT/system addresses, or already used by static NAT, firewall, port-forwarding, or load-balancing rules;
• auto-acquired and operator-supplied endpoint addresses have different, verified release behavior;
• partial and ambiguous gateway creation failures preserve enough ownership state for safe operator recovery and never adopt an unrelated deterministic NSX service;
• connection creation rolls back partial sessions, routes, and NAT exemptions;
• changed peer CIDR lists do not strand old routes or NO_SNAT rules during deletion;
• deterministic VTI address collisions are detected against live Tier-1 sessions and fail closed;
• invalid/unsupported cipher, hash, DH group, IKE version, lifetime, and PSK values are rejected before an NSX mutation;
• stop, restart, reset, permanent connection deletion, gateway deletion, missing-row cleanup, and full VPC teardown are separately covered;
• source-NAT rule demotion preserves the complete original rule and sequence, and the exact sequence is restored when the final VPN exemption is removed;
• operator-created VPN services on the Tier-1 are not swept during CloudStack teardown;
• provider ownership remains stable if an offering mapping changes after gateway creation;
• legacy virtual-router VPN selection and command generation remain covered, while NSX-owned connections are excluded from virtual-router commands and monitoring;
• repeated status-query failures alert without forcing a false state transition, and successful UP, DOWN, degraded, and not-found results map to the intended CloudStack states;
• missing VPC or gateway state fails start/stop explicitly instead of reporting a false success, while permanent deletion remains idempotent for cleanup.

No generic CloudStack command proxy, broad exception suppression, schema migration, retry of an ambiguous CloudStack mutation, or unrelated UI/CKS/network-offering change is included.

[Dogface2k]

Follow-up validation and lifecycle hardening

Commit 98cd4ed6f4 adds the lifecycle hardening identified by live NSX validation.

• The existing marked-for-deletion retry is now shared by route-based session creation and the complete static-route/NO_SNAT programming operation. It remains bounded at 24 attempts with a 15-second interval.
• NsxApiClientTest.testAddVpnConnectionRoutesRetriesMarkedForDeletion forces the NSX marked-for-deletion response during route creation and verifies the deterministic route is retried before NO_SNAT programming continues.
• State changes use a read-modify-write of the existing route-based session, preserving the mandatory tunnel_interfaces field implicated by the pre-deploy failure.
• VPN connection, gateway, customer-gateway, and VPC lifecycle operations use VPC-scoped synchronization and deterministic cleanup.

Live validation on 4.22.1.0 with NSX 4.2.4 used an intentionally unreachable RFC 5737 test peer. The connection reached Disconnected; the first reset reproduced the marked-for-deletion 431, the same reset succeeded after the purge window, and connection/customer-gateway/VPN-gateway/VPC deletion completed with the dedicated test IP released. No production connection was modified, and no tunnel-traffic success is claimed from this test peer.

The supplied pre-deploy tunnel_interfaces error is a distinct historical failure; the current read-modify-write path passed the stop half of the live reset test. The local checkout has no JDK/Maven executable, so the new focused test must be run by repository CI; git diff --check is clean.

[Dogface2k]

Additional live data-plane evidence

Operator-side testing between 192.168.46.5 and 10.35.0.1 succeeded in both directions over the documented FortiGate/NSX path. The original direction returned 2/2 ICMP replies with 0% loss and 17.45 ms average latency; traceroute showed 6x.xxx.xxx.xxx (FortiGate) followed by 10.35.0.1.

CloudStack connection 8 remained Connected before and after. This is positive bidirectional endpoint-reachability evidence through the VPN path.