Ability to capture packets of Instance NICs on KVM

[wido]

As an Operator I would like to capture the network traffic of a specific NIC of a running Instance, so that I can debug network problems and comply with lawful interception requests, without having to log in to the hypervisor myself and figure out which tap device belongs to which Instance.

I/we have done this too many times and its always a mess. VMs get migrated, rebooted, etc. Sometimes traffic needs to be captured for a while for debugging things.

Why

Today there is no way to capture the traffic of an Instance from CloudStack. When an operator needs a packet capture, the procedure is manual:

1. Find out which host the Instance is running on.
2. SSH into that KVM host.
3. Work out which vnetX tap device belongs to the NIC in question (virsh domiflist, matching MAC addresses).
4. Start tcpdump by hand, usually in a screen/tmux session or with nohup.
5. Remember to stop it, and redo all of the above whenever the Instance is stopped, started or live-migrated to another host.

This is error-prone, requires giving people shell access to hypervisors, leaves orphaned tcpdump processes behind, and silently stops working the moment an Instance migrates.

There are two distinct drivers for wanting this in the product:

Debugging. Network issues inside an Instance (MTU problems, DHCP failures, unexpected drops, asymmetric routing, misbehaving guest applications) are frequently only diagnosable from a packet capture taken at the hypervisor, outside the guest. Being able to switch that on from the API for a single NIC would substantially shorten the debugging loop, especially when the guest itself is not reachable or not under the operator's control.

Lawful interception. Operators running CloudStack as a public cloud or hosting platform can be legally required to intercept the traffic of a specific customer Instance and hand it over to the requesting authority. The exact requirements (what is captured, where it is stored, how it is transported, retention, chain of custody) differ per country, per operator and per request, so CloudStack cannot sensibly implement the handling of that data itself. What it can do is provide the mechanism to start and stop the capture for a specific NIC, reliably and auditably.

Proposed solution

Because the requirements around what to capture and what to do with the data vary so much per organisation, the proposal is that CloudStack orchestrates when and where a capture runs, and leaves how it runs to the operator:

• Packet capture becomes a flag on an Instance NIC, toggled through new root-admin-only APIs (enablePacketCapture, disablePacketCapture, getPacketCaptureStatus, taking a nicid).
• When enabled, the KVM agent resolves the host-side tap device of that NIC by MAC address and starts a systemd template unit (cloudstack-pcap@<device>.service) on the host the Instance is running on.
• The agent writes the context of the NIC to an environment file before starting the unit: Instance name and UUID, NIC UUID, MAC address, tap device, bridge, IPv4 and IPv6 address, and the network UUID.
• The unit executes a script shipped as an editable example in pcap-capture.sh. The example runs tcpdump writing to a rotated file in /tmp, and every operator is expected to replace it with something that fits their situation — a different filter, a different output location, streaming to a central collector, encrypting the output, or whatever their lawful interception process requires. It is installed as a config file so package upgrades do not overwrite local changes.
• The flag follows the Instance: the capture is restarted on the new host after a start or a live migration, and the systemd unit is bound to the tap device, so it stops automatically when the Instance stops, migrates away or the NIC is unplugged.

This keeps CloudStack's part small and hypervisor-side policy fully in the operator's hands. No new database table, no capture files flowing through the management server, no secondary storage involved.

Notes and considerations

• Given the nature of the feature, the APIs are root-admin only. Feedback on whether domain admins should be able to enable this for Instances in their domain is welcome.
• This is KVM only. Other hypervisors would need their own implementation of the host-side part.
• Enabling capture on a stopped Instance only sets the flag; the capture starts the next time the Instance starts.
• The capture itself runs on the hypervisor and consumes CPU and disk there, which is why the size/rotation policy deliberately lives in the operator's script rather than in CloudStack.

I have an implementation of the above ready and will open a PR referencing this issue.

[wido]

Working on this at the moment in this repository/branch: https://github.com/wido/cloudstack/commits/packet-capture/