feat(multi-node): engine DaemonSet bundling + Caddy sticky LB + GPU/CPU pool placement

[aucahuasi]

Summary

Enables Graphistry to run correctly across multiple GPU nodes from a single Helm release. All GPU-bound services (nginx, forge-etl-python, dask-cuda-worker, streamgl-gpu, streamgl-viz, streamgl-sessions) are consolidated into a single engine DaemonSet pod per GPU node, fronted by Caddy as a sticky-LB L7. The prior multi-node model (leader + follower helm releases per node) is retired in favour of a single-namespace deployment.

The architectural change

engine DaemonSet (one bundled pod per GPU node)

New templates/engine/engine-daemonset.yaml colocates nginx, fep, dask, and the streamgl-* services as sibling containers in one pod per GPU node. Tier-aware: 3 containers (nginx + fep + dask) at analytics, 6 containers (adds streamgl-{viz,sessions,gpu}) at viz/full. Intra-pod hostAliases pin the app-layer hostnames (streamgl-viz, streamgl-gpu, forge-etl-python, dask-cuda-worker) to 127.0.0.1, so every intra-stack HTTP hop is localhost. streamgl-gpu's PM2 localhost IPC continues to work unchanged because all gpu-router and PM2-forked gpu-worker children are intra-pod.

Companion templates:

• templates/engine/engine-nginx-cfg.yml -- supplementary nginx conf.d ConfigMap that adds an intra-pod :8080 Host-header dispatcher, loaded alongside the production default.conf.template.
• templates/engine/engine-service.yaml -- engine-headless Service for the Caddy upstream pool (gated on tier.analytics) plus four shim Services (streamgl-viz, streamgl-sessions, streamgl-gpu, forge-etl-python, gated on tier.viz) so the production nginx FQDN-suffixed hostnames keep resolving through CoreDNS for any path that does not go through hostAliases.

This bundle supersedes the earlier streamgl-gpu router/spawner split (added in 511876d, retired in 699bc4f). Bundling keeps PM2 IPC working unchanged, makes every intra-stack hop localhost (lower latency, better data locality), and has the same blast radius as a router-split design because browser sessions are cookie-pinned across engine pods anyway. Per-node redundancy comes from running multiple engine pods (one per GPU node) under the headless Service.

The engine DaemonSet's forge-etl-python container now sets GRAPHISTRY_DASK_LOCAL_AFFINITY=1 so each fep submission's persist/compute carries a soft worker-affinity hint pinning it to the dask-cuda-worker on the same pod (matched by HOSTNAME prefix). This is a cross-repo dependency on graphistry/graphistry#3097, which adds the dask_affinity helper that reads this env var and produces the kwargs. With multiple GPU nodes registered to the scheduler, the hint eliminates the cross-node shuffle path that would otherwise ship ~256 MiB partitions over the cluster overlay (~2x ETL speedup measured at 4/92/512 MiB). The hint is soft (allow_other_workers=True) and decays to a no-op on miss, so single-node deployments are byte-identical to pre-change.

Caddy as sticky-LB L7

Caddy now load-balances browser sessions across engine-headless using Caddy's dynamic a resolver (live pod-IP refresh, refresh 10s) and lb_policy cookie graphistry_sticky for session stickiness. The cookie is HMAC-signed on engine.cookieSecret; WebSocket upgrades to /graph/socket.io and /streamgl/* ride the same pin.

New operator-tunable knobs in values.yaml:

• caddy.enabled -- toggle Caddy + caddy-ingress entirely. false lets operators front engine-headless with their own ingress controller (Pattern B); the operator owns TLS termination and cookie affinity in that case. The render gate on caddy-cfg.yml, caddy-deployment.yaml, and caddy-ingress.yml is now tier.analytics AND caddy.enabled.
• caddy.tls.mode -- external | self | off. external trusts X-Forwarded-Proto: https from private_ranges so the sticky cookie keeps Secure+SameSite=None. self terminates from existingSecret or ACME. off is plain HTTP only.
• caddy.lb.fallback -- first-time-assignment policy when no cookie is set yet (default round_robin).
• caddy.service -- type / loadBalancerIP / nodePort / nodePortHttps / annotations / externalTrafficPolicy. Cloud, Tanzu, MetalLB, NodePort all selectable from values; per-platform annotation hints inline.
• caddy.upstreamImage -- escape hatch to use caddy:2.10-alpine directly while the bundled wrapper image lags upstream PR reverseproxy: cookie should be Secure and SameSite=None when TLS caddyserver/caddy#6115 (the cookie-LB Secure+SameSite=None fix). Liveness probe switches to httpGet when set (no curl in the official image).

Caddy Pod template gains a checksum/config annotation that hashes the rendered Caddyfile into the Pod spec, so any change to TLS mode / lb.fallback / cookieSecret triggers an automatic rollout on helm upgrade. Without this, ConfigMap-only changes left Caddy with stale parsed config in memory until something forced a pod bounce.

Heterogeneous cluster placement: GPU/CPU pools + dedicated-tenant taints

Two new operator-tunable values let the chart land correctly on clusters that mix GPU and CPU node pools, or that use NodePool/managed taints to keep workloads off shared infra:

• engine.nodeSelector (default {}) -- when non-empty, the engine DaemonSet uses this selector instead of global.nodeSelector (falls back to global when empty). Lets operators target the engine pod at GPU-labelled nodes (graphistry.io/role=gpu) while keeping the chart's CPU-side workloads (caddy, nexus, redis, dask-scheduler, notebook, pivot, gak-{private,public}) on a separate pool via global.nodeSelector (graphistry.io/role=cpu).
• global.tolerations (default []) -- applied to every chart-rendered workload (11 templates: caddy, dask-scheduler, gak-private/public, http-tools netshoot+whoami, nexus, notebook, pivot, redis, engine DaemonSet). Lets operators opt the chart's pods into nodes carrying operator-defined taints: NVIDIA GPU Operator's nvidia.com/gpu=true:NoSchedule, GKE/EKS managed GPU node-pool taints (nvidia.com/gpu=present:NoSchedule), dedicated-tenant taints (dedicated=graphistry:NoSchedule), etc. Empty [] keeps current behaviour byte-identical. Tolerations are permissive (not directive), so adding a GPU-pool toleration to global.tolerations is harmless for non-GPU workloads -- they still land on whichever node global.nodeSelector admits them to.

Bug fix: dcgm-exporter and http-tools (netshoot, whoami) were the only chart workloads that did not honour global.nodeSelector; now they do. Pre-fix these would leak onto non-GPU nodes in mixed-pool clusters even when the operator constrained the rest of the chart to GPU nodes.

PVC tier-gating dropped (correctness fix)

gak-private, gak-public, and uploads-files PersistentVolumeClaims are no longer gated on tier.full / tier.analytics. With Retain reclaim policy, gating made tier downgrades leave PVs Released with stale claimRef; later upgrades created new PVCs with fresh UIDs that no longer matched, and the new PVCs sat Pending indefinitely. Consuming Deployments stay tier-gated; only the storage object is unconditional.

Validation

End-to-end on a 2-node k3s cluster (node1 = k3s server with NFS server colocated, node2 = k3s agent / NFS client) with NFS RWX storage, tier viz, two engine pods (one per node), Caddy as ClusterIP fronted by k3s Traefik on node1.

• ETL 5-call sequence at analytics tier: /readarrow, /upload, /preshape, /properties, /download all returned 200 on both nodes. Local-worker affinity hint (added in graphistry_master) was exercised: each fep submission's persist/compute kwargs named the dask-cuda-worker on its own node, verified by HOSTNAME-prefix match against scheduler worker info. Behavior decayed to a no-op when a fep call landed on a node without a registered local worker, so single-node deployments are byte-identical to pre-change.
• Multi-node session test at viz tier: browsers opened sessions against the public Caddy endpoint on node1. Each browser pinned to one engine pod via the graphistry_sticky cookie across page reloads and WebSocket reconnects; sessions on node1 were undisturbed by traffic landing on node2, and vice versa. Verified sticky distribution by tab count vs cookie value. Long-idle sessions (60s+ inactivity) survived without disconnect.
• Tier transitions: upgraded viz -> analytics -> viz against a release with non-empty PVCs. gak/uploads PVCs survived the round trip and re-bound to their existing PVs (PV/PVC UID stable, claimRef intact). Pre-fix this would have left the new PVCs Pending.
• Caddyfile config rollout: edited only caddy.lb.fallback in values and re-ran helm upgrade. Caddy Pod rolled automatically because the rendered Caddyfile checksum changed; ConfigMap-only edits no longer require manual pod bounce.
• Concurrent upload test (carried over from earlier validation in this branch): 1.77 GB of concurrent uploads across both ingress replicas (now both inside engine pods) with zero errors; the only render glitch observed on a 7th concurrent tab was the HTTP/1.1 6-connections-per-origin browser limit on plain-HTTP localhost (documented upstream, dissolves under HTTPS/HTTP/2), not chart or backend.
• Telemetry stack on multi-node k3s (3 GPUs across 2 nodes): Pre-fix, Grafana's DCGM dashboard showed metrics for only 1 GPU because prometheus scraped the Service VIP and kube-proxy load-balanced to a single DaemonSet pod per scrape. Post-fix (kubernetes_sd_configs role: pod + relabel rules), all 3 GPUs visible with stable per-node labels. node-exporter validated equivalently. The new prometheus-rbac.yaml ServiceAccount + Role + RoleBinding satisfies prometheus's apiserver pod-discovery; automountServiceAccountToken: true is explicit on the prometheus pod for clusters that flip the namespace default.
• Platform-tier nexus-proxy end-to-end (tier=platform, postgres + nexus + nexus-proxy slice): All five v1-to-v2 deprecation shims (/etl, /api/check, /api/encrypt, /api/decrypt, /api/v1/etl/vgraph/*) returned 410 Gone with the documented upgrade message. Live /api/v1/* routes (/datasets/, /files/, /organization/, /team/, /named-endpoint/, /my/user/entitlements/) returned 200 with real data after a Django session login at /accounts/login/. v2 ETL routes routed correctly through nginx (returned upstream errors at platform tier as expected — the GPU backends streamgl-gpu and forge-etl-python intentionally don't render at this tier). Tier transition platform -> analytics removed nexus-proxy and brought up the engine DaemonSet's nginx container; analytics -> platform reversed it cleanly.
• Caddy stability: Two crash modes fixed. Single-line handle @grafana { reverse_proxy ... } block syntax tripped Caddy v2's parser ("Unexpected next token after '{' on same line") — expanded to multi-line form. The /caddy/health/ endpoint was being intercepted by the catch-all handle and proxied to engine-headless (no endpoints) after respond wrote 200, because respond is non-terminal in Caddy v2; wrapping in a terminal handle /caddy/health/ { respond 200 ... } broke the resulting 30s SIGTERM-then-restart liveness loop. New templates/caddy/_helpers.tpl with graphistry.caddy.healthHandle and graphistry.caddy.telemetryHandles defines removes 3-4× duplication of identical handle blocks across the tls.mode = external | self | off branches.

Telemetry as a properly-structured subchart

telemetry is now a Helm subchart consumed via dependencies: with condition: global.ENABLE_OPEN_TELEMETRY. Helm fully prunes the subchart's templates when the flag is off (was previously per-template {{- if }} gates inside the parent).
Telemetry-specific values move to a top-level telemetry: block in the parent's values.yaml (subchart-canonical layout); only cross-cutting values (OTLP endpoint, instance name, image-pull config, default scheduling, storage class) stay under global.*.

The parent's shared _helpers.tpl (graphistry.tier.* helpers) is extracted into a new graphistry-common library subchart so the telemetry subchart can reuse the same tier gating via a dependency entry.

New operator-tunable knobs:

• telemetry.dcgmExporter.useExternal + telemetry.dcgmExporter.externalEndpoint (default false / "") — when true the chart skips its own dcgm-exporter DaemonSet+Service and points prometheus + otel-collector at an externally-managed endpoint.
  Use cases: GKE with NVIDIA GPU Operator (the bundled exporter image fails on Container-Optimized OS), or any cluster already running the GPU Operator's DCGM module (avoids two DaemonSets scraping the same GPUs). Format host:port, no scheme.
• telemetry.nodeExporter.useExternal + telemetry.nodeExporter.externalEndpoint — same pattern for clusters already running kube-prometheus-stack's node-exporter; avoids two DaemonSets per node.
• telemetry.prometheus.enableAdminAPI (default false) — when true passes --web.enable-admin-api to prometheus, enabling tsdb/delete_series, tsdb/clean_tombstones, snapshot, and shutdown. Mirrors kube-prometheus-stack's prometheusSpec.enableAdminAPI default. Off in production; flip in dev/test to drop stale series after scrape-config refactors.
• telemetry.prometheus.retention (default 15d) — local TSDB retention.
• telemetry.{prometheus,jaeger,grafana}.persistence.{enabled,size,storageClassName} — per-component PVC config for the otel-collector backend stack. Storage class falls back to global.storageClassNameOverride then retain-sc. Without persistence,
  Grafana dashboards/sessions and Jaeger traces are lost on every pod restart.

Topology + correctness changes:

• Prometheus is now a single-replica Deployment with Recreate strategy + RWO PVC (was per-node DaemonSet, no global view, no retention). Multi-replica HA is out of scope; users who need it should add Thanos or remote_write to a managed backend in their overrides.
• prometheus scrape config: dcgm-exporter and node-exporter jobs switched from Service-VIP static_configs to kubernetes_sd_configs with role: pod + relabel rules that set the node label from pod_node_name and __address__ from pod_ip. Pre-fix prometheus scraped the Service VIP and kube-proxy load-balanced to one DaemonSet pod per scrape, silently dropping the other nodes' GPU/host metrics from dashboards. Post-fix every DaemonSet pod is a distinct scrape target with a stable per-node label.
• New templates/prometheus-rbac.yaml: ServiceAccount + namespace-scoped Role + RoleBinding granting read-only pods / services / endpoints get / list / watch. Required by prometheus's kubernetes_sd_configs apiserver discovery. The prometheus pod sets automountServiceAccountToken: true explicitly — the K8s default is true, but locked-down clusters (some Tanzu/OpenShift profiles, hardened GKE namespaces) flip it to false at namespace level, which would silently break pod discovery.
• otel-collector ConfigMap unified: previously otel-collector-cloud-configmap.yaml and otel-collector-configmap.yaml rendered separately based on OTEL_CLOUD_MODE; now one templated ConfigMap with cloud-mode/self-hosted branches inline. Cloud-mode credentials read from a pre-created Secret via secretKeyRef (never inlined into values.yaml).
• checksum/config annotation on otel-collector / prometheus / grafana / jaeger pod templates — ConfigMap content changes now trigger a rollout on helm upgrade (was previously a no-op until manual pod delete).
• Per-component telemetry Ingresses (grafana-ingress.yaml, jaeger-ingress.yaml, prometheus-ingress.yaml) removed; replaced by Caddy path routes (/grafana, /jaeger, /prometheus) on the parent chart's main Ingress.

Platform-tier nexus-proxy (nginx fronting nexus, v1-to-v2 endpoint rewrites)

New templates/nexus-proxy/nexus-proxy-deployment.yaml + nexus-proxy-service.yaml: platform-tier-only nginx Deployment + Service that fronts nexus and provides the v1-to-v2 endpoint rewrites + deprecated-endpoint 410 shims baked into the graphistry/nginx image. Renders only when global.tier == "platform" (exact match, not >=); at analytics+ the engine DaemonSet's nginx container plays the same role with intra-pod localhost dispatch to the streamgl/forge backends, so the standalone Deployment would be redundant.

Why it exists: the v1-to-v2 rewrites Graphistry clients depend on (deprecation 410s on /etl, /api/check, /api/encrypt, /api/decrypt, /api/v1/etl/vgraph/*; live forwarding for /api/v1/{datasets,files,organization,team,named-endpoint,...}; parallel /api/v2/etl/... routes) live in the graphistry/nginx image's default.conf.template rendered by render_templates.sh. Pre-fix, tier=platform rendered postgres + nexus only, so a nexus-only deployment had no externally-reachable HTTP surface honouring the v1 paths.

The platform tier now ships a deployable slice of postgres + nexus + nexus-proxy with no transitive dependencies on analytics-tier services. The Deployment's only init container waits for the nexus Service alone (not redis / dask-scheduler / streamgl), so the slice is genuinely self-contained — useful for downstream products that need only the auth foundation (e.g. as the Nexus-only auth backend behind another product's chart).

In-cluster reachability (the actual ask for service-to-service integrations like Louie):

• http://nexus-proxy.<ns>.svc.cluster.local:80 — both v1 and v2 paths
• http://nexus.<ns>.svc.cluster.local:8000 — raw nexus, v2 only

External reachability is deliberately not chart-managed at platform tier (no Ingress, no Caddy). Operators bring their own L7 (Pulumi, Ansible, kustomize, manual Ingress), or use port-forward for dev/test:

kubectl -n <ns> port-forward --address 0.0.0.0 svc/nexus-proxy 8080:80

Reuses existing values and PVCs: NginxResources, global.{nodeSelector,tolerations,imagePullSecrets,restartPolicy}, postgres secret refs, and the local-media-mount + data-mount PVCs. The image's content-directory expectations for /streamgl, /pivot, and /upload paths are satisfied by emptyDir mounts — those routes return 404 at platform tier, which is the correct behaviour when the backends don't exist.

Complementary changes (details in CHANGELOG.md)

• Storage-agnostic chart: PVC templates unified on global.storage.accessMode + global.storageClassNameOverride. Removed ENABLE_CLUSTER_MODE, IS_FOLLOWER, multiNode, clusterVolume, provisioner, REDIS_URL_NEXUS_FEP, longhornDashboard, and the hardcoded datamount-longhorn / postgres-longhorn / retain-sc-cluster StorageClass names. Longhorn becomes just another backend operators can point a SC at.
• Dedicated retain-sc-postgres StorageClass for the postgres-cluster chart, per Crunchy PGO's documented pattern.
• Dask Kubernetes Operator removed: dask-cluster.yml CRD, dask.operator toggle, operator sections from all platform READMEs and Sphinx docs, ArgoCD app, CD subchart, chart bundler entries, ACR import script, dev-compose setup script.
• OTEL / Redis Service hardening: otel-collector Service flipped to ClusterIP + internalTrafficPolicy: Local (DaemonSet collector model, per opentelemetry-operator#1401); Redis Service flipped to ClusterIP (was LoadBalancer, a latent security default).
• charts/values-overrides/examples/cluster/ rewritten end-to-end (~+800 lines): legacy leader/follower multi-namespace example deleted. New single-namespace multi-node guide adds:
  • Architecture diagram of the engine-DaemonSet topology and Caddy sticky-LB ingress
  • cookieSecret rotation procedure (HMAC key for graphistry_sticky)
  • Two-axis scheduling model (nodeSelector for where-allowed vs tolerations for what-taints-accepted) with a worked 5-node A100 walkthrough
  • Cost-optimised variant (mixed GPU/CPU pools using engine.nodeSelector + global.tolerations)
  • Three approaches for pinning CPU singletons (caddy, nexus, redis, postgres) to a small CPU pool
  • End-to-end verification commands (pod placement, sticky-cookie distribution, ETL 5-call)
  • cluster/retain-sc-nfs.yaml for operators who prefer to apply the Retain StorageClass separately
• k3s README updated: new "Postgres StorageClass" / "Graphistry StorageClass" subsections, two-SC install flow, same-namespace invariant note. PV-cleanup runbook reorganized into three explicit options (Graphistry-only / postgres-only / both) so operators don't accidentally wipe the wrong chart's data.
• Chart bug fixes: dcgm-exporter DaemonSet and netshoot / whoami http-tools gained the missing nodeSelector blocks (they were the only workload templates that did not honour global.nodeSelector).

Chart version: graphistry-helm 0.4.3 -> 0.4.4.

Test plan

• 2-node k3s + NFS RWX: helm install succeeds end-to-end
• ETL 5-call sequence (analytics tier) succeeds on both nodes; local-worker affinity exercised and decays to no-op on miss
• Multi-node viz sessions distributed across engine pods via graphistry_sticky cookie; reconnects stay pinned
• Tier transitions (viz -> analytics -> viz) preserve PVC binding
• Caddyfile config edits trigger automatic Pod rollout via checksum/config on helm upgrade
• dcgm-exporter respects global.nodeSelector (prior bug: leaked onto non-GPU nodes)
• helm uninstall + reinstall rebinds PVCs via volumeName workflow, preserves data
• Concurrent multi-GB uploads across nodes, zero backend errors
• Mixed GPU/CPU node-pool test: engine.nodeSelector=graphistry.io/role=gpu + CPU singletons pinned via global.nodeSelector=graphistry.io/role=cpu; verify engine pods land only on GPU nodes and caddy/nexus/redis/dask-scheduler land only on CPU nodes
• global.tolerations validation against NVIDIA GPU Operator taint (nvidia.com/gpu=true:NoSchedule) on a managed cluster (GKE/EKS GPU node-pool)
• Dedicated-tenant taint validation (dedicated=graphistry:NoSchedule) -- chart pods schedule onto tainted nodes only when the toleration is set
• Telemetry stack on multi-node k3s: all GPUs and all nodes visible via per-node labels (post kubernetes_sd_configs switch)
• Platform-tier slice (tier=platform): postgres + nexus + nexus-proxy renders standalone; v1-to-v2 deprecation 410s + authenticated /api/v1/* round-trips return real data; tier transitions to/from analytics are clean
• Caddy stable on tls.mode = external | self | off paths; /caddy/health/ no longer trips the 30s SIGTERM liveness loop
• Production-scale load
• Longhorn RWX integration

Related

• Supersedes: Multi node GPU support (cluster mode) #73 (the prior leader/follower multi-namespace cluster design)
• Fixes: feat(v0.4.3): deployment tiers, dependency alignment, modular chart architecture #79 (deployment tiers, dependency alignment, modular chart architecture)

[albarralnunez]

LGTM