Bug: NullReferenceException in TryClusterPublishAsync after gossip timeout breaks cluster pub/sub

[sanamhub]

Describe the bug

In a 2-node Garnet cluster (v1.1.10) deployed across two regions with ~150ms RTT between nodes, a single gossip round timeout causes the cluster pub/sub forwarding to crash permanently. The error is a NullReferenceException at GarnetClientExtensions.ClusterPublishNoResponse, thrown because the gossip GarnetClient has been disposed but the code does not null-check it before use.

After the first NRE, every subsequent PUBLISH on either node triggers the same exception. Cluster pub/sub forwarding is completely dead from that point forward. The Garnet process remains running but becomes progressively more unstable over the next hour, eventually requiring a restart.

This makes Garnet v1.1.10 cluster mode unreliable for any deployment where nodes are connected over a WAN link, since gossip timeouts are expected to occur periodically on high-latency paths.

Steps to reproduce the bug

1. Deploy Garnet v1.1.10 in cluster mode with 2 nodes (1 primary, 1 replica) across two regions. Configure with --gossip-delay 30 --cluster-timeout 15 --cluster-announce-ip <public-ip>.

2. Wait for a gossip round to time out. On a WAN link this happens naturally due to latency variance or any brief network blip. The following appears in stdout:

warn: ClusterManager[0] GOSSIP round faulted
warn: ClusterManager[0] GOSSIP to remote node [<node-id> <remote-ip>:6379] timeout!

3. After the gossip timeout, publish any message that requires cluster forwarding. The null reference crash appears:

warn: ClusterManager[0] ClusterManager.TryClusterPublishAsync
System.NullReferenceException: Object reference not set to an instance of an object.
   at Garnet.cluster.GarnetClientExtensions.ClusterPublishNoResponse(GarnetClient client, ...)
   at Garnet.cluster.GarnetServerNode.TryClusterPublish(...)
   at Garnet.cluster.ClusterManager.TryClusterPublishAsync(...)

4. Every subsequent PUBLISH triggers the same NullReferenceException. The cluster pub/sub forwarding never recovers without a full process restart.

5. After approximately 1 hour in this degraded state, additional cascading failures appear: ObjectDisposedException on the socket, GarnetClientDisposedException on AUTH reconnect attempts, and eventually the process becomes unresponsive to client commands (even DBSIZE times out).

Expected behavior

A transient gossip timeout should not permanently dispose the gossip client. TryClusterPublishAsync should handle a null or disposed gossip client gracefully, either by skipping cluster forwarding for that round (degraded but not crashing) or by re-creating the client. A single gossip failure should not cascade into complete cluster pub/sub failure followed by process unresponsiveness.

Screenshots

No response

Release version

v1.1.10 (latest stable)

IDE

N/A - issue is in the Garnet server process, not related to any IDE.

OS version

Windows Server 2022 (both nodes)

Additional context

Cluster configuration used:

--port 6379
--bind 0.0.0.0
--memory 8g
--aof
--aof-commit-freq 1000
--aof-size-limit 512m
--compaction-freq 3600
--pubsub-pagesize 1m
--cluster
--cluster-timeout 15
--gossip-delay 30
--cluster-replication-reestablishment-timeout 30
--cluster-announce-ip <public-ip>

Environment details:

• 2-node cluster (1 primary, 1 replica), cross-region deployment
• RTT between nodes: approximately 150ms
• Garnet runs as a Windows service via NSSM
• StackExchange.Redis 2.12.14 client on .NET Framework 4.8
• Approximately 1 million keys in the cache at time of failure

Stack traces from production:

The NullReferenceException occurs at:

// /_/libs/cluster/Server/Gossip/GarnetClientExtensions.cs:line 69
ClusterPublishNoResponse(GarnetClient client, RespCommand cmd, Span<byte> channel, Span<byte> message, CancellationToken cancellationToken)

// /_/libs/cluster/Server/Gossip/GarnetServerNode.cs:line 295
TryClusterPublish(RespCommand cmd, Span<byte> channel, Span<byte> message)

// /_/libs/cluster/Server/Gossip/Gossip.cs:line 266
TryClusterPublishAsync(RespCommand cmd, Span<byte> channel, Span<byte> message)

The gossip timeout originates from:

// /_/libs/cluster/Server/Gossip/GarnetServerNode.cs:line 182
GossipAsync(byte[] configByteArray)

After the NRE cascade, the process eventually reaches this state (from production logs):

fail: ClusterManager[0] AUTH returned error
Garnet.client.GarnetClientDisposedException: Cannot access a disposed object.
Object name: 'GarnetClient'.
   at Garnet.client.GarnetClient.ConnectAsync(...)
   at Garnet.client.GarnetClient.ReconnectAsync(...)
   at Garnet.client.GarnetClient.ExecuteForMemoryResultWithCancellationAsync(...)
   at Garnet.client.GarnetClient.ReconnectAsync(...)
   ... (30+ nested reconnect attempts in the stack)

Why this matters for production use:

Cluster pub/sub is the mechanism Garnet uses to forward publishes between nodes so subscribers on any node receive messages published on any other node. This is essential for cache invalidation in multi-region deployments. When it breaks, cache entries become stale indefinitely (no TTL-independent invalidation across regions).

We chose Garnet v1.1.10 because it is the latest stable release. The v2.0.x series has been in beta/preview since December 2025 and is not yet suitable for production. We would prefer to stay on the stable release line if this bug can be patched.

Related: fixes on main not backported to v1.1.x

There are at least 3 fixes merged to main that address production stability issues but were never backported to the v1.1.x stable line. This matters because v1.1.10 is the end of the v1.1.x series (no v1.1.11 exists as of July 2026):

PR	Fix	Status
#1753	"Entry does not fit on page" transient X-lock leak (fixes #1749). PostUpsert AOF entry exceeding AofPageSize throws TsavoriteException which unwinds before TransientXUnlock runs, leaving hash bucket locked forever at 100% CPU.	Merged to main. Not backported to v1.1.x.
#1838	Fix AOF page/memory size defaults and validation (fixes #1811, #1770). Bumps v1 defaults: AofPageSize 4m-64m, AofMemorySize 64m-128m. Adds startup validation preventing AofPageSize < 2 * main-log PageSize.	Merged to main. A backport PR #1844 targeting release/v1 was closed as draft, never merged. The fix remains main-only.
#1305	Fix replica checkpointing divergence with CommitAOF calls. Adds ClusterReplicationReestablishmentTimeout setting for auto-resync after restart.	Merged to main. We use the --cluster-replication-reestablishment-timeout flag it introduced, but the checkpointing fix itself is main-only.

The comparison between v1.1.10 and main shows 194 commits across 1,192 changed files. The Garnet team appears to have shifted all development to the v2.0.x beta series with no patch releases planned for the v1.1.x stable line. If this bug (NullReferenceException in TryClusterPublishAsync) exists in v1.1.10 and the fix is straightforward (a null check on the gossip client), a v1.1.11 patch release would allow production users to remain on the stable line until v2.0.x is production-ready.