Firestore emulator leaks one thread per WebChannel session, even when the client closes it cleanly

[stevecastaneda]

[REQUIRED] Environment info

firebase-tools: 15.30.0. The Firestore emulator jar is cloud-firestore-emulator-v1.22.0.jar, which 15.30.2 and main still pin.

Platform: macOS 26.6.2 on arm64, OpenJDK 27

[REQUIRED] Test case

The script opens 20 WebChannel sessions on the Listen endpoint and sends TYPE=terminate for each one, the same request the web SDK sends from terminate(). No SDK, no browser, no data.

firebase.json

{ "emulators": { "firestore": { "port": 8787 }, "ui": { "enabled": false }, "singleProjectMode": true } }

repro.sh

#!/bin/sh
PORT=${1:-8080}
PID=$(lsof -nP -iTCP:$PORT -sTCP:LISTEN -t | head -1)
BASE="http://127.0.0.1:$PORT/google.firestore.v1.Firestore/Listen/channel"
parked() { jcmd "$PID" Thread.print | grep -c "ChannelProcessor.run"; }

echo "parked ChannelProcessor threads before: $(parked)"
for i in $(seq 1 20); do
  SID=$(curl -s -X POST "$BASE?VER=8&database=projects%2Fdemo-repro%2Fdatabases%2F(default)&RID=$((1000 + i))&CVER=22&X-HTTP-Session-Id=gsessionid&zx=open$i&t=1" \
    -H 'Content-Type: application/x-www-form-urlencoded' --data 'count=0' |
    sed -n 's/.*\["c","\([^"]*\)".*/\1/p')
  curl -s -o /dev/null -X POST "$BASE?VER=8&SID=$SID&TYPE=terminate&zx=close$i"
done
sleep 90
echo "parked ChannelProcessor threads 90 s after 20 terminated sessions: $(parked)"

[REQUIRED] Steps to reproduce

firebase emulators:exec --only firestore --project demo-repro "./repro.sh 8787"

jcmd ships with the JDK.

[REQUIRED] Expected behavior

A session the client terminated gives its thread back. The parked count returns to 0.

[REQUIRED] Actual behavior

parked ChannelProcessor threads before: 0
parked ChannelProcessor threads 90 s after 20 terminated sessions: 20

Every thread has this stack and never exits.

"Thread-N" ... WAITING (parking)
  at com.google.net.webchannel.server.common.ChannelProcessor.run(ChannelProcessor.java:70)

I disassembled the jar with javap to find out why. ChannelProcessor.run() leaves its loop only when closed is true, and only ChannelProcessor.shutdown() sets it. The one caller of shutdown() is ChannelInternal.shutdown(), which runs when the processor thread is interrupted or consume() throws. ChannelInternalImpl.channelClose() handles the client's terminate. It removes the session from HttpServerHandlerImpl, closes the forward and back channels, sets the status to CLOSED and calls onClose() on the handler. It never calls shutdown(), so the thread waits on hasEvent for the life of the process. The abort path, which sets ABORTED, has the same gap.

Calling shutdown() at the end of channelClose() looks like the whole fix. I tested that from outside the jar. A small Java agent finds parked ChannelProcessor threads whose channelState is CLOSED or ABORTED and calls shutdown() on them through reflection. With the agent loaded, a run of 15 terminated sessions and 5 abandoned ones ends at 5 parked threads, the 5 abandoned ones. Without it, the same run ends at 20.

Why it matters. The web SDK opens a channel per page load, often more than one, so a Playwright suite against the emulator leaks about 3.5 threads per test. Ours grew the emulator to about 6,000 threads and 6.8 GB over eight hours. macOS caps a process at 6,144 threads, the kern.num_taskthreads sysctl. Near the cap the emulator stalls and tests time out, and restarting the emulator is the only way back. With the agent, an 11-test Playwright run that used to leave 34 parked threads leaves 3.

Two related things I saw and did not chase. A session whose client vanishes without a terminate stays OPEN forever, since nothing reaps it, and it keeps its thread too. The 60 second back channel inactivity timeout closes the back channel writer but does not close the session. Those are worth a look by whoever owns the WebChannel server, but the missing shutdown() is the larger leak by far.

I searched for an existing report. #4953 and #4916 describe memory growth during imports, #5197 is about back channel buffering on large queries, #3477 is about batch writes. None covers threads or session close.