filter: migrate to ZSTD_compressStream2 (fix flush-promotion stall, worker spin, 131072 truncation)

[hsw]

filter: migrate to ZSTD_compressStream2 (fix flush-promotion stall, worker spin, 131072 truncation)

Summary

Migrate the compression filter from the legacy three-call streaming API
(ZSTD_compressStream / ZSTD_flushStream / ZSTD_endStream driven by an
action state machine) to libzstd's unified
ZSTD_compressStream2(cctx, &out, &in, op), modelled on ngx_brotli's
per-call-op pattern.

This fixes three long-standing bugs at the root:

• bugfix: fix zstd module infinite loop when upstream return content-length abnormal #23 — infinite loop / worker at 100% CPU when the upstream returns an
  abnormal Content-Length.
• endless loop  #25 — silent body truncation at exactly 131072 bytes for inputs larger
  than ZSTD_CStreamInSize() (the bug filter: fix silent truncation at 131072 bytes for inputs >ZSTD_CStreamInSize #49 also targets).
• The proxied chunked / SSE / Upgrade flush-promotion stall (small flush'd
  chunks delivered late, only when the upstream connection closes).

Root cause

The body filter only promotes COMPRESS -> FLUSH when libzstd reports pending
output:

if (rc > 0) {
    if (ctx->action == NGX_HTTP_ZSTD_FILTER_COMPRESS) {
        ctx->action = NGX_HTTP_ZSTD_FILTER_FLUSH;
    }
    ...

Two failure modes fall out of this:

1. Stall / no flush — when a small flush'd chunk is swallowed by libzstd
   without producing output (rc == 0), no ZSTD_flushStream() is ever
   issued. The bytes sit in libzstd's internal buffer and the response stalls
   until the upstream connection closes. Reported on chunked / SSE / Upgrade
   responses.

2. Infinite loop (bugfix: fix zstd module infinite loop when upstream return content-length abnormal #23) — when add_data hands the encoder a zero-length
   input buffer (NULL chain on a not-yet-finalized request) and
   ZSTD_compressStream returns 0, the loop makes no progress and never
   breaks — the worker spins at 100% CPU.

The 131072 truncation (#25) is the same family: the per-call bookkeeping around
the input buffer drops the remainder of an oversized buffer instead of draining
it.

The fix

Replace the action state machine with the per-call-op pattern from
ngx_brotli:

1. Per-call op. Sticky ctx->last / ctx->flush flags are set in
   add_data from the input buf's last_buf / flush, and the
   ZSTD_EndDirective is derived per call (last -> ZSTD_e_end,
   flush -> ZSTD_e_flush, else ZSTD_e_continue). A finish op is complete
   when libzstd returns rc == 0. When a flush/end boundary carries no pending
   bytes, a separate zero-size sync buf is emitted so the boundary is visible
   to the write filter (a recycled temporary out_buf would be dropped by
   ngx_buf_special).

2. Zero-progress guard. In ZSTD_e_continue mode the encoder must make
   forward progress; if a call consumes no input and emits no output the
   request is aborted instead of spinning the loop. This closes bugfix: fix zstd module infinite loop when upstream return content-length abnormal #23.

3. NULL+0 UB guard. in_buf->pos / out_buf->last are advanced only when
   the delta is non-zero — a flush-only call can leave buffer_in.src NULL,
   and NULL + 0 is undefined behaviour (UBSan-reported) even though the
   result is unused.

4. Modernized CCtx setup. ZSTD_CCtx_reset(session_only) followed by
   ZSTD_CCtx_refCDict (dict case) or
   ZSTD_CCtx_setParameter(ZSTD_c_compressionLevel), dropping the legacy
   ZSTD_initCStream / ZSTD_initCStream_usingCDict path and the >= 1.5.0
   version gate around the deprecated usingCDict call.

Split into two commits — a mechanical ZSTD_compressStream2 swap, then the
state-machine retirement — so each step is reviewable and bisectable on its own.

Compatibility note

ZSTD_compressStream2 has been the stable streaming API since libzstd 1.4.0
(Apr 2019), so this raises the minimum to 1.4.0 via a compile-time #error.
Every currently supported distro ships >= 1.4.4 (Ubuntu 20.04 is 1.4.4,
22.04 is 1.4.8). The static .zst module is unaffected — it does not include
zstd.h.

Testing

Verified against nginx.org-mainline builds on Ubuntu 22.04 / 24.04 / 26.04
(libzstd 1.4.8 / 1.5.5 / 1.5.7):

• Worker spin / no-hang (bugfix: fix zstd module infinite loop when upstream return content-length abnormal #23): fails on the pre-migration baseline (worker
  burns ~2000 ms CPU over a 2 s idle window after a short read); passes after
  the change.
• Proxy flush-promotion (chunked / chunked-off-tiny / SSE / Upgrade, HTTP/1.1
  and HTTP/2): delivered within the TTFB budget after the change.
• 131072 truncation (endless loop  #25): the >128 KB roundtrip cases
  (131073 / 200000, and the unbuffered-proxy 131071 / 131073 / 200000) fail on
  the baseline and pass after the change.
• Full compress / roundtrip / dict / HTTP/2-multiplex / WebSocket suite green.
• ASan + UBSan (-DNGX_DEBUG_PALLOC=1): no diagnostics. Five-tool SAST sweep
  (scan-build / clang-tidy / cppcheck / gcc-fanalyzer / flawfinder): no new
  findings versus the unmodified baseline.

This has been running in production in a downstream fork; the fork carries a
broader pytest + Docker regression matrix (multi-distro, ASan/UBSan, Valgrind,
SAST) that exercises the change end to end.

Relationship to existing PRs

• Supersedes the bugfix: fix zstd module infinite loop when upstream return content-length abnormal #23 approach by fixing the loop at the API level rather than
  special-casing the abnormal-Content-Length input.
• Also resolves endless loop  #25 (the silent 131072 truncation), which filter: fix silent truncation at 131072 bytes for inputs >ZSTD_CStreamInSize #49 targets with a
  narrower drain-loop patch on the legacy API.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Updates the NGINX zstd body filter to use modern libzstd streaming APIs and fixes flush/end propagation when zstd produces no output for small inputs.

Changes:

• Require libzstd >= 1.4.0 and migrate streaming compression to ZSTD_compressStream2().
• Replace the previous action/redo state machine with sticky flush/last flags and unified post-compress handling (including signal-only buffers).
• Modernize cstream initialization using ZSTD_CCtx_reset() + ZSTD_CCtx_refCDict() / ZSTD_CCtx_setParameter().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[eilandert]

hi @hsw

I would like to invite you to https://github.com/eilandert/zstd-nginx-module to test the module and fix it more up. i fixed 36 bugs and did multiple optimizations, new directives and a load of new testing, (and fixed these bugs). I feel the only real thing i am missing is a second review and production testing before i can submit back.

I have emailed the original author but he doesn't respond.

[hsw]

Thanks, I already have my own fork with bug fixes and better memory management (auto window).