Sudden RSS explosion (500 MB → 6 GB in under a minute) with full process freeze, no preceding change in query or failure rate

[rgear14-design]

Sudden RSS explosion (500 MB → 6 GB in under a minute) with full process freeze, no preceding change in query or failure rate

Summary

On a recursive-only resolver serving ~9M queries/day, the process memory jumps
from a stable ~500 MiB to the container's memory limit in under a minute,
pegs one core at ~95% CPU, and stops logging entirely for about 4 minutes.
It then either recovers or is OOM-killed by the kernel.

There is no precursor in the logs: query rate and resolution-failure rate
are completely normal right up to the second it happens.

This has occurred at least 7 times in 12 days, at irregular hours.

Environment

Technitium DNS Server	15.4
Runtime	.NET 10
OS	Debian, inside an unprivileged LXC container on Proxmox VE 6.17
Role	Recursive resolver only — 0 zones, no forwarders configured
Load	~9M queries/day, ~105 qps sustained, 1070 clients
Cache → Maximum Entries	10000 (default)
Block list URLs	2 lists, ~160,000 entries total (urlhaus.abuse.ch hostfile, phishing.army extended)
DNSSEC validation	enabled
Query logging	enabled (file log)

Dashboard for a typical hour: 378,254 total queries, 95.72% NoError,
1.88% ServFail, 2.29% NXDomain, 82.62% cached, 15.58% recursive.

Timeline of the most recent occurrence (all times UTC)

06:00–06:19   normal. 12–25 resolution failures logged per minute,
              steady, no anomalies. RSS ~500 MiB, CPU ~8%.
06:20         3 failures logged, then the minute is cut short.
              RSS still 578 MiB at 06:20:00.
06:21–06:24   NOTHING IS LOGGED AT ALL. Not even background
              cache-prefetch failures, which never normally stop.
              RSS at container limit (6 GiB), CPU ~95%.
06:25         logging resumes (25 entries), service recovers.

Container CPU and memory graphs for the same window show a flat baseline
(~8% CPU, ~512 MiB) with a vertical rise to 95% / 6 GiB and an equally
vertical fall.

Kernel OOM records (from the Proxmox host)

Six kills before the memory limit was raised from 2 GiB to 6 GiB. Note the
RSS at kill time is identical every time — it is the cgroup ceiling, not
the end of a gradual climb:

Aug 28 23:43:29  oom_memcg=/lxc/105 task=dotnet  anon-rss:2050096kB
Aug 29 09:38:31  oom_memcg=/lxc/105 task=dotnet  anon-rss:2050292kB
Aug 31 03:13:16  oom_memcg=/lxc/105 task=dotnet  anon-rss:2043704kB
Sep 01 09:23:21  oom_memcg=/lxc/105 task=dotnet  anon-rss:2049444kB

systemd logs the death as code=killed, status=9/KILL and restarts the unit
10 s later (Restart=always, RestartSec=10).

What has been ruled out

• Cache growth — Maximum Entries is at the default 10000. That cannot
  account for gigabytes.
• Block lists — measured: 379 + 159,377 = ~160k entries.
• A query flood or amplification abuse — top client for the hour had
  6,979 queries and the top domain was graph.facebook.com (8,494), i.e.
  ordinary traffic. Recursion is not reachable from the internet
  (dig @<public IP> times out from outside).
• A resolution-failure storm — failures per minute were flat (12–25)
  through the entire hour, including the minute before the freeze.
• A gradual leak — RSS is flat at ~500 MiB for hours and then rises
  vertically. It does not creep up.

Chronic resolution failures present during (and outside) the window

These fail continuously all day, every day, and are the top failing domains
in the incident window as well — so they are not obviously the trigger, but
they are what the resolver spends its failures on:

160  netflix.com        (many *.internal.dradis.netflix.com names)
 98  al-array.com
 35  nflxso.net
 32  farap2p.ir

Their failures are DnsClientNoResponseException … request timed out
against the delegation's name servers.

What I can provide

A 20-second-interval RSS sampler is now running as a systemd unit, so the
next occurrence will be captured with the shape of the rise (ramp vs. step).
Happy to provide that, the full log for the window, or run any diagnostic
build.

Question

Is there a bound on in-flight recursive resolution state, or anything that
could allocate multiple GB in response to a single query or internal
operation? The complete absence of logging during the freeze suggests the
process is not merely busy but stalled — possibly GC thrashing against the
cgroup ceiling.

[ShreyasZare]

Thanks for the feedback. There is already a bug reported in #2093 which causes the DNS server to use up all memory and crash when a certain domain names are resolved. I guess the issue you see is related to the same bug. This bug has been fixed and will be available in the upcoming update. If you see any different error in the DNS logs related to out of memory then do share the complete log here.

Chronic resolution failures present during (and outside) the window

If the error logs just say request timed out and if you are running the DNS server in recursive resolver mode then such operational issues can occur depending on network conditions. If you have Serve Stale enabled then it handles most of these issues by using stale data from cache so end users do not notice it. These timeout errors are something that is related to the network conditions and there is no fix to such issues.