cbor: install allocation cap to mitigate oversized cbor_load() alloc

[nabbi]

Summary

cbor_load() (libcbor) pre-allocates storage for definite-length CBOR
arrays/maps sized by the element count declared in the message header,
before reading any element data. The only check on that count is an
integer-overflow check — there's no bound relative to the size of the
input itself. A handful of bytes can therefore make cbor_load() attempt
an allocation of an arbitrary size.

In libfido2, cbor_load() is reachable from a FIDO authenticator's CTAP
responses (src/assert.c, src/cbor.c, src/cred.c, src/credman.c,
src/largeblob.c, src/winhello.c). A malicious or malfunctioning
authenticator can therefore use a tiny CBOR message to make a host
process attempt a multi-gigabyte (or larger) allocation. The existing
FIDO_MAXMSG/FIDO_MAXMSG_CRED limits don't help here — they bound the
number of bytes read, not the element count declared inside those
bytes.

This was reported upstream as
PJK/libcbor#418 (a 5-byte
input triggering a ~2.3GB allocation, a 9-byte input triggering a ~128GB
allocation) and closed won't-fix — libcbor considers this the
consumer's responsibility.
PJK/libcbor#422 (merged in
0.14.0) documents the recommended mitigation: install a capping allocator
via cbor_set_allocs() (see examples/capped_alloc.c in libcbor).

Fix

fido_init() now calls fido_init_cbor_allocs() (new file
src/cbor_alloc.c), which installs a capping allocator via
cbor_set_allocs(). It tracks the total size of live libcbor allocations
and rejects any allocation that would push the running total above
FIDO_CBOR_MAX_ALLOC (src/fido/param.h, 64MB by default, overridable
at build time). Once the budget is exhausted, cbor_load() returns NULL
with result.error.code == CBOR_ERR_MEMERROR, which every cbor_load()
caller in libfido2 already treats as an ordinary decode failure — no
caller-side changes needed.

64MB is well above FIDO_MAXMSG (2048B), FIDO_MAXMSG_CRED (4096B), and
any realistic largeBlob array, but far below a level that could exhaust
process memory or trigger the OOM killer.

Since cbor_set_allocs() configures a single, process-wide set of
function pointers, and fido_init() is documented (fido_init(3)) to be
called once per thread, the allocation counter is TLS — a per-thread
budget, per the alternative suggested in PJK/libcbor#422.

Not related to fuzz/README's libcbor patch

fuzz/README documents a separate, fuzzing-only libcbor patch that caps
_cbor_alloc_multiple at 1000 items, to bound ASAN/MSAN/UBSAN memory
during fuzzing. That patch is far too restrictive for production (it'd
reject legitimate messages with >1000 elements) and only applies to the
local libcbor build used by the fuzz harness. This PR's mitigation ships
in libfido2 itself, applies to every build, and uses a memory budget
rather than an element-count limit so it doesn't reject legitimate large
messages.

That fuzz/README patch was first added in 2018 (~7 years ago, relative
to libcbor 0.10.1) — the underlying allocation-sizing issue it works
around upstream has been around at least that long, and PJK/libcbor#418
is simply the most recent (2024) report of it.

Changes

• src/cbor_alloc.c (new): capping allocator, installed via
  cbor_set_allocs()
• src/dev.c: fido_init() calls fido_init_cbor_allocs()
• src/fido/param.h: new FIDO_CBOR_MAX_ALLOC (64MB default)
• src/extern.h, src/CMakeLists.txt: wire up the new TU
• regress/dev.c: new cbor_alloc_cap() regression test — feeds
  cbor_load() a 9-byte array header declaring 268M elements (would
  otherwise provoke a ~2GB allocation) and asserts it's rejected with
  CBOR_ERR_MEMERROR
• SECURITY-CONSIDERATIONS.md (new), README.adoc, man/fido_init.3:
  document the threat model and mitigation

Test plan

• New regression test cbor_alloc_cap() in regress/dev.c passes
• Existing regress/dev suite still passes (capping allocator doesn't
  affect normal-sized CBOR messages)

References

• [Security] Uncontrolled memory allocation in cbor_load() allows DoS via small crafted input PJK/libcbor#418 — oversized cbor_load() allocations from tiny inputs
  (won't-fix upstream)
• Document large-allocation risk in cbor_load and clarify test-only CMake flags PJK/libcbor#422 — cbor_set_allocs() capping-allocator mitigation
  (examples/capped_alloc.c)

[martelletto]

Is there something wrong with setrlimit(RLIMIT_DATA, ...)?

[nabbi]

setrlimit

Following upstream libcbor guidance of cbor_set_allocs.
pam_u2f fuzzer tests fail against a non-hacky-CI patched instance of libfido2

[dacav]

Hello,

Thanks for your contribution, but I'd vote against it.

libfido2 is a library, and the capping of the allocated memory via cbor_set_allocs would have side effects on the global application state.

libfido2 already does what it is supposed to do: properly handle the error condition when malloc(3) returns NULL. It is up to the application that uses libfido2 to define global settings such as the one you propose.

Additionally, the CBOR object is read from the authenticator, which is supposed to be trusted.