Self-hosted v0.0.9: a failed ingestion discards the document content — large documents (>~10k chars) silently lose their data #1827

Summary

On the self-hosted Docker image supermemory/supermemory:0.0.9, when a document's ingestion fails, the document's content is discarded: GET /v3/documents/{id} returns content: "" and 0 memories, permanently. The document stays in queued/failed and is retried in a loop, so the ingestion queue never drains.

In our store every document above roughly 10–16k characters fails this way. The upload returns HTTP 200s and nothing ever tells you the content is gone — which makes this silent, permanent data loss.

Related, but distinct:

• Re-ingesting unchanged content is a silent no-op — a document that fails extraction can never be repaired through the API #1302 — re-ingesting unchanged content is a silent no-op (there the content survives; here it is gone, so re-ingesting is a change and re-runs — only to fail and wipe it again).
• Self-hosted v0.0.3: encrypted snapshot OOMs (RangeError in node:crypto) past ~150 MB DB, orphaning docs into a permanent retry-cron loop (plus lossy deletion and a runaway memory agent) #1177 — orphaned documents stuck in a permanent retry loop (that one is about the loop; this one is about the content loss inside the loop).
• Self-hosted v0.0.6: Document ingest queue stuck at 'queued' — embeddings never process #1312 — ingest queue stuck at queued.

Environment

• supermemory/supermemory:0.0.9, official prebuilt Docker image, on a Proxmox VM (8 vCPU / 9 GB), store in a Docker volume.
• Embeddings: SUPERMEMORY_EMBEDDING_PROVIDER=openai, SUPERMEMORY_EMBEDDING_BASE_URL=http://172.17.0.1:11434/v1 — a local Ollama on the VM host (nomic-embed-text-v2-moe, 768d). Ollama starts embedding models with a single server slot (-np 1), so requests are serialised.
• LLM for extraction / memory agent: an OpenAI-compatible cloud endpoint.
• SUPERMEMORY_INGEST_CONCURRENCY=8, SUPERMEMORY_EMBEDDING_RAM_LIMIT=2gb.
• Store size: ~430 documents, ~3.6k memories (lite licence).

Measured correlation (same store, 590 exported documents)

21 of the failed documents are exactly 100 013 – 100 015 characters — that is the length at which our client truncates its input, so the client cap is not the limiting factor. The server has its own practical ceiling somewhere between ~8k and ~16k.

Reproduction

1. POST /v3/documents with a ~20 000+ character document (any customId).
2. Poll GET /v3/documents/{id}. After roughly two minutes the status becomes failed and content is "" — permanently.
3. Upload the same content again for the same customId: the document returns to queued, fails again, and the content is discarded again.
4. In the container log, in bursts:

[embeddings] primary failed after <=20000ms, falling back: The operation timed out.

5. supermemory-server doctor reports no problem, and GET /v3/documents/{id} exposes no error field — only an empty content.

Expected

1. A failed ingestion should keep the document's content, together with a failure reason, so it can be inspected and retried. Today the content is destroyed and the API shows nothing.
2. The embedding client timeout should be configurable (or raised + retried with backoff). With a local/CPU embedding endpoint that has a single server slot, a 512-token chunk cannot always be answered within 20 s; when it is not, the whole document dies and its content is lost.
3. It would help to expose the ingestion queue with document ids and the current error, so a stuck document can actually be identified instead of guessed from 8 running · N queued.

Impact

Silent data loss on a self-hosted deployment: documents are accepted and then destroyed. The only recovery path is a logical export (/v3/documents/list + GET per id) taken before the loss, plus splitting the content into smaller pieces and re-uploading. We were only able to recover our backlog because we happened to keep an export.

We worked around it by splitting every document into ≤8 000-character parts and restoring them in small batches — that is the only configuration in which our backlog makes progress at all.

Happy to attach the full export, container logs, and a minimal reproduction script if useful.