A production-grade, air-gapped semantic search and document intelligence system. Drop PDFs, videos, or audio files into a folder and chat with them using an AI that cites its sources — all running on your own hardware with zero internet dependency.
I built this system to solve a real problem: how do you make thousands of documents — PDFs, meeting recordings, video training, scanned archives — semantically searchable and queryable by natural language, without sending any data to a third party?
The answer is a distributed, multi-worker ingestion pipeline that runs entirely on dedicated LAN hardware. Every component — from the OCR fallback chain to the HAProxy load balancers to the DuckDB-backed state machine — is explicitly implemented for production-grade semantic search, not as a demo.
This is production semantic search infrastructure, not a notebook. It runs continuously, recovers from failures, load-balances across multiple GPUs, and has been tested against real-world document volumes.
| Format | How it's processed |
|---|---|
.pdf (scanned) |
OCR via Docling/EasyOCR |
.pdf (text layer) |
Direct extraction via pdfplumber |
.mp4, .mov, .mkv |
WhisperX speech transcription |
.mp3, .wav, .m4a, .aac, .flac |
WhisperX audio transcription |
.txt, .md, .html |
Direct read with charset detection |
What you get out: A semantic search chat interface where every answer sentence is traced back to its source with clickable citations linking to the original file and page.
Everything runs locally on dedicated LAN hosts — no cloud services, no external API calls, no data leaves your network.
staging/ → Gatekeeper (extract + normalize to Markdown) → Producer (chunk + enqueue)
→ Consumer (embed + store in Qdrant)
→ success/
Six workers coordinate via Redis queues and a DuckDB-backed state machine with atomic UPDATE ... RETURNING * transitions. Each worker runs in its own Docker container with restart: unless-stopped for self-healing. The result is a semantic search pipeline that ingests, indexes, and retrieves documents by meaning, not just keyword matches.
| Decision | Rationale |
|---|---|
| Dual-LLM architecture | Separate models for normalization (gatekeeper) and chat (RAG). Each optimized for its task — smaller context for normalization, larger for chat. |
| Markdown-first normalization | Raw text → clean Markdown → chunk. Eliminates OCR noise, page numbers, and footers before they reach the vector DB. |
| Zero-drop chunking | Hierarchical header splitting with recursive sub-splitting. No content is ever truncated — oversized chunks are split, not dropped. |
| DuckDB staging | Chunks are persisted to DuckDB before embedding. A file_end sentinel triggers atomic batch upsert to Qdrant — zero partial-visibility for RAG. |
| HAProxy load balancing | Auto-detects multiple LLM/embedding/Whisper/OCR backends and distributes requests with health checks and failover. |
| Temporal durability (optional) | When enabled, WhisperX transcription runs as Temporal Activities — crash-recovery, automatic retries, and workflow observability via Web UI. |
| Dual vector DB support | Qdrant (gRPC + REST) and Chroma both supported via VECTOR_DB_PROFILE. |
| Deterministic chunk IDs | MurmurHash3-based IDs prevent duplicate vectors on re-ingestion. |
| 20-retry exponential backoff | DuckDB lock contention is handled gracefully under high concurrency. |
| Session-managed chat history | Chat history stored in Redis per session ID, not passed client-side. Enables stateless API load balancing. |
Backend: Python 3.12, FastAPI, llama-cpp-python, LangChain (vector store wrappers only), Redis, DuckDB, Qdrant/Chroma, WhisperX, Docling, HAProxy, Temporal (optional)
Frontend: Astro v6, Tailwind CSS v4, daisyUI 5.5
Infrastructure: Docker Compose with profile support (cuda/qdrant/chroma/temporal), multi-architecture worker images
- Docker v2.20+
- Node.js v22.12.0+ (frontend only)
All services connect over HTTP. Run them on any hosts in your LAN and set these variables before starting the stack.
# --- Filesystem ---
# Root directory for the local ingestion pipeline. Lifecycle subdirectories
# (staging/, preprocessing/, ingestion/, consuming/, success/) are created here.
export DEFAULT_DOC_INGEST_ROOT=/path/to/docs
# --- Vector Database ---
# Qdrant (default) or Chroma.
export VECTOR_DB_PROFILE=qdrant
# Remote Qdrant host. The system connects over gRPC on port 6334 by default.
# To use the REST API on port 6333 instead, set VECTOR_DB_USE_GRPC=false and prefix the URL with http://.
export VECTOR_DB_URL=http://<vector-db-host>:6334
export VECTOR_DB_USE_GRPC=true
# --- LLMs ---
# Chat model that answers RAG queries. Remote http(s):// URL or local .gguf path.
export LLM_PATH=http://<llm-host>:11435/v1/chat/completions
# Gatekeeper model that normalizes raw extracted text to Markdown during ingestion.
# Often runs on the same host as LLM_PATH but typically uses a different port (11534).
export SUPERVISOR_LLM_ENDPOINTS=http://<llm-host>:11534/v1/chat/completions
# --- Embedding ---
# Model that vectorizes document chunks during ingestion and user queries during chat.
# Remote http(s):// URL or local model directory path.
export EMBEDDING_ENDPOINTS=http://<embedding-host>:11434/v1/embeddings
# --- Audio / Video Transcription ---
# WhisperX host. Transcribes audio files (MP3, WAV, etc.) and extracts speech
# from video files (MP4, MOV, MKV) during ingestion.
export WHISPER_MODEL_ENDPOINTS=http://<whisper-host>:1145/inference
# --- OCR Fallback ---
# docling-serve host. Used when pdfplumber cannot extract text from a page
# (scanned documents, image-heavy pages). Set to "LOCAL" to run Docling
# inside the container instead.
export OCR_ENDPOINTS=http://<ocr-host>:5001/v1/convert/file
# --- Tokenizer (Required) ---
# Path to tokenizer model for chunk size calculations (e.g., mxbai-embed-large-v1).
export TOKENIZER_MODEL_PATH=/path/to/tokenizer-model
# --- Optional Tuning Flags (all false by default) ---
export PDF_FORCE_OCR=false
export HA_INTERLEAVE=false
export FORCE_MARKDOWN_LLM=false
export EMBEDDING_BATCH_SIZE=25
# --- Temporal (Optional - Durable WhisperX Transcription) ---
# Enable Temporal-based WhisperX transcription for crash-recovery and retries.
# Requires a remote Temporal server (not provided by docker-compose).
export USE_TEMPORAL_WHISPER=false
export TEMPORAL_HOST=<temporal-host>
export TEMPORAL_PORT=7233
export TEMPORAL_WHISPER_TASK_QUEUE=whisperx
# --- Redis (Required) ---
# Redis server for queue coordination and chat session storage.
export REDIS_HOST=<redis-host>
export REDIS_PORT=6380Full variable reference, local-file alternatives, and optional tuning flags are in docs/quickstart.md.
When you have multiple hosts running the same service, set *_ENDPOINTS env vars with comma-separated URLs. HAProxy starts automatically and distributes requests across all backends with health checks and failover.
export SUPERVISOR_LLM_ENDPOINTS=http://gpu0:11534/v1/chat/completions,http://gpu1:11534/v1/chat/completions
export EMBEDDING_ENDPOINTS=http://gpu0:11434/v1/embeddings,http://gpu1:11434/v1/embeddings./doc-ingest-chat/run-compose.sh --buildTo enable durable transcription via Temporal (crash-recovery for audio/video jobs), deploy a remote Temporal server and set:
USE_TEMPORAL_WHISPER=true TEMPORAL_HOST=<temporal-host> ./run-compose.sh --buildDrop files into $DEFAULT_DOC_INGEST_ROOT/staging/. The system auto-detects and processes them. Open http://localhost:4321 to chat with your documents.
| Quick Start | Full environment configuration, service setup, and deployment diagram | |
| Architecture | System flow, component map, state machine, and Redis queue architecture | |
| Deep Dive | Design rationale, dual-LLM architecture, chunking strategy, production roadmap | |
| Operations | Setup checklist and Day 1 / Day 2 operational playbooks with symptom-driven runbooks | |
| Edge Agent | Deploy MQTT SRE agents on standalone Debian minipcs for telemetry and task execution | |
| Changelog | Version history and feature tracking |
Three reasons:
- Data sovereignty: Legal, medical, and internal documents cannot be sent to third-party APIs. This system processes everything on your own hardware.
- Offline resilience: No dependency on internet connectivity. Works in air-gapped environments, remote sites, or during outages.
- Cost predictability: No per-token API costs. Once the hardware is in place, the marginal cost per document is effectively zero.
The system was designed from day one for air-gapped deployment — HuggingFace offline mode is baked into every container, all model paths support local files or LAN HTTP endpoints, and there are no hardcoded external service dependencies.
I'm a software engineer who builds semantic search and AI infrastructure that bridges the gap between research and production deployment. This project represents my approach to engineering: understand the problem deeply, build for reliability first, and make every architectural decision explicit and auditable.
If this kind of work interests you, let's talk. The best way to reach me is through GitHub.