Improve Docker startup scripts with retry logic and health checks

[anchapin]

Problem

Docker startup scripts use wait-for-it --strict -t 0 which blocks forever on TCP port checks. No Docker health checks exist. No application-level MongoDB/Redis readiness verification. This means:

1. If MongoDB takes a while to initialize (beyond TCP accept), the app starts but can't connect
2. If a service crashes and restarts, dependent containers are already past their wait and don't retry
3. No way to know if the overall stack is actually healthy from Docker's perspective

Current State

Startup scripts (all identical pattern):

wait-for-it --strict -t 0 db:27017    # Blocks forever on TCP
wait-for-it --strict -t 0 queue:6379   # Blocks forever on TCP

docker-compose.yml:

• depends_on ensures start order but not readiness
• No healthcheck sections defined
• No condition: service_healthy on dependencies

What's missing:

• Finite timeout with retry (instead of infinite block)
• Application-level readiness check (can MongoDB actually accept queries?)
• Docker health checks for monitoring
• Backoff between retries

Proposed Fix

1. Add retry logic to startup scripts

Replace infinite wait with bounded retry:

wait-for-it --strict -t 30 db:27017 || {
  echo "MongoDB not ready after 30s, retrying..."
  for i in 1 2 3; do
    sleep 5
    wait-for-it --strict -t 30 db:27017 && break
  done
}

2. Add application-level MongoDB readiness check

After TCP check passes, verify MongoDB can accept connections:

# After wait-for-it succeeds
mongosh --eval "db.runCommand({ ping: 1 })" --quiet db:27017 || exit 1

(Note: need to verify mongosh or mongo is available in the container)

3. Add Docker health checks

In docker-compose.yml:

db:
  healthcheck:
    test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 30s

queue:
  healthcheck:
    test: ["CMD", "redis-cli", "ping"]
    interval: 10s
    timeout: 5s
    retries: 5

4. Use health check conditions on depends_on

web:
  depends_on:
    db:
      condition: service_healthy
    queue:
      condition: service_healthy

5. Add startup timeout

Set a maximum startup time so containers don't block forever:

wait-for-it --strict -t 60 db:27017 || { echo "FATAL: MongoDB not ready"; exit 1; }

Files to Modify

• docker/server/start-server.sh
• docker/server/start-web-background.sh
• docker/server/start-workers.sh
• docker/server/start-server-dev.sh
• docker-compose.yml (and variants: test, deploy, local, aws)
• docker/server/rails-entrypoint.sh (optional: add app-level check)

Testing

• Test with docker-compose up from clean state
• Test with delayed MongoDB start (add sleep 10 to mongo entrypoint)
• Verify health checks report correctly via docker-compose ps