Limit updates and Disk IO

[pcace]

Hi there,

i am running this awesome overpass docker on slightly smaller hardware. with enough, but not fast SSDs. The issue i am having is that the constant updates every minute generate a lot of disk IOs.
Is there a way to limit the update cycles? (for example OVERPASS_UPDATE_SLEEP=86400?)
is it possilbe to use a daily update branch for example: OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/day/?

what would be the correct way to do that?

Thanks a lot for help!

[arnesetzer]

Both of them: Yes, but keep in mind, that the amount of updated nodes, ways and relations keeps more or less the same. So I don't know whether this speeds up the process/minimizes the I/O load. But I'm looking forward for your results :)

[pcace]

Thanks for your reply!

for the question regarding OVERPASS_DIFF_URL, i see this in the documentation:

clone - data is copied from an existing server, given by OVERPASS_CLONE_SOURCE, and then updated using OVERPASS_DIFF_URL. This mode is faster to set up, as the OSM planet file is already indexed. The default clone source provides an Overpass instance using minute diffs covering the whole world (hourly or daily diffs will not work with this image).

so is it possible to use the daily updates with a cloned instance or not? i am running the container like this:

services:
  overpass_world:
    image: wiktorn/overpass-api
    container_name: overpass_world
    restart: unless-stopped
    ports:
      - "12345:80"
    volumes:
      - /big/docker/overpass_world:/db
    environment:
      - OVERPASS_MODE=clone
      - OVERPASS_META=yes
      - OVERPASS_DIFF_URL=https://planet.openstreetmap.org/replication/minute/
      - OVERPASS_FASTCGI_PROCESSES=20
      - OVERPASS_RATE_LIMIT=20
      - OVERPASS_UPDATE_SLEEP=3600
      - OVERPASS_SPACE=10737418240
      - OVERPASS_ALLOW_DUPLICATE_QUERIES=yes
    stdin_open: true
    tty: true
    deploy:
      resources:
        limits:
          cpus: '7'
          memory: 6G
        reservations:
          memory: 6G

can i just swap that to a daily diff url?

PS.:
OVERPASS_UPDATE_SLEEP=3600 does help a LOT on a smaller machine! it is quite snappy again. :)

[arnesetzer]

If I understand the code correctly the difference between a cloned instance and a initialised instance is only whether you copy the database or create it from scratch (eg. for extracts). When your database is ready the diff will come from your diff source (OVERPASS_DIFF_URL). If no URL is present the database will remain in the cloned/initialised state.

[pcace]

but why the section in the readme then? i do understand it so that isi not possible to switch this.

[arnesetzer]

This one?

clone - data is copied from an existing server, given by OVERPASS_CLONE_SOURCE, and then updated using OVERPASS_DIFF_URL. This mode is faster to set up, as the OSM planet file is already indexed. The default clone source provides an Overpass instance using minute diffs covering the whole world (hourly or daily diffs will not work with this image).```

[pcace]

yes, i thought in this context it means i cannot change the url

[arnesetzer]

You can change the URL, but I am not so sure how good the container handles that. Given the fact, that a cold reboot from the host machine brings the area dispatcher in a broken state I don't know whether this may messes up the instance as well.