Logs from VahaC

self-hosting-overpass-api-us-only-on-docker-a-journey-with-bumps

, ,

Self‑hosting Overpass API (US‑only) on Docker: a Journey with Bumps

Public Overpass API instances are great… until they’re not. Rate limits, timeouts, and the constant fear of being that person who DDoS’иs the public endpoint with one bad query. If you need reliable, high‑volume OSM queries for your own project, self‑hosting Overpass API is the way to go.

In my case, I needed an Overpass API instance that:

  • Runs in Docker 🐳
  • Lives on my own server (OpenMediaVault)
  • Is loaded only with US OSM data 🇺🇸

Sounds simple, right? In practice it turned into a mini‑adventure with:

  • CPU/architecture mismatches (arm64 vs amd64)
  • PBF vs .osm.bz2 confusion
  • 10.7 GB downloads and 12‑hour imports
  • File permissions, sockets, and Permission denied everywhere 🙃

This post walks through the final working setup and also the real‑world pitfalls I hit along the way, so you (hopefully) won’t repeat them.

1. The basic goal 🧭

Target architecture:

  • Data: only the US extract from Geofabrik (us-latest.osm.pbf)
  • Database: Overpass internal DB built from that extract
  • Runtime: Docker container (wiktorn/overpass-api), exposed on a port, ready for Overpass queries

High‑level phases:

  1. Initialize Overpass DB from us-latest.osm.pbf (one‑time, very heavy).
  2. Reuse that DB in a lightweight container on the actual server.
  3. Expose it via HTTP and point your services to it.

Because the initial import is so heavy, it’s totally reasonable to build the DB on one machine (e.g. a strong Windows PC with Docker Desktop) and then copy the DB to a Linux server.

2. Picking the image & understanding the data format 🧱

The image used:

overpass:
  image: wiktorn/overpass-api

Key detail: Overpass internally expects compressed XML (“), not PBF. But Geofabrik now provides PBF (*.osm.pbf) for most modern extracts, including the US.

So the pipeline is:

  1. Download us-latest.osm.pbf via HTTP.
  2. Convert PBF → compressed XML .osm.bz2 with osmium.
  3. Let Overpass import that .osm.bz2 into its DB.

That’s why you’ll see environment variables like:

  • OVERPASS_PLANET_URL=https://download.geofabrik.de/north-america/us-latest.osm.pbf
  • OVERPASS_COMPRESSION=gz
  • OVERPASS_PLANET_PREPROCESS=... osmium cat ...

The preprocess script is what converts the downloaded “fake .bz2” into the actual .osm.bz2 Overpass wants.

3. A critical note about architecture when copying Overpass DB 🧬

If you build the Overpass database on one machine and then move it to another (like I did), both machines must use the same CPU architecture — e.g. amd64 → amd64.

Overpass stores binary index files in /db/db/ that depend on architecture. Mixing architectures (e.g. building on ARM and running on x86_64) will cause the runtime container to crash or fail to read the DB.

In my case, I simply launched the init container on the wrong server once — no big story there — but the important rule is:

✅ Build and run Overpass DB on the same architecture. If you plan to export the DB to another machine, make sure both systems are amd64.

With that clarified, let’s move on.

4. Composing the init container ⚙️

For the initial import (one‑time, heavy operation), the docker-compose.yml looked roughly like this:

services:
  overpass:
    image: wiktorn/overpass-api
    container_name: overpass-us
    restart: unless-stopped

    ports:
      - "8084:80"

    environment:
      - OVERPASS_MODE=init
      - OVERPASS_META=yes
      - OVERPASS_PLANET_URL=https://download.geofabrik.de/north-america/us-latest.osm.pbf
      - OVERPASS_COMPRESSION=gz
      - OVERPASS_PLANET_PREPROCESS=mv /db/planet.osm.bz2 /db/planet.osm.pbf && \
        osmium cat -o /db/planet.osm.bz2 /db/planet.osm.pbf && \
        rm /db/planet.osm.pbf

    volumes:
      - /path/to/overpass-db:/db

What happens here:

  1. Container downloads ~10.7 GB us-latest.osm.pbf to /db/planet.osm.bz2 (yes, mis‑named on purpose).
  2. OVERPASS_PLANET_PREPROCESS runs:
    • mv /db/planet.osm.bz2 /db/planet.osm.pbf – we admit it’s actually PBF.
    • osmium cat -o /db/planet.osm.bz2 /db/planet.osm.pbf – convert PBF → compressed XML.
    • rm /db/planet.osm.pbf – clean up.
  3. Overpass then reads planet.osm.bz2 and builds the internal DB in /db/db.

This phase took hours. On Windows 11 with Docker Desktop it was >12h. On a small Linux server it was less happy (segfaults, OOM‑like behaviour), so doing the heavy lifting on a powerful workstation turned out to be the better move.

When it’s done successfully, logs end with something like:

Reorganizing the database ... done.
Overpass container initialization complete. Exiting.

That’s your green light ✅

5. When init keeps failing: broken pipes, segfaults, and flaky downloads 💣

OVERPASS_MODE=init looks simple on paper — run once, wait long enough, get a ready database. In reality, with a huge US extract, this step can be extremely unstable.

On my smaller Linux server, the initialization failed multiple times, often very late in the process, with errors like:

Reorganizing the database ... done.
Reading XML file ... elapsed node 1118025330. /app/bin/init_osm3s.sh: line 44: 55109 Broken pipe             bunzip2 < $PLANET_FILE
     55110 Segmentation fault      (core dumped) | $EXEC_DIR/bin/update_database --db-dir=$DB_DIR/ $META $COMPRESSION
Failed to process planet file

Sometimes it wasn’t even Overpass’ fault — the remote server hosting the US dataset would stop responding halfway through the 10.7 GB download, causing curl to fail and the whole run to restart.

Because of this, I launched the init process both on the server and on my Windows workstation at the same time. If one of them failed, I simply restarted only that instance, while the other continued running normally.

This pattern repeated several times:

  • server run fails → restart server instance,
  • Windows run fails → restart Windows instance,
  • remote source stops responding → whichever machine was downloading has to retry,
  • long import progresses for hours… then randomly dies near the end.

Eventually, the second attempt on my Windows machine completed successfully, printing the magical:

Overpass container initialization complete. Exiting.

That was the moment I stopped trying to perform the heavy initialization on the Linux server and decided to reuse the successfully built Windows DB instead.

6. Copying the finished DB from Windows to Linux 💾➡️🐧

Copying the finished DB from Windows to Linux 💾➡️🐧

Once the DB was built on Windows, the next goal was to reuse it on the OMV server.

On Windows:

cd D:\docker
# overpass-db contains the /db directory from the container
# e.g. mapped like: ./overpass-db:/db

tar -czf overpass-db-us.tar.gz overpass-db

Then the fun part: file transfer.

5.1. First attempts: permission hell & /tmp being too small 😅

  • Copying via scp directly into /srv/docker/ → Permission denied.
  • Copying into /home/pi/ → also permission issues (OMV user shell/home constraints).
  • Copying into /tmp → started transferring, got to ~6 GB, then:write remote "/tmp/overpass-db-us.tar.gz": FailureBecause /tmp is tmpfs in RAM and simply doesn’t have 17+ GB free.

5.2. The working approach

  1. Allow user to write into /srv/docker:sudo mkdir -p /srv/docker sudo chown pi:pi /srv/docker
  2. From Windows, copy the archive there:scp D:\docker\overpass-db-us.tar.gz pi@omv:/srv/docker/
  3. On OMV, unpack it:cd /srv/docker sudo mkdir -p /srv/docker/overpass-db sudo rm -rf /srv/docker/overpass-db/* sudo tar -xzf overpass-db-us.tar.gz -C /srv/docker # Result: /srv/docker/overpass-db/db/...

Now the Linux server has the full Overpass DB, identical to what Docker Desktop created on Windows.

6. Final runtime setup on the server 🐧

For the actual runtime container on OMV, the config can (and should) be much simpler. No more heavy init.

services:
  overpass:
    image: wiktorn/overpass-api
    container_name: overpass-us
    restart: unless-stopped

    ports:
      - "8084:80"

    environment:
      - OVERPASS_MODE=clone
      - OVERPASS_META=yes
      - OVERPASS_STOP_AFTER_INIT=false
      # Optional: silence healthcheck issues
      # - OVERPASS_HEALTHCHECK=/bin/true

    volumes:
      - /srv/docker/overpass-db:/db

Key differences vs init‑compose:

  • No OVERPASS_PLANET_URL, no preprocess.
  • The container assumes DB already exists in /db/db.

7. One more trap: file permissions & Unix sockets 🔐

After copying the DB, the first launches on OMV exploded with errors like:

File_Error Address already in use 98 /db/db//osm3s_osm_base Unix_Socket::4
File_Error Permission denied 13 /db/db//osm3s_areas Unix_Socket::4
/app/bin/rules_loop.sh: line 42: ///db/db/rules_loop.log: Permission denied

and repeated dispatcher_areas, overpass_dispatch restarts.

Root cause:

  • On the host, chown was done to root:root.
  • Inside the container, Overpass runs as a non‑root user (uid 1000).
  • That user couldn’t:
    • create/remove Unix sockets (osm3s_osm_base, osm3s_areas),
    • write to rules_loop.log.

Fix:

# stop container
cd /srv/docker/overpass-data
docker compose down

# give ownership of the DB to uid 1000 (Overpass user in the container)
sudo chown -R 1000:1000 /srv/docker/overpass-db
sudo chmod -R 775 /srv/docker/overpass-db

# clean stale sockets & pid
sudo rm -f /srv/docker/overpass-db/db/osm3s_osm_base
sudo rm -f /srv/docker/overpass-db/db/osm3s_areas
sudo rm -f /srv/docker/overpass-db/db/dispatcher.pid

# start again
docker compose up -d

After that, overpass_dispatch and dispatcher_areas stopped crashing, and the API finally started responding 🎉

8. Verifying that Overpass actually works ✅

Once the container is up, simple health checks are your best friend.

From the OMV server itself:

curl "http://localhost:8084/api/interpreter?data=[out:json];node(around:1000,40.7128,-74.0060);out;"

or from another machine, replacing localhost with the server IP.

  • If you get valid JSON with nodes → your Overpass US instance is alive.
  • If you get 5xx or timeouts → check docker logs overpass-us for permission/sockets errors.

You can then put this behind Nginx Proxy Manager or any other reverse proxy, add TLS, and point your applications (e.g. mapping services, backend jobs, geocoding tools) to this internal Overpass instead of the public one.

9. Lessons learned (so you don’t suffer the same way) 🤕➡️😎

A few takeaways from this adventure:

  1. Architecture matters
    • wiktorn/overpass-api is amd64‑oriented. On ARM you’ll need another image or emulation.
  2. Overpass wants XML, not PBF
    • Use OVERPASS_PLANET_URL + OVERPASS_PLANET_PREPROCESS with osmium to convert *.osm.pbf → .osm.bz2.
  3. Init is expensive
    • Importing us-latest.osm.pbf can easily take 10–12+ hours.
    • It’s perfectly fine to do it once on a powerful Windows box and then copy the DB to Linux.
  4. Volumes & permissions are everything
    • Map /db to a persistent host folder.
    • Make sure that folder is owned by the same uid as the Overpass user in the container (usually 1000:1000).
    • Avoid putting huge files on /tmp (tmpfs) or in directories you can’t write to.
  5. Use different compose configs for init and runtime
    • Init compose: heavy OVERPASS_MODE=init, PLANET_URL, preprocess.
    • Runtime compose: just OVERPASS_MODE=clone + the DB volume.
  6. Don’t trust “ blindly
    • Docker healthcheck for Overpass is nice, but you can always override with OVERPASS_HEALTHCHECK=/bin/true and do your own external checks.

10. Why it was worth it 🚀

After all the retries, copying, and permission battles, the end result is a fast, local, US‑only Overpass instance that:

  • doesn’t depend on public Overpass servers,
  • can handle your own traffic patterns,
  • can be backed up along with the rest of your Docker volumes,
  • and can be rebuilt in a predictable way if needed.

If you’re building anything serious on top of OpenStreetMap in the US – routing, geofencing, location‑based analytics, or custom searches – having your own Overpass is a superpower 🦾

And now you know not only how to run it, but also what can go wrong along the way… so hopefully your journey will be slightly less painful 😉

, ,

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.