Self-Host a Matrix Map Tile Server

for secure location sharing with Matrix

I set up my own OpenStreetMap tiles server for use with matrix location sharing.

This guide explains how I did it, including the Ansible role scripts I wrote.

Ansible role available at: https://lab.trax.im/matrix/map-tile-server-ansible

Discussion room: #matrix-map-tile-server:foad.me.uk

What is this for?

FluffyChat implemented location sharing in Matrix last year. More recently, Element announced support for it.

Interesting in Element's announcement is not only the mechanism for sharing a location data point, but also how their matrix clients would display a little map to the recipient and how they would support the case where the data for this map display could be self-hosted. Read their post to understand how that is important for privacy. I am delighted that they did this.

Why am I doing this?

(And not because I want to run a tile server for its own sake. It is resource-intensive, and self-hosting this particular function is not important for myself. However, I strongly support the principle that others should have the ability to do so, and so I want to strengthen that support.)

How?

To get started, I followed this Matrix map-tile-server guide which is linked from Element's announcement. This guide is based on

It worked. Partly. It needed some more work to serve the area I wanted, reliably.

As that blog post puts it, and this one is the same, “this is not a guide on how to host a robust, production-ready map tile server”. However I made mine a bit more production-ready than just a local development setup.

The main changes I made:

To switch to the latest version of the server software I simply used tiles_docker_image: "overv/openstreetmap-tile-server:latest" (see in defaults/main.yml below) instead of specifying an old version. I cannot remember if any other tweaks were needed related to this change. When publishing a script for re-use, as I am doing here, it is a good idea to pin to the specific version that worked. So I will tell you, that is v1.8.2.

The other changes are described in their own sections.

Server Specs

To get it to serve data reliably, I mainly needed to bump up the server specs considerably higher than I had at first assumed. To begin with I chose something like 4 GB RAM and 6 CPU cores, 50 GB SSD disk space. My first import step, for the 'Great Britain' data set, completed in a bit under an hour and used 30 to 35 GB disk space for the PostGresQL database it created. Many of the tiles then rendered ok while browsing the map, but several failed, leaving error messages in the container log, indicating it ran out of RAM. And when I sorted out the multi-region and loaded more regions, it ran out of disk space.

Bumping the specs to 12 GB RAM, 10 CPU cores, 80 GB SSD disk space I seem to have no more hard errors. Tiles do sometimes take longer to render than the timeout used in the default map display, which seems to be somewhere around 10 to 20 seconds, leading to the map not updating that area until I re-load the page in the browser and zoom/pan back to that area, and then it loads.

Importing Multiple Map Region Data Files

To import multiple data files, I had to search the documentation of osm2pgsql. At first I assumed that running the input again with a second file would append data, but it does not. There are just a few clues there, such as “pre-merging is not necessary for newer osm2pgsql version which can read multiple input files at once”. The run.sh script used by the Dockerfile of Overv/openstreetmap-tile-server assumes only one input file will be provided, when it runs the import. See:

    # Import data
    sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes ${THREADS:-4} -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf ${OSM2PGSQL_EXTRA_ARGS:-}

It (and the Dockerfile) expects to provide a single input file mounted at the in-container path /data.osm.pbf. But notice that it also supports an OSM2PGSQL_EXTRA_ARGS option. For now, we can use that to pass the remainder of a set of multiple files. That is what I do in the Ansible scripts below. Ideally, Overv/openstreetmap-tile-server should be updated to support multiple files directly.

Set up with Ansible

Ansible playbook task

This task:

In my main playbook, setup.yml:

    - import_role: name=tiles
      tags: tiles
      vars:
        tiles_data_sets:
          - europe/great-britain
          - europe/guernsey-jersey
          - europe/ireland-and-northern-ireland
          - europe/isle-of-man
        tiles_public_base_url: "https://tiles.{{ my_base_domain }}"
        tiles_attribution: "{{ my_domain_friendly_name }} Map Server | Maps &copy; 2022 <a href=\"http://www.geofabrik.de/\">Geofabrik GmbH</a> &amp; <a href=\"http://www.openstreetmap.org/\">OSM Contributors</a>"
        tiles_base_dir: "/srv/tiles"
        tiles_pull: "{{ pull }}"

Note: the tiles_attribution goes into the style.json file and thereby shows up on maps viewed through (matrix) clients. It does not show up on the web map viewer page that is served by default on tiles_public_base_url. It would be possible to do that too but I have not done that here.

Ansible role 'tiles'

This Ansible role is available at: https://lab.trax.im/matrix/map-tile-server-ansible . The initial version of it, which may be out of date, is reproduced here.

<roles>/tiles/tasks/main.yml:

---
# OSM tile server
# intended for Matrix location sharing
# see: https://matrix.org/docs/guides/map-tile-server

- name: data directories
  file: path="{{ item }}" state=directory
  loop:
      - "{{ tiles_base_dir }}"
      - "{{ tiles_db_dir }}"

- name: data directories
  file: path="{{ item }}" state=directory group=1000 mode=775
  loop:
      - "{{ tiles_tiles_dir }}"

- name: update and import
  when: tiles_pull|bool
  include_tasks: import.yml

- name: style.json file
  template: src=style.json.j2 dest="{{ tiles_style_file }}"
  register: tiles_style_json_file_result

- name: tiles container
  docker_container:
    container_default_behavior: compatibility
    name: tiles
    image: "{{ tiles_docker_image }}"
    pull: "{{ tiles_pull|bool }}"
    restart_policy: unless-stopped
    restart: "{{ tiles_style_json_file_result.changed }}"
    command: ['run']
    detach: true
    shm_size: "192m"  # 192 = (3 x the default 64)
    env:
      THREADS: "8"
      ALLOW_CORS: "enabled"
    volumes:
      - "{{ tiles_db_dir }}:/var/lib/postgresql/12/main"
      - "{{ tiles_tiles_dir }}:/var/lib/mod_tile"
      - "{{ tiles_style_file }}:/var/www/html/style.json:ro"
    published_ports:
      - "8080:80"

<roles>/tiles/tasks/import.yml:

---
# OSM tile server
# intended for Matrix location sharing
# see: https://matrix.org/docs/guides/map-tile-server

- name: data directories
  file: path="{{ tiles_pbf_dir }}" state=directory group=1000 mode=755

- name: data directories
  file: path="{{ tiles_pbf_dir }}/{{ item }}" state=directory group=1000 mode=755
  loop: "{{ tiles_data_sets | map('dirname') | unique }}"

- name: get OSM data files
  get_url:
    url: "https://download.geofabrik.de/{{ item }}-latest.osm.pbf"
    dest: "{{ tiles_pbf_dir }}/{{ item }}-latest.osm.pbf"
    checksum: "{{ item.checksum|default(omit) }}"
  loop: "{{ tiles_data_sets }}"
  register: tiles_get_osm_data_files_result

- name: record the available data version
  copy:
    content: "{{ tiles_get_osm_data_files_result.results | json_query('[*].{dest: dest, item: item, size: size, url: url}') | to_nice_yaml(width=100) }}"
    dest: "{{ tiles_pbf_dir }}/pbf-available-meta.yaml"

- name: check if db exists
  stat: path="{{ tiles_db_dir }}/PG_VERSION"
  register: tiles_db_stat_result

- name: manual update required
  when: tiles_db_stat_result.stat.exists and tiles_get_osm_data_files_result.changed
  fail:
    msg: "data changed and DB already exists: to update, manually delete the DB, then re-run"

- name: import into db
  when: not tiles_db_stat_result.stat.exists
  block:

    - name: stop the tiles server container before import
      docker_container:
        name: tiles
        state: absent

    - name: import new data into db
      docker_container:
        container_default_behavior: compatibility
        name: tiles-import
        image: "{{ tiles_docker_image }}"
        pull: "{{ tiles_pull|bool }}"
        restart_policy: "no"
        command: ['import']
        detach: false
        cleanup: true
        env:
          THREADS: "8"
          OSM2PGSQL_EXTRA_ARGS: "{{ other_pbfs }}"
        volumes:
          - "{{ tiles_pbf_dir }}/{{ first_pbf }}-latest.osm.pbf:/data.osm.pbf:ro"
          - "{{ tiles_pbf_dir }}:/other_pbfs:rw"
          - "{{ tiles_db_dir }}:/var/lib/postgresql/12/main"
          - "{{ tiles_tiles_dir }}:/var/lib/mod_tile"
      vars:
        first_pbf: "{{ tiles_data_sets | first }}"
        other_pbfs: "{% for d in tiles_data_sets %}{% if not loop.first %}/other_pbfs/{{ d }}-latest.osm.pbf {% endif %}{% endfor %}"

    - name: record the imported data version
      copy:
        remote_src: true
        src: "{{ tiles_pbf_dir }}/pbf-available-meta.yaml"
        dest: "{{ tiles_pbf_dir }}/pbf-imported-meta.yaml"

<roles>/tiles/templates/style.json.j2:

{
  "version": 8,
  "sources": {
    "localsource": {
      "type": "raster",
      "tiles": [
        "{{ tiles_public_base_url }}/tile/{z}/{x}/{y}.png"
      ],
      "tileSize": 256,
      "attribution": {{ tiles_attribution|to_json }}
    }
  },
  "layers": [
    {
      "id": "locallayer",
      "source": "localsource",
      "type": "raster"
    }
  ]
}

<roles>/tiles/defaults/main.yml:

---
tiles_public_base_url: ~
tiles_attribution: "Maps &copy; 2022 <a href=\"http://www.geofabrik.de/\">Geofabrik GmbH</a> &amp; <a href=\"http://www.openstreetmap.org/\">OSM Contributors</a>"
tiles_base_dir: "/home/tiles"
tiles_pbf_dir: "{{ tiles_base_dir }}/pbf"
tiles_db_dir: "{{ tiles_base_dir }}/db"
tiles_tiles_dir: "{{ tiles_base_dir }}/tiles"
tiles_style_file: "{{ tiles_base_dir }}/style.json"
tiles_docker_image: "overv/openstreetmap-tile-server:latest"

Reverse-proxy configuration

In a Traefik config file such as rules.yml:

http:
  routers:
    tiles:
      entryPoints:
        - web_https
      rule: "Host(`tiles.<MY_BASE_DOMAIN>`)"
      service: tiles

  services:
    tiles:
      loadBalancer:
        servers:
          - url: "http://<MY_VM_IP_ADDRESS>:8080"

With that all configured, a typical zoom-and-pan map display is then available on a web page served my tiles server domain name. That is not the primary purpose of the server, but it is particularly useful to test the setup, and can be useful in itself.

The configuration presented here does not provide authentication or authorization options. I do not know whether any authn/authz support will be available in Element's implementation or in any potential Matrix specification for this map service.

Serving the well-known info

To let matrix clients know they can use this mapping server, as explained in the Matrix.org guide, I added the extra entry in .well-known/matrix/client that tells matrix clients where they can find the JSON file. Using matrix-docker-ansible-deploy, it can be done by setting a variable like this:

matrix_well_known_matrix_client_configuration_extension_json: |-
  {
    "m.tile_server": {
      "map_style_url": "https://tiles.foad.me.uk/style.json"
    }
  }

(I did not use the option of adding the equivalent option to my Element-web client's configuration, that is also described in that guide, because my self-hosted Element-web is only one of many matrix clients that I can and do use, and in fact I hardly use that one at all.)

And there it is

I don't intend my instance to be used by others but feel free to take a quick look at it if that helps inform you in your journey of considering running your own self-hosted mapping.

Please join the discussion room if you are interested: #matrix-map-tile-server:foad.me.uk

Conclusions

I added this section in January 2023, looking back on 2022.

Most of the projects I write about are ones I would recommend to others. Looking back on this project, I have mixed feelings. Location sharing with private mapping is certainly a step in the right direction. However there are considerable problems with this approach.

For vector mapping together with a small-tech server side, a promising recent development is Protomaps.

#degoogled #awesomeFOSS

Follow/Feedback/Contact: RSS feed · Fedi follow this blog: @julian​@wrily.foad.me.uk · use the Cactus Comments box above · matrix me · Fedi follow me · email me · julian.foad.me.uk Donate: via Liberapay All posts © Julian Foad and licensed CC-BY-ND except quotes, translations, or where stated otherwise