Self-Host a Matrix Map Tile Server
for secure location sharing with Matrix
I set up my own OpenStreetMap tiles server for use with matrix location sharing.
This guide explains how I did it, including the Ansible role scripts I wrote.
Ansible role available at: https://lab.trax.im/matrix/map-tile-server-ansible
Discussion room: #matrix-map-tile-server:foad.me.uk
What is this for?
FluffyChat implemented location sharing in Matrix last year. More recently, Element announced support for it.
Interesting in Element's announcement is not only the mechanism for sharing a location data point, but also how their matrix clients would display a little map to the recipient and how they would support the case where the data for this map display could be self-hosted. Read their post to understand how that is important for privacy. I am delighted that they did this.
Why am I doing this?
- To be able to check how well it works for Matrix location sharing, and to be able to submit bug reports about it if I find issues.
- To learn more about OSM mapping technology.
- To encourage and help others to self-host their mapping.
- To encourage Element and other software producers to consider and to support these self-hosting use cases.
(And not because I want to run a tile server for its own sake. It is resource-intensive, and self-hosting this particular function is not important for myself. However, I strongly support the principle that others should have the ability to do so, and so I want to strengthen that support.)
How?
To get started, I followed this Matrix map-tile-server guide which is linked from Element's announcement. This guide is based on
- OpenStreetMap tile server software stack
- OSM mapping data compiled and hosted by GeoFabrik
- a guide by Switch2OSM to running the software stack in docker
It worked. Partly. It needed some more work to serve the area I wanted, reliably.
As that blog post puts it, and this one is the same, “this is not a guide on how to host a robust, production-ready map tile server”. However I made mine a bit more production-ready than just a local development setup.
The main changes I made:
- set up with Ansible
- update the map server software version
- import multiple data files, to make the map span multiple regions
To switch to the latest version of the server software I simply used tiles_docker_image: "overv/openstreetmap-tile-server:latest"
(see in defaults/main.yml
below) instead of specifying an old version. I cannot remember if any other tweaks were needed related to this change. When publishing a script for re-use, as I am doing here, it is a good idea to pin to the specific version that worked. So I will tell you, that is v1.8.2.
The other changes are described in their own sections.
Server Specs
To get it to serve data reliably, I mainly needed to bump up the server specs considerably higher than I had at first assumed. To begin with I chose something like 4 GB RAM and 6 CPU cores, 50 GB SSD disk space. My first import step, for the 'Great Britain' data set, completed in a bit under an hour and used 30 to 35 GB disk space for the PostGresQL database it created. Many of the tiles then rendered ok while browsing the map, but several failed, leaving error messages in the container log, indicating it ran out of RAM. And when I sorted out the multi-region and loaded more regions, it ran out of disk space.
Bumping the specs to 12 GB RAM, 10 CPU cores, 80 GB SSD disk space I seem to have no more hard errors. Tiles do sometimes take longer to render than the timeout used in the default map display, which seems to be somewhere around 10 to 20 seconds, leading to the map not updating that area until I re-load the page in the browser and zoom/pan back to that area, and then it loads.
Importing Multiple Map Region Data Files
To import multiple data files, I had to search the documentation of osm2pgsql
. At first I assumed that running the input again with a second file would append data, but it does not. There are just a few clues there, such as “pre-merging is not necessary for newer osm2pgsql version which can read multiple input files at once”. The run.sh
script used by the Dockerfile of Overv/openstreetmap-tile-server
assumes only one input file will be provided, when it runs the import. See:
# Import data
sudo -u renderer osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script /home/renderer/src/openstreetmap-carto/openstreetmap-carto.lua --number-processes ${THREADS:-4} -S /home/renderer/src/openstreetmap-carto/openstreetmap-carto.style /data.osm.pbf ${OSM2PGSQL_EXTRA_ARGS:-}
It (and the Dockerfile) expects to provide a single input file mounted at the in-container path /data.osm.pbf
. But notice that it also supports an OSM2PGSQL_EXTRA_ARGS
option. For now, we can use that to pass the remainder of a set of multiple files. That is what I do in the Ansible scripts below. Ideally, Overv/openstreetmap-tile-server
should be updated to support multiple files directly.
Set up with Ansible
- created a VM accessible by Ansible (not described here)
- wrote an Ansible role named 'tiles' (see below)
- added an Ansible playbook task to my playbook, to import that role (see below)
- routed the domain name
tiles.<mydomain>
to it in my Traefik reverse-proxy configuration (see below) - serve the
styles.json
configuration file that tells the mapping display clients how to read the data from the tiles server (included in the Ansible role below) - serve the extra entry in
.well-known/matrix/client
that tells matrix clients where they can find that file (see below)
Ansible playbook task
This task:
- configures my personal settings and preferences such as my domain name and the map areas I want it to serve;
- runs the
tiles
role (see below), either to generate the tiles from scratch (whentiles_pull
is true, e.g. run withansible-playbook -e pull=true ...
), or by default just to start/restart the tiles server.
In my main playbook, setup.yml
:
- import_role: name=tiles
tags: tiles
vars:
tiles_data_sets:
- europe/great-britain
- europe/guernsey-jersey
- europe/ireland-and-northern-ireland
- europe/isle-of-man
tiles_public_base_url: "https://tiles.{{ my_base_domain }}"
tiles_attribution: "{{ my_domain_friendly_name }} Map Server | Maps © 2022 <a href=\"http://www.geofabrik.de/\">Geofabrik GmbH</a> & <a href=\"http://www.openstreetmap.org/\">OSM Contributors</a>"
tiles_base_dir: "/srv/tiles"
tiles_pull: "{{ pull }}"
Note: the tiles_attribution
goes into the style.json
file and thereby shows up on maps viewed through (matrix) clients. It does not show up on the web map viewer page that is served by default on tiles_public_base_url
. It would be possible to do that too but I have not done that here.
Ansible role 'tiles'
This Ansible role is available at: https://lab.trax.im/matrix/map-tile-server-ansible . The initial version of it, which may be out of date, is reproduced here.
<roles>/tiles/tasks/main.yml
:
---
# OSM tile server
# intended for Matrix location sharing
# see: https://matrix.org/docs/guides/map-tile-server
- name: data directories
file: path="{{ item }}" state=directory
loop:
- "{{ tiles_base_dir }}"
- "{{ tiles_db_dir }}"
- name: data directories
file: path="{{ item }}" state=directory group=1000 mode=775
loop:
- "{{ tiles_tiles_dir }}"
- name: update and import
when: tiles_pull|bool
include_tasks: import.yml
- name: style.json file
template: src=style.json.j2 dest="{{ tiles_style_file }}"
register: tiles_style_json_file_result
- name: tiles container
docker_container:
container_default_behavior: compatibility
name: tiles
image: "{{ tiles_docker_image }}"
pull: "{{ tiles_pull|bool }}"
restart_policy: unless-stopped
restart: "{{ tiles_style_json_file_result.changed }}"
command: ['run']
detach: true
shm_size: "192m" # 192 = (3 x the default 64)
env:
THREADS: "8"
ALLOW_CORS: "enabled"
volumes:
- "{{ tiles_db_dir }}:/var/lib/postgresql/12/main"
- "{{ tiles_tiles_dir }}:/var/lib/mod_tile"
- "{{ tiles_style_file }}:/var/www/html/style.json:ro"
published_ports:
- "8080:80"
<roles>/tiles/tasks/import.yml
:
---
# OSM tile server
# intended for Matrix location sharing
# see: https://matrix.org/docs/guides/map-tile-server
- name: data directories
file: path="{{ tiles_pbf_dir }}" state=directory group=1000 mode=755
- name: data directories
file: path="{{ tiles_pbf_dir }}/{{ item }}" state=directory group=1000 mode=755
loop: "{{ tiles_data_sets | map('dirname') | unique }}"
- name: get OSM data files
get_url:
url: "https://download.geofabrik.de/{{ item }}-latest.osm.pbf"
dest: "{{ tiles_pbf_dir }}/{{ item }}-latest.osm.pbf"
checksum: "{{ item.checksum|default(omit) }}"
loop: "{{ tiles_data_sets }}"
register: tiles_get_osm_data_files_result
- name: record the available data version
copy:
content: "{{ tiles_get_osm_data_files_result.results | json_query('[*].{dest: dest, item: item, size: size, url: url}') | to_nice_yaml(width=100) }}"
dest: "{{ tiles_pbf_dir }}/pbf-available-meta.yaml"
- name: check if db exists
stat: path="{{ tiles_db_dir }}/PG_VERSION"
register: tiles_db_stat_result
- name: manual update required
when: tiles_db_stat_result.stat.exists and tiles_get_osm_data_files_result.changed
fail:
msg: "data changed and DB already exists: to update, manually delete the DB, then re-run"
- name: import into db
when: not tiles_db_stat_result.stat.exists
block:
- name: stop the tiles server container before import
docker_container:
name: tiles
state: absent
- name: import new data into db
docker_container:
container_default_behavior: compatibility
name: tiles-import
image: "{{ tiles_docker_image }}"
pull: "{{ tiles_pull|bool }}"
restart_policy: "no"
command: ['import']
detach: false
cleanup: true
env:
THREADS: "8"
OSM2PGSQL_EXTRA_ARGS: "{{ other_pbfs }}"
volumes:
- "{{ tiles_pbf_dir }}/{{ first_pbf }}-latest.osm.pbf:/data.osm.pbf:ro"
- "{{ tiles_pbf_dir }}:/other_pbfs:rw"
- "{{ tiles_db_dir }}:/var/lib/postgresql/12/main"
- "{{ tiles_tiles_dir }}:/var/lib/mod_tile"
vars:
first_pbf: "{{ tiles_data_sets | first }}"
other_pbfs: "{% for d in tiles_data_sets %}{% if not loop.first %}/other_pbfs/{{ d }}-latest.osm.pbf {% endif %}{% endfor %}"
- name: record the imported data version
copy:
remote_src: true
src: "{{ tiles_pbf_dir }}/pbf-available-meta.yaml"
dest: "{{ tiles_pbf_dir }}/pbf-imported-meta.yaml"
<roles>/tiles/templates/style.json.j2
:
{
"version": 8,
"sources": {
"localsource": {
"type": "raster",
"tiles": [
"{{ tiles_public_base_url }}/tile/{z}/{x}/{y}.png"
],
"tileSize": 256,
"attribution": {{ tiles_attribution|to_json }}
}
},
"layers": [
{
"id": "locallayer",
"source": "localsource",
"type": "raster"
}
]
}
<roles>/tiles/defaults/main.yml
:
---
tiles_public_base_url: ~
tiles_attribution: "Maps © 2022 <a href=\"http://www.geofabrik.de/\">Geofabrik GmbH</a> & <a href=\"http://www.openstreetmap.org/\">OSM Contributors</a>"
tiles_base_dir: "/home/tiles"
tiles_pbf_dir: "{{ tiles_base_dir }}/pbf"
tiles_db_dir: "{{ tiles_base_dir }}/db"
tiles_tiles_dir: "{{ tiles_base_dir }}/tiles"
tiles_style_file: "{{ tiles_base_dir }}/style.json"
tiles_docker_image: "overv/openstreetmap-tile-server:latest"
Reverse-proxy configuration
In a Traefik config file such as rules.yml
:
http:
routers:
tiles:
entryPoints:
- web_https
rule: "Host(`tiles.<MY_BASE_DOMAIN>`)"
service: tiles
services:
tiles:
loadBalancer:
servers:
- url: "http://<MY_VM_IP_ADDRESS>:8080"
With that all configured, a typical zoom-and-pan map display is then available on a web page served my tiles server domain name. That is not the primary purpose of the server, but it is particularly useful to test the setup, and can be useful in itself.
The configuration presented here does not provide authentication or authorization options. I do not know whether any authn/authz support will be available in Element's implementation or in any potential Matrix specification for this map service.
Serving the well-known info
To let matrix clients know they can use this mapping server, as explained in the Matrix.org guide, I added the extra entry in .well-known/matrix/client
that tells matrix clients where they can find the JSON file. Using matrix-docker-ansible-deploy, it can be done by setting a variable like this:
matrix_well_known_matrix_client_configuration_extension_json: |-
{
"m.tile_server": {
"map_style_url": "https://tiles.foad.me.uk/style.json"
}
}
(I did not use the option of adding the equivalent option to my Element-web client's configuration, that is also described in that guide, because my self-hosted Element-web is only one of many matrix clients that I can and do use, and in fact I hardly use that one at all.)
And there it is
I don't intend my instance to be used by others but feel free to take a quick look at it if that helps inform you in your journey of considering running your own self-hosted mapping.
Please join the discussion room if you are interested: #matrix-map-tile-server:foad.me.uk
Conclusions
I added this section in January 2023, looking back on 2022.
Most of the projects I write about are ones I would recommend to others. Looking back on this project, I have mixed feelings. Location sharing with private mapping is certainly a step in the right direction. However there are considerable problems with this approach.
- The promise of privacy is not well met. Many client apps ignore the configured mapping server and send the user's coordinates to some other mapping server that may not be good for privacy. Even among the matrix clients that purport to support this discovery mechanism, I was unable to get some of them to actually discover and use my advertised mapping server. And, so far, there is no mechanism for enforcing or for alerting the users to this issue.
- Raster mapping is a poor choice for today's needs. It's the digital equivalent of a cartographer painting a map and printing copies. Anyone looking at a digital map could reasonably expect and benefit from some degree of interactivity, whether it be zooming and panning, level of detail, addition and removal of features, or localisation of the text labels. Vector mapping is far superior for those purposes.
- The technology stack is optimised for a big centralised service, with a heavy-weight server generating millions of tiny maps for thousands of users. To further Matrix's potential for decentralising and enabling people to own our own communications, we need to pursue “small technology” solutions optimised for serving one or two or five people in a family.
For vector mapping together with a small-tech server side, a promising recent development is Protomaps.
Follow/Feedback/Contact: RSS feed · Fedi follow this blog: @julian@wrily.foad.me.uk · use the Cactus Comments box above · matrix me · Fedi follow me · email me · julian.foad.me.uk Donate: via Liberapay All posts © Julian Foad and licensed CC-BY-ND except quotes, translations, or where stated otherwise