Now I need to address end-to-end-encryption (E2EE). PubHubs exclusively uses encrypted matrix rooms, and Draupnir doesn't yet have E2EE functionality built-in. (Why is that? Moderation in public rooms is Draupnir's main use case, and for several reasons public matrix rooms are usually not encrypted. However PubHubs is different.)

There is a generic solution for adding E2EE to a matrix bot, and it's called Pantalaimon, an “E2EE aware proxy daemon for matrix clients.” So this week I'm setting up Pantalaimon.

The handy matrix-docker-ansible-deploy has no Pantalaimon role ... yet. I may be able to contribute one, but no promises. First I will make a stand-alone Pantalaimon setup role, in my quest to automate, with Ansible, as much as possible of the set-up.

Simple Ansible Role for Pantalaimon

I have made and published a simple Ansible role to run Pantalaimon. It consists of:

• default variables, with (minimal) documentation
• main, build, install tasks
• a pantalaimon.conf template

I use the role in my playbook like this:

  - name: "pantalaimon"
    include_role: name=trax_im.matrix_ansible.pantalaimon
    vars:
      pantalaimon_docker_image: "{{ my_docker_registry_host }}/pantalaimon:latest"
      pantalaimon_docker_network: "{{ matrix_docker_network }}"
      pantalaimon_base_path: /srv/PubHubs/pantalaimon
      pantalaimon_log_level: Debug
      pantalaimon_homeserver_name: example
      pantalaimon_homeserver_url: "https://matrix.example.org"

My intention is to run Pantalaimon in a container, managed by Docker (alternatives like Podman should work similarly), on the target machine where the matrix server runs. Note that, in general, to minimise exposure of encrypted data, Pantalaimon should be run “close to” the bot or client that uses it. At least the unencrypted connection between the bot or client and Pantalaimon should be secured.

There is no Pantalaimon docker image published by its creators, so my role builds it, and pushes it to my private registry. (You could delegate the building to a different machine such as localhost; that's commented out in tasks/main.yml in v1.1.0. Or, building on the target machine, you could avoid using a registry if you set my_docker_registry_host to localhost, I think; you might need to disable the 'push' option in the build task as well.)

Connecting Draupnir to Pantalaimon

To connect Draupnir to Pantalaimon, I am going to show a playbook task that invokes the existing draupnir role. (Usually to set up a regular matrix system I run the 'matrix-docker-ansible-deploy' playbook as a whole. Invoking a role individually from another playbook is working towards using it with PubHubs rather than a regular matrix system.)

The key part here is to override the Draupnir config's homeserverUrl to point to Pantalaimon, while leaving the Draupnir config's rawHomeserverUrl pointing to the matrix server's public client-server API, as documented in Draupnir's example config.

  - name: "draupnir" 
    include_role: name=playbooks-from/matrix-docker-ansible-deploy/roles/custom/matrix-bot-draupnir 
    vars: 
      matrix_bot_draupnir_access_token: "{{ matrix_draupnir_bot_access_token }}"  # using value generated by login step earlier; alternatively read it from a vault-var
      matrix_bot_draupnir_management_room: "{{ matrix_draupnir_management_room_id }}"  # using value generated by room-creation step earlier; alternatively read it from an inventory var
      matrix_bot_draupnir_configuration_extension_yaml: | 
        homeserverUrl: "http://pantalaimon:8009" 
        pantalaimon: 
          use: true 
          username: "bot.draupnir-admin" 
          password: "{{ matrix_draupnir_bot_password }}"  # using an inventory vault-var
        protectAllJoinedRooms: true  # my preference
        commands: 
          allowNoPrefix: true  # my preference
      matrix_bot_draupnir_systemd_required_services_list: 
        - docker.service 
        - matrix-synapse.service 
        - matrix-pantalaimon.service 

These vars would better be placed in the inventory than here. To use the role stand-alone like this we also need to set a few other base variables from matrix-docker-ansible-deploy, which we can do like this:

  vars_files:  # defined at the playbook level
    - playbooks-from/matrix-docker-ansible-deploy/roles/galaxy/com.devture.ansible.role.playbook_help/defaults/main.yml
    - playbooks-from/matrix-docker-ansible-deploy/roles/galaxy/com.devture.ansible.role.systemd_docker_base/defaults/main.yml
    - playbooks-from/matrix-docker-ansible-deploy/roles/custom/matrix-base/defaults/main.yml

  vars:  # defined at the task or block level
    matrix_user_uid: 991  # to match our existing matrix installation
    matrix_user_gid: 991
    matrix_homeserver_url: "https://matrix.example"

We have looked at a simple stand-alone Pantalaimon role. Now to integrate it in matrix-docker-ansible-deploy.

M-D-A-D Playbook Role for Pantalaimon

The first draft of a suitable role is published on TraxLab: roles/matrix_pantalaimon.

[2023-10-19] I prepared a merge request to the m-d-a-d playbook:

Pantalaimon role is activated by matrix_pantalaimon_enabled and connects to the homeserver by default. No further config is needed. Integration with Draupnir is included, activated by matrix_bot_draupnir_use_pantalaimon. No integration with Mjolnir or anything else. Alternatively Pantalaimon can be configured alone, and then whatever component is meant to talk to it, in or out of the playbook, may be configured separately/manually.

... and Slavi has responded to me with a bunch of good review comments which I need to address.

UPDATE 2023-12-21: Addressed those review comments.

UPDATE 2024-03-29: This is now accepted and included in matrix-docker-ansible-deploy.

#matrix #awesomeFOSS #Draupnir #PubHubs

Follow/Feedback/Contact: RSS feed · Fedi follow this blog: @julian​@wrily.foad.me.uk · matrix me · Fedi follow me · email me · julian.foad.me.uk Donate: via Liberapay All posts © Julian Foad and licensed CC-BY-ND except quotes, translations, or where stated otherwise

Very glad to be able to use matrix-docker-ansible-deploy's Draupnir setup to automate the majority of the Draupnir deployment.

I also want to automate, with Ansible, as much as possible of the set-up that is required before running that playbook. I aim to document here what I have done and open questions about it. The numbered steps here correspond to the manual instructions in that documentation linked above.

1. Register the bot account

I register a matrix account for the bot, using a little Ansible role I wrote, calling it like this in my Ansible playbook (personal one, specific to this installation).

  - include_role: name=matrix-synapse-register-user
    vars:
      matrix_synapse_user:
        name:  "{{ matrix_draupnir_bot_username }}"
        pw:    "{{ matrix_draupnir_bot_password }}"
        admin: "{{ matrix_draupnir_bot_admin }}"

defining those vars in my inventory file inventories/prod/host_vars/<host>/matrix-draupnir.yml:

matrix_draupnir_bot_username: "bot.draupnir"
matrix_draupnir_bot_password: !vault [... I use vault encoded values here ...]
matrix_draupnir_bot_admin: false

I published my matrix-synapse-register-user role. I thought I should, if there isn't already a more widely available alternative. It looks like this, in tasks/main.yml:

---
- name: "register a matrix synapse user"
  command:
    argv: "{{ (cmd_check if ansible_check_mode else cmd_real) + args }}"
  vars:
    cmd_real:
      - /matrix/synapse/bin/register-user
    cmd_check:
      - echo
      - "Would run: /matrix/synapse/bin/register-user"
    args:
      - "{{ matrix_synapse_user.name }}"
      - "{{ matrix_synapse_user.pw }}"
      - "{{ '1' if matrix_synapse_user.admin|default(false) else '0' }}"
  check_mode: false
  register: result
  changed_when: result.rc == 0
  failed_when: result.rc != 0 and 'User ID already taken.' not in result.stdout
  # if already registered, errors with (on stdout):
  # > Sending registration request...
  # > ERROR! Received 400 Bad Request
  # > User ID already taken.

2. Get an access token

Manually get an access token. OK I know how to do that. First question, though: as that's a speed bump in the road, and seems fragile, has anyone shown interest in making either Draupnir itself or the playbook do a login automatically? Can't see any issue filed about “token” or “login”.

Basically these days I know I'm going to have to repeat my steps on a different rig later, tear things down and build them up again, so I always look to automate the whole procedure. (Aware it's sometimes more efficient to go manually at first and automate only when worth the effort. And aware I'm at risk of volunteering myself to contribute.)

Well, I asked about this. The kind folks let me know their thoughts. There are several issues.

When I said “fragile” I meant I wouldn't expect an access token to remain valid forever. At the time this was designed, access tokens were considered permanent until explicitly revoked (“logged out”). Putting an access token into a bot's config was taken for granted, and perhaps still is. However, nowadays the client authentication spec is more complex and tokens may need refreshing. That seems to me to suggest that's no longer good practice. It can still work if we take care that the token is not invalidated.

In one sense using an access token is considered more secure than knowing an account's password. That's the sense in which many systems allow getting an API access token and giving it to an external service that will call the API. However, I think this argument applies to accessing other accounts, not for the bot's own account.

Also, if Draupnir connects through Pantalaimon to get E2EE (as makes sense when it's being run by someone other than the server operator), then Pantalaimon needs the account password to create an E2EE device.

Currently I'm feeling the bot should know its password and we should automate its login. As a lesser step, I will do this in Ansible.

  - include_role: name=matrix-login-password
    vars:
      matrix_login:
        hs_cs_api: "{{ matrix_draupnir_hs_cs_api }}"
        user: "{{ matrix_draupnir_bot_user_id }}"
        password: "{{ matrix_draupnir_bot_password }}"
    # output: matrix_login_result
  - set_fact:
      matrix_draupnir_bot_access_token: "{{ matrix_login_result.access_token }}"

with additional inventory vars:

matrix_draupnir_hs_cs_api: "https://matrix.example.net"
matrix_draupnir_bot_user_id: "@bot.draupnir:example.net"

My matrix-login-password role has this in its tasks/main.yml:

- name: "log in to matrix"
  uri:
    method: POST
    url: "{{ matrix_login.hs_cs_api }}/_matrix/client/r0/login"
    body:
      type: "m.login.password"
      identifier:
        type: "m.id.user"
        user: "{{ matrix_login.user }}"
      password: "{{ matrix_login.password }}"
    body_format: json
  register: _result
  changed_when: _result.json.access_token

- set_fact:
    matrix_login_result: "{{ _result.json }}"
  when: not ansible_check_mode

# matrix_login_result contains at least: access_token, device_id, user_id
# see e.g.: https://spec.matrix.org/v1.8/client-server-api/#login and older versions

[Update: now published too.] (I hesitated because I thought I saw about a year ago someone had already published a set of ansible roles for matrix admin tasks like this. Maybe. Can't find it now.)

3. Make sure the account is free from rate limiting

Same for this 'override_ratelimit' step of course. Would be nice to automate.

One way to change this permission is through some other admin account. In that case, “know an access token” is an appropriate way to use that other admin account.

In the case where the bot account itself is configured to be a (matrix) server admin account, then in Synapse's case at least it would already have sufficient permission to use Synapse's admin API to override its own rate limit.

Now about admin APIs. Unfortunately matrix admin APIs are not standardised. Synapse has its admin API, Dendrite has another, and Conduit I gather doesn't have a REST admin API. On Dendrite, “the username has to be specified in dendrite.yaml to disable rate limiting, and personally i hate when anything other than the sysadmins write to configs” said bones_was_here. The playbook, if instructed to install Dendrite, controls that config file, but Draupnir would (or should) not be able to do that by itself unlike with Synapse.

So, there are lots of cases. Different homeservers, and the playbook being used to install Draupnir with or without also its homeserver, and choice of whether the playbook or Draupnir itself performs this configuration.

Currently I'm think I will automate it for the Synapse admin set-ratelimit API only, as that's the server we use in PubHubs.

  - name: "disable rate limiting for Draupnir bot"
    uri:
      method: POST
      url: "{{ matrix_draupnir_hs_cs_api }}/_synapse/admin/v1/users/{{ matrix_draupnir_bot_user_id }}/override_ratelimit"
      headers:
        # access token must be for a user with synapse admin access;
        # can be the bot's if it is an admin, else of another account.
        Authorization: "Bearer {{ matrix_draupnir_bot_access_token if matrix_draupnir_bot_admin else matrix_draupnir_some_synapse_admin_account_access_token }}"
      body_format: json
      body:
        messages_per_second: 0
        burst_count: 0

4. Create a management room

I'm on a roll now. I'll just check the matrix spec for room creation, take a guess at which parameters make most sense for my case, and write it out in Ansible language.

  - name: "create a management room for Draupnir bot" 
    uri: 
      method: POST 
      url: "{{ matrix_draupnir_hs_cs_api }}/_matrix/client/v3/createRoom" 
      headers: 
        Authorization: "Bearer {{ matrix_draupnir_bot_access_token }}" 
      body: 
        name: "{{ matrix_draupnir_management_room_name }}" 
        creation_content: 
          m.federate: false 
        visibility: private 
        preset: trusted_private_chat 
        invite: "{{ matrix_draupnir_operator_user_ids }}" 
      body_format: json 
    register: _result 
    when: matrix_draupnir_management_room_id is undefined 
    # output: _result.json.room_id 
  - set_fact: 
      matrix_draupnir_management_room_id: "{{ _result.json.room_id }}" 
    when: matrix_draupnir_management_room_id is undefined 

with inventory vars:

# the management room (bot and its operators); create if room_id undefined
matrix_draupnir_management_room_name: "Draupnir management"
matrix_draupnir_management_room_id: '!xxxxxxxxxxxx:example.net'

Starting Up

After running my playbook with the above set-up, and pasting the resulting access token and room id into the corresponding inventory vars (TODO: join the two parts together in a better way than cut-n-paste), here we go with matrix-docker-ansible-deploy:

ansible-playbook .../matrix-docker-ansible-deploy/setup.yml -l example.net --tags=setup-bot-draupnir,start -Dv

In the logs, journalctl -n100 -fu matrix-bot-draupnir.service:

Starting Matrix Draupnir bot...
Started Matrix Draupnir bot.
[INFO] [index] Starting bot...
[INFO] [index] Resolving management room...
[INFO] [index] Mjolnir is starting up. Use !mjolnir to query status.
[INFO] [ProtectedRoomsConfig] Resolving protected rooms...
[WARN] [ProtectedRoomsConfig] Couldn't find any explicitly protected rooms from Mjolnir's account data, assuming first start. MatrixError: Error during MatrixClient request GET /_matrix/client/v3/user/%40bot.draupnir%3Aexample.net/account_data/org.matrix.mjolnir.protected_rooms: 404 Not Found -- {"errcode":"M_NOT_FOUND","error":"Account data not found"}
[... three of these 'Account data not found' errors ...]
[INFO] [Mjolnir@startup] Checking permissions...
[INFO] [Mjolnir@startup] Syncing lists...
[INFO] [Mjolnir@startup] Startup complete. Now monitoring rooms.

And then I found and joined the management room. I used a Hydrogen web client.

_@bot.draupnir:example.net joined the room_
_bot.draupnir named the room "Draupnir management"_
_admin1 was invited to the room by bot.draupnir_

**bot.draupnir:**
Mjolnir is starting up. Use !mjolnir to query status.
Checking permissions...
All permissions look OK.
Syncing lists...
Done updating rooms - no errors
Startup complete. Now monitoring rooms.

_admin1 joined the room_

It's alive! Perhaps a little confused about its new name. Responds to both !mjolnir and !draupnir, either way replying:

Old Commands:
`!mjolnir        - Print status information`
`!mjolnir status - Print status information`
[...]

mjolnir commands:`ban <entity> <list> [...reason]` - Bans an entity from the policy list.Parameters:
entity - no description
list - no description
[...]

Well, there we are. The bot's alive. Next it's time for me to learn its commands and put it to work.

Update: All these Ansible roles are now published. TODO: contribute some of this to matrix-docker-ansible-deploy? TODO: re-implement these roles as Ansible modules instead, using a matrix python API such as synadm or mautrix-python?

#matrix #awesomeFOSS #Draupnir #PubHubs

Follow/Feedback/Contact: RSS feed · Fedi follow this blog: @julian​@wrily.foad.me.uk · matrix me · Fedi follow me · email me · julian.foad.me.uk Donate: via Liberapay All posts © Julian Foad and licensed CC-BY-ND except quotes, translations, or where stated otherwise