mobifree — julian

UnifiedPush -- Wider Developments

Wed, 27 Nov 2024 14:48:00 +0000

An article about Building the Self-Agency Mobile Ecosystem: Push Messaging

Push messaging is the system that enables incoming messages to wake up and reach our apps, instantly and efficiently. But Who Cares Who Delivers Our Notifications? Our answer is: we care, and we do not like being locked in to depending on one mega-corp's system.

Therefore, in our libre mobile computing devices, we require a push-messaging infrastructure built from open standard technology that gives us freedom to choose our service providers and authority over them.

How We Get There

How can we build this? What is needed to get widespread UnifiedPush support in /e/ and other mobile OS?

The big picture involves several areas of work.

OS integration of the distributor, including how it's configured, auto-detected, optimised, etc.
Server Software: Making sure a suitable server-side component is available, supported, optimised for this use case.
Server hosting: making sure a server component is easily deployable by others, including making generic deployment methods (Debian, Docker, Nix, K8s, ...) and integrating it in some hosting systems.
Apps: Working with other apps to make sure they connect and work with UP out-of-the-box.
Sharing this work among other mobile OS projects (microG, CalyxOS, GrapheneOS, PostmarketOS, ...) .
Consider standardising a UP client-server protocol so that a built-in distributor can work with several different server implementations and vice-versa.

(Links in this and the following sections lead to relevant issues or discussions, where I could find any.)

OS Integration of the UP Distributor

Including how the UP distributor is configured, auto-detected, optimised, Android permissions, etc.

I recently analysed how /e/OS 2.5 has started the ball rolling

Server Software

Making sure a suitable server-side component is available, supported, optimised for this use case

Nextcloud: UnifiedPush Provider — exists
It could alternatively begin with a fork of ntfy-server for example.

Server Hosting: Technical

Making sure a server component is easily deployable by others, including making generic deployment methods (Debian, Docker, Nix, K8s, ...) and integrating it in some hosting systems, ranging from hobbyist to serious commercial systems.

Docker ntfy — exists
K8s ntfy helm-chart — exists
Nextcloud: UnifiedPush Provider — exists
Matrix-Docker-Ansible-Deploy: ntfy-server — exists
Yunohost: ntfy-server — exists
SelfHostBlocks — not yet
CoopCloud — not yet
Fediversity (an upcoming hosting project) — not yet

Work out and write up (shareable) policies about who hosts, terms and conditions, privacy, rate limiting and other special measures, etc.?

Apps

Working with other apps to make sure they connect and work with UP out-of-the-box.

Category 1: default apps in /e/OS etc., like...

Mail: K9/Thunderbird-andriod #5165 and UnifiedPush for IMAP (NGI proposal 2024-10-535)
Nextcloud -uppush #17 -notifications #1225 -android #8684
Accounts/Calendar/Contacts (DAVx5 #36 + DAV-push)

Category 2: other popular apps like...

HomeAssistant #446344
Fluffychat — exists
Element — exists
Signal forks
Telegram-FOSS #577
Conversations.im — exists
Pixelfed #86

Standardising UP client-server protocol?

Should we consider standardising the UP client-server protocol? At present, by design, one must choose a UP distributor and UP server together as a pair. If we standardised the protocol between them, so any distributor would work with any server, this would certainly have some benefits, but also down-sides, and is not necessarily the right thing to do. UP was designed this way on purpose and achieves its goals without needing to standardise there. No ordinary person needs to be able to change their UP implementation. Only techies and service providers might like to do that.

Therefore this is an exploration rather than a development. I discuss this further in a separate article.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

Follow/Feedback/Contact: RSS feed · Fedi follow this blog: @julian@wrily.foad.me.uk · matrix me · Fedi follow me · email me · julian.foad.me.uk Donate: via Liberapay All posts © Julian Foad and licensed CC-BY-ND except quotes, translations, or where stated otherwise

UnifiedPush: Standardise UP Client-Server Protocol?

Tue, 26 Nov 2024 20:00:00 +0000

This is a technical article. For a more general introduction to UnifiedPush you might read the first half of Going Google-Free with UnifiedPush in /e/OS or my other articles about UnifiedPush .

Should we consider standardising a UP client-server protocol so that a built-in distributor can work with several different server implementations and vice-versa?

Not necessarily, and I will attempt to explain why.

UnifiedPush High Level Architecture

Look at the diagrams on the unifiedpush.org home page. There are four connections involved, shown arranged in a square. UP standardises the protocol for one of the connections, the bottom one (UP distributor <–> app). It also standardises something about how the whole system works.

The protocol on the top connection (App server –> UP server) has recently been upgraded to be compatible with WebPush.

The other two (App <–> app-server and UP <–> UP-server) are two other, separate, independent protocols. The protocol the app developer uses for app <–> app-server is not relevant to UnifiedPush. At present, UnifiedPush standards also do not specify a protocol to be used for the connection between the UP server and the UP distributor (shown on the left).

The ntfy distributor requires a ntfy server, whereas the nextpush distributor requires a nextpush server, and so on. They use different protocols for that link in the chain.

Multiple Protocols or Standardised Protocol

It would be possible to make a UP server that supports (for example) both the NextPush and ntfy protocol(s). In fact ntfy supports two different protocols (the “json” one and the “websocket” one) and you choose one in the settings. The ntfy server supports both of those.

The point is, UP achieves its goals without needing to standardise there. In some respects it's beneficial that the user and/or service provider can choose those components and protocol that best suit their needs, while the end-user apps and app servers all still remain completely compatible (standardised) no matter what UP components are used.

Additionally standardising the UP <–> UP-server protocol could make different UP servers and distributors interoperable. It could potentially help with users bringing their own choice of device (assuming it has a UP distributor built in) from one service provider to another, for example.

But that's more of a deployment-level standard: something that service providers might want to standardise. Android-compatible mobile OS's (for example) might get together and declare “we” (as a loose group) like to standardise on (let's say) ntfy-websocket protocol here, because (suppose) it works best with android's OS network layer and sleep optimisations. (Just examples; I have not researched what protocol actually works best in each scenario.) Then, Linux-based mobile OS's might choose a different protocol that works better with Linux's networking and sleep optimisations. Then, if a certain libre mobile service provider wanted to support both android-compatible and linux-based devices, they might choose to support two protocols. Or would standardising on one protocol be more helpful in such a scenario? It's not clear.

No ordinary person needs to be able to change their UP implementation. Only techies and service providers might like to do that.

So, at this stage this is perhaps the least important of all the areas to work on.

First explained in #unifiedpush:matrix.org.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

Going Google-Free with UnifiedPush in /e/OS

Tue, 19 Nov 2024 12:00:00 +0000

Congratulations UnifiedPush! Congratulations Murena!

Murena's /e/OS 2.5 ships with UnifiedPush support included as announced by a small note in the 2.5-t release notes .

This exciting development brings Google-free push messaging to the regular users of an important player in the freedom mobile OS space.

So What?

Push messaging is the system that enables incoming messages to wake up and reach our apps, instantly and efficiently.

To achieve this, the phone operating system keeps a single network connection open to a push messaging server. The apps don't need to keep running in the background: they can go to sleep. When a push message arrives, the OS wakes up the relevant app and hands it the message.

All our apps share this push service — even the “private” ones — if they want push messaging.

That's great, except... Who Cares Who Delivers Our Notifications?

Until now, most people's phones run Google or Apple software. The Big Techs operate and control those phones. Along with everything else, they provide the push messaging service. And so they, Big Tech, control it.

The UnifiedPush public open standard changes this. With UnifiedPush, we are no longer merely “users” of our devices but are in charge of them. With UnifiedPush, we can choose which service provider will deliver our push messaging. Thereby we are in control of who has the ability to monitor or block our push messaging.

UnifiedPush has been available to those interested, for a while now, but until now it has required installing extra software.

Putting the support for push messaging into the operating system means ~~users~~ people who choose not to be operated by Big Tech will now be able to simply install the apps they care about and immediately have the alternative system working for them.

Technical

I'm going to delve into the technical side of this initial release, because it's a big interest area of mine. If you are a regular user, I suggest at this stage you might concentrate on switching to /e/OS or another freedom mobile OS and expect this functionality will be maturing quickly from now on and soon be seamless. If you're an early adopter or interested in its development, read on.

I am trying it out. I already run /e/OS on my main phone and upgrading to version 2.5 was simple with the built-in upgrader.

The OS embeds a custom fork of ntfy, calling itself “foundation.e.ntfy 1.17.0”. So, congratulations also to ntfy! At this stage the developer of ntfy, Philip Heckel, told me in #ntfy room he was “not yet involved” in this development and this is the “First I'm hearing about it.”

ntfy is a notification system that can also provide UnifiedPush service. It is so good at this job that it has for some time been the UnifiedPush implementation recommended for most people. One can install the ntfy app and it provides UnifiedPush service to our other apps.

When we install the ntfy app ourself, we may either let it use the creator's default server (at ntfy.sh) or configure it to use our choice of ntfy server, if we know of one or if we run our own. As with anything, using someone else's server comes with conditions and restrictions, especially if we don't pay for the service, so we need to think about that.

Let's see what /e/OS does.

If not Google's then Whose Server?

In this push service built in to /e/OS, there is no configuration UI. There is a tiny settings UI where we have to “enable” the service — see below. Technically, the full ntfy UI is also available, but not accessible through the normal system UI — neither the app launcher nor the settings. But as curious techies we can hack in to it if we like. See below.

What we can do is install and use the UP-example app. Click the “REGISTER” button, and we discover it's preconfigured to connect to... . o O (their own ntfy server?) No! It connects to ntfy.sh, the server run by the author of ntfy. Hmm.

I should say that I found this while not signed in to Murena's “e cloud” services. It's possible and would make some sense if it would point to a different server if signed in. I will be interested to find out.

The first thing to comment on, then, is Murena will surely need to run their own push server, for privacy and scaling and economic reasons, or else make an agreement with ntfy. In the short term, to get things started, it may be acceptable for everyone to be subject to the free (gratis) ntfy service's rate limits and other conditions. That's something that can be changed later, seamlessly, because UnifiedPush doesn't require app (server) developers to know or care which service provider they will eventually be connecting to.

Secondly, for privacy and self-determination reasons, there should be a way to use our own choice of server, but not necessarily as a dedicated configuration option. A better way might be to tie it to the main Nextcloud account through a single-sign-on, so that when we set up the phone we have only to make that one choice of service provider and expect it to cover all services. That would give people the advantage of simplicity which is how the Big Techs play that game, but now with open standard technologies enabling us to have a choice among service providers.

When an app registers for UnifiedPush service, if more than one distributor is installed, it asks which to use. A normal user would have only one, and would not see this. Here as an experimentor and developer I already had both ntfy and NextPush installed:

It's not ideal that it identifies as “ntfy”, exactly the same as the official ntfy app. I expect this will soon be changed to /e/OS branding.

The tiny settings UI is found a the bottom of the system settings menu.

It's disabled by default. In this state, the UnifiedPush service is still advertised to apps, and they can register, but push messages are not delivered. (I suspect this isn't how we want the disabled state to be. Probably it should not advertise as available.)

That's all the configuration we get in this version of /e/OS.

(Aside: I couldn't find this in settings, to begin with. Searching in settings for “ntfy” or “push” or “unifiedpush” doesn't return any results, for me.)

Let's Hack In

As a developer we can access the ntfy UI through the system terminal, for example through an ADB shell (running this command on a USB-connected computer with “adb” tool installed, with ADB USB debugging enabled in the phone's developer settings):

adb shell am start -n foundation.e.ntfy/io.heckel.ntfy.ui.MainActivity

Just the same as in the user-installed ntfy app. (I can compare them side by side because I also have it installed, because I was already using it before this upgrade.)

We first notice this screen prominently shows some warnings about battery optimisation and choice of connection protocol. Are these warnings relevant to this version?, I wonder. (At first it showed only the first, and later both.)

And it lets us open its settings. In the ntfy settings we can choose our own server, and I can confirm this takes effect and works.

This screen also shows the currently subscribed topics — here, the “example” one — and after configuring it to use my own server, the displayed URL confirms that. (Generally one “topic” corresponds to one app.)

Ntfy Not NextPush?

I'm a little surprised Murena chose ntfy, seeing as their system is closely coupled to Nextcloud: there is also Nextpush (source on Codeberg, yay!), a Nextcloud-hosted UnifiedPush server module and its client app, that could be used instead. Perhaps they found ntfy is more stable or refined than Nextpush.

UnifiedPush developer S1m says the “Next version of NextPush will be a lot more stable, and I've rewritten the UI.”

The Future

While UnifiedPush is pretty good standard already, it is still maturing. Some important, recent improvements are making their way into the standard and into implementations like ntfy right now. These include full compatibility with the WebPush standard, and a better way to wake an app without it needing permission for unrestricted battery usage.

An adjacent recent development is DAV Push, an open standard for pushing instant updates to our shared files, calendars and contacts through CalDAV, CardDAV and WebDAV, which can also work through UnifiedPush.

To ensure people can exercise their right NOT to depend on Google, we have wider work to do. Integrating a UnifiedPush distributor is one useful step towards it. Beyond this, we need to be working with the developers of popular apps to get them supported, sharing this development across the freedom mobile OS ecosystem, deploying servers, ensuring reliability, and more. I write about this in a follow-up post: UnifiedPush — Wider Developments .

Conclusion

The details of this initial implementation seem to me to indicate an early preview release, with significant changes still needed. But that's OK: these technical implementation details can be changed.

Leaving that aside, the social side of this is amazing: it's making UP available to main-stream users. That's a big deal for the libre mobile ecosystem. Hurray!

Add UnifiedPush to /e/OS to make it possible for developers to avoid FCM and better support F-Droid applications — an /e/ community forum thread requesting UP support.
Source code of /e/ fork of ntfy: https://gitlab.e.foundation/e/os/ntfy-android
“Integrate ntfy in /e/OS” —merge request, including a simple settings UI (just enable/disable).
Romain Hunault of Murena is presenting it at Capitole du Libre 2024 this afternoon.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

mobifree — julian

UnifiedPush -- Wider Developments

How We Get There

OS Integration of the UP Distributor

Server Software

Server Hosting: Technical

Server Hosting: Social/Business Aspects

Apps

Sharing with other mobile OS projects

Standardising UP client-server protocol?

UnifiedPush: Standardise UP Client-Server Protocol?

UnifiedPush High Level Architecture

Multiple Protocols or Standardised Protocol

Going Google-Free with UnifiedPush in /e/OS

So What?

Technical

If not Google's then Whose Server?

Let's Hack In

Ntfy Not NextPush?

The Future

Conclusion