ntfy — julian

UnifiedPush -- Wider Developments

Wed, 27 Nov 2024 14:48:00 +0000

An article about Building the Self-Agency Mobile Ecosystem: Push Messaging

Push messaging is the system that enables incoming messages to wake up and reach our apps, instantly and efficiently. But Who Cares Who Delivers Our Notifications? Our answer is: we care, and we do not like being locked in to depending on one mega-corp's system.

Therefore, in our libre mobile computing devices, we require a push-messaging infrastructure built from open standard technology that gives us freedom to choose our service providers and authority over them.

How We Get There

How can we build this? What is needed to get widespread UnifiedPush support in /e/ and other mobile OS?

The big picture involves several areas of work.

OS integration of the distributor, including how it's configured, auto-detected, optimised, etc.
Server Software: Making sure a suitable server-side component is available, supported, optimised for this use case.
Server hosting: making sure a server component is easily deployable by others, including making generic deployment methods (Debian, Docker, Nix, K8s, ...) and integrating it in some hosting systems.
Apps: Working with other apps to make sure they connect and work with UP out-of-the-box.
Sharing this work among other mobile OS projects (microG, CalyxOS, GrapheneOS, PostmarketOS, ...) .
Consider standardising a UP client-server protocol so that a built-in distributor can work with several different server implementations and vice-versa.

(Links in this and the following sections lead to relevant issues or discussions, where I could find any.)

OS Integration of the UP Distributor

Including how the UP distributor is configured, auto-detected, optimised, Android permissions, etc.

I recently analysed how /e/OS 2.5 has started the ball rolling

Server Software

Making sure a suitable server-side component is available, supported, optimised for this use case

Nextcloud: UnifiedPush Provider — exists
It could alternatively begin with a fork of ntfy-server for example.

Server Hosting: Technical

Making sure a server component is easily deployable by others, including making generic deployment methods (Debian, Docker, Nix, K8s, ...) and integrating it in some hosting systems, ranging from hobbyist to serious commercial systems.

Docker ntfy — exists
K8s ntfy helm-chart — exists
Nextcloud: UnifiedPush Provider — exists
Matrix-Docker-Ansible-Deploy: ntfy-server — exists
Yunohost: ntfy-server — exists
SelfHostBlocks — not yet
CoopCloud — not yet
Fediversity (an upcoming hosting project) — not yet

Work out and write up (shareable) policies about who hosts, terms and conditions, privacy, rate limiting and other special measures, etc.?

Apps

Working with other apps to make sure they connect and work with UP out-of-the-box.

Category 1: default apps in /e/OS etc., like...

Mail: K9/Thunderbird-andriod #5165 and UnifiedPush for IMAP (NGI proposal 2024-10-535)
Nextcloud -uppush #17 -notifications #1225 -android #8684
Accounts/Calendar/Contacts (DAVx5 #36 + DAV-push)

Category 2: other popular apps like...

HomeAssistant #446344
Fluffychat — exists
Element — exists
Signal forks
Telegram-FOSS #577
Conversations.im — exists
Pixelfed #86

Standardising UP client-server protocol?

Should we consider standardising the UP client-server protocol? At present, by design, one must choose a UP distributor and UP server together as a pair. If we standardised the protocol between them, so any distributor would work with any server, this would certainly have some benefits, but also down-sides, and is not necessarily the right thing to do. UP was designed this way on purpose and achieves its goals without needing to standardise there. No ordinary person needs to be able to change their UP implementation. Only techies and service providers might like to do that.

Therefore this is an exploration rather than a development. I discuss this further in a separate article.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

Follow/Feedback/Contact: RSS feed · Fedi follow this blog: @julian@wrily.foad.me.uk · matrix me · Fedi follow me · email me · julian.foad.me.uk Donate: via Liberapay All posts © Julian Foad and licensed CC-BY-ND except quotes, translations, or where stated otherwise

UnifiedPush: Standardise UP Client-Server Protocol?

Tue, 26 Nov 2024 20:00:00 +0000

This is a technical article. For a more general introduction to UnifiedPush you might read the first half of Going Google-Free with UnifiedPush in /e/OS or my other articles about UnifiedPush .

Should we consider standardising a UP client-server protocol so that a built-in distributor can work with several different server implementations and vice-versa?

Not necessarily, and I will attempt to explain why.

UnifiedPush High Level Architecture

Look at the diagrams on the unifiedpush.org home page. There are four connections involved, shown arranged in a square. UP standardises the protocol for one of the connections, the bottom one (UP distributor <–> app). It also standardises something about how the whole system works.

The protocol on the top connection (App server –> UP server) has recently been upgraded to be compatible with WebPush.

The other two (App <–> app-server and UP <–> UP-server) are two other, separate, independent protocols. The protocol the app developer uses for app <–> app-server is not relevant to UnifiedPush. At present, UnifiedPush standards also do not specify a protocol to be used for the connection between the UP server and the UP distributor (shown on the left).

The ntfy distributor requires a ntfy server, whereas the nextpush distributor requires a nextpush server, and so on. They use different protocols for that link in the chain.

Multiple Protocols or Standardised Protocol

It would be possible to make a UP server that supports (for example) both the NextPush and ntfy protocol(s). In fact ntfy supports two different protocols (the “json” one and the “websocket” one) and you choose one in the settings. The ntfy server supports both of those.

The point is, UP achieves its goals without needing to standardise there. In some respects it's beneficial that the user and/or service provider can choose those components and protocol that best suit their needs, while the end-user apps and app servers all still remain completely compatible (standardised) no matter what UP components are used.

Additionally standardising the UP <–> UP-server protocol could make different UP servers and distributors interoperable. It could potentially help with users bringing their own choice of device (assuming it has a UP distributor built in) from one service provider to another, for example.

But that's more of a deployment-level standard: something that service providers might want to standardise. Android-compatible mobile OS's (for example) might get together and declare “we” (as a loose group) like to standardise on (let's say) ntfy-websocket protocol here, because (suppose) it works best with android's OS network layer and sleep optimisations. (Just examples; I have not researched what protocol actually works best in each scenario.) Then, Linux-based mobile OS's might choose a different protocol that works better with Linux's networking and sleep optimisations. Then, if a certain libre mobile service provider wanted to support both android-compatible and linux-based devices, they might choose to support two protocols. Or would standardising on one protocol be more helpful in such a scenario? It's not clear.

No ordinary person needs to be able to change their UP implementation. Only techies and service providers might like to do that.

So, at this stage this is perhaps the least important of all the areas to work on.

First explained in #unifiedpush:matrix.org.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

Going Google-Free with UnifiedPush in /e/OS

Tue, 19 Nov 2024 12:00:00 +0000

Congratulations UnifiedPush! Congratulations Murena!

Murena's /e/OS 2.5 ships with UnifiedPush support included as announced by a small note in the 2.5-t release notes .

This exciting development brings Google-free push messaging to the regular users of an important player in the freedom mobile OS space.

So What?

Push messaging is the system that enables incoming messages to wake up and reach our apps, instantly and efficiently.

To achieve this, the phone operating system keeps a single network connection open to a push messaging server. The apps don't need to keep running in the background: they can go to sleep. When a push message arrives, the OS wakes up the relevant app and hands it the message.

All our apps share this push service — even the “private” ones — if they want push messaging.

That's great, except... Who Cares Who Delivers Our Notifications?

Until now, most people's phones run Google or Apple software. The Big Techs operate and control those phones. Along with everything else, they provide the push messaging service. And so they, Big Tech, control it.

The UnifiedPush public open standard changes this. With UnifiedPush, we are no longer merely “users” of our devices but are in charge of them. With UnifiedPush, we can choose which service provider will deliver our push messaging. Thereby we are in control of who has the ability to monitor or block our push messaging.

UnifiedPush has been available to those interested, for a while now, but until now it has required installing extra software.

Putting the support for push messaging into the operating system means ~~users~~ people who choose not to be operated by Big Tech will now be able to simply install the apps they care about and immediately have the alternative system working for them.

Technical

I'm going to delve into the technical side of this initial release, because it's a big interest area of mine. If you are a regular user, I suggest at this stage you might concentrate on switching to /e/OS or another freedom mobile OS and expect this functionality will be maturing quickly from now on and soon be seamless. If you're an early adopter or interested in its development, read on.

I am trying it out. I already run /e/OS on my main phone and upgrading to version 2.5 was simple with the built-in upgrader.

The OS embeds a custom fork of ntfy, calling itself “foundation.e.ntfy 1.17.0”. So, congratulations also to ntfy! At this stage the developer of ntfy, Philip Heckel, told me in #ntfy room he was “not yet involved” in this development and this is the “First I'm hearing about it.”

ntfy is a notification system that can also provide UnifiedPush service. It is so good at this job that it has for some time been the UnifiedPush implementation recommended for most people. One can install the ntfy app and it provides UnifiedPush service to our other apps.

When we install the ntfy app ourself, we may either let it use the creator's default server (at ntfy.sh) or configure it to use our choice of ntfy server, if we know of one or if we run our own. As with anything, using someone else's server comes with conditions and restrictions, especially if we don't pay for the service, so we need to think about that.

Let's see what /e/OS does.

If not Google's then Whose Server?

In this push service built in to /e/OS, there is no configuration UI. There is a tiny settings UI where we have to “enable” the service — see below. Technically, the full ntfy UI is also available, but not accessible through the normal system UI — neither the app launcher nor the settings. But as curious techies we can hack in to it if we like. See below.

What we can do is install and use the UP-example app. Click the “REGISTER” button, and we discover it's preconfigured to connect to... . o O (their own ntfy server?) No! It connects to ntfy.sh, the server run by the author of ntfy. Hmm.

I should say that I found this while not signed in to Murena's “e cloud” services. It's possible and would make some sense if it would point to a different server if signed in. I will be interested to find out.

The first thing to comment on, then, is Murena will surely need to run their own push server, for privacy and scaling and economic reasons, or else make an agreement with ntfy. In the short term, to get things started, it may be acceptable for everyone to be subject to the free (gratis) ntfy service's rate limits and other conditions. That's something that can be changed later, seamlessly, because UnifiedPush doesn't require app (server) developers to know or care which service provider they will eventually be connecting to.

Secondly, for privacy and self-determination reasons, there should be a way to use our own choice of server, but not necessarily as a dedicated configuration option. A better way might be to tie it to the main Nextcloud account through a single-sign-on, so that when we set up the phone we have only to make that one choice of service provider and expect it to cover all services. That would give people the advantage of simplicity which is how the Big Techs play that game, but now with open standard technologies enabling us to have a choice among service providers.

When an app registers for UnifiedPush service, if more than one distributor is installed, it asks which to use. A normal user would have only one, and would not see this. Here as an experimentor and developer I already had both ntfy and NextPush installed:

It's not ideal that it identifies as “ntfy”, exactly the same as the official ntfy app. I expect this will soon be changed to /e/OS branding.

The tiny settings UI is found a the bottom of the system settings menu.

It's disabled by default. In this state, the UnifiedPush service is still advertised to apps, and they can register, but push messages are not delivered. (I suspect this isn't how we want the disabled state to be. Probably it should not advertise as available.)

That's all the configuration we get in this version of /e/OS.

(Aside: I couldn't find this in settings, to begin with. Searching in settings for “ntfy” or “push” or “unifiedpush” doesn't return any results, for me.)

Let's Hack In

As a developer we can access the ntfy UI through the system terminal, for example through an ADB shell (running this command on a USB-connected computer with “adb” tool installed, with ADB USB debugging enabled in the phone's developer settings):

adb shell am start -n foundation.e.ntfy/io.heckel.ntfy.ui.MainActivity

Just the same as in the user-installed ntfy app. (I can compare them side by side because I also have it installed, because I was already using it before this upgrade.)

We first notice this screen prominently shows some warnings about battery optimisation and choice of connection protocol. Are these warnings relevant to this version?, I wonder. (At first it showed only the first, and later both.)

And it lets us open its settings. In the ntfy settings we can choose our own server, and I can confirm this takes effect and works.

This screen also shows the currently subscribed topics — here, the “example” one — and after configuring it to use my own server, the displayed URL confirms that. (Generally one “topic” corresponds to one app.)

Ntfy Not NextPush?

I'm a little surprised Murena chose ntfy, seeing as their system is closely coupled to Nextcloud: there is also Nextpush (source on Codeberg, yay!), a Nextcloud-hosted UnifiedPush server module and its client app, that could be used instead. Perhaps they found ntfy is more stable or refined than Nextpush.

UnifiedPush developer S1m says the “Next version of NextPush will be a lot more stable, and I've rewritten the UI.”

The Future

While UnifiedPush is pretty good standard already, it is still maturing. Some important, recent improvements are making their way into the standard and into implementations like ntfy right now. These include full compatibility with the WebPush standard, and a better way to wake an app without it needing permission for unrestricted battery usage.

An adjacent recent development is DAV Push, an open standard for pushing instant updates to our shared files, calendars and contacts through CalDAV, CardDAV and WebDAV, which can also work through UnifiedPush.

To ensure people can exercise their right NOT to depend on Google, we have wider work to do. Integrating a UnifiedPush distributor is one useful step towards it. Beyond this, we need to be working with the developers of popular apps to get them supported, sharing this development across the freedom mobile OS ecosystem, deploying servers, ensuring reliability, and more. I write about this in a follow-up post: UnifiedPush — Wider Developments .

Conclusion

The details of this initial implementation seem to me to indicate an early preview release, with significant changes still needed. But that's OK: these technical implementation details can be changed.

Leaving that aside, the social side of this is amazing: it's making UP available to main-stream users. That's a big deal for the libre mobile ecosystem. Hurray!

Add UnifiedPush to /e/OS to make it possible for developers to avoid FCM and better support F-Droid applications — an /e/ community forum thread requesting UP support.
Source code of /e/ fork of ntfy: https://gitlab.e.foundation/e/os/ntfy-android
“Integrate ntfy in /e/OS” —merge request, including a simple settings UI (just enable/disable).
Romain Hunault of Murena is presenting it at Capitole du Libre 2024 this afternoon.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

Google-Free Push Messaging for Google-Free Phones

Mon, 04 Dec 2023 12:42:55 +0000

UnifiedPush open-standard push messaging complements degoogled android-compatible phone OS's such as LineageOS.

People who do not want to depend on Google or have them control our devices are using android-compatible but not google-controlled phones, a.k.a. “degoogled phones”. We have been asking (ourselves) for several years if we can have google-free push notifications. Thanks to the developers of the UnifiedPush standard, the answer is now, “yes!”

The open standard UnifiedPush.org has now been created. While not a large number yet, a useful handful of apps already support UnifiedPush, including several matrix and fediverse apps. For its servers and the associated client-side “distributor” component, there are multiple successful implementations deployed.

The current situation is such that anyone can use UnifiedPush on an android-compatible device by installing their choice of UnifiedPush distributor app (which must run in the background), configuring it to connect to their chosen U-P server (compatible with chosen distributor), and then installing any number of U-P-aware apps which will then use it (without needing per-app configuration to do so).

Why Would I Care?

Why would I care how my push notifications reach my phone? What difference does it make to me?

That's a good question. Inside a building that has a good heating, ventilation and air conditioning system, we don't notice the system, we just feel comfortable. With push notification delivery, part of the answer is the same: the system just does its job and our notifications come through. Whether the delivery channel is controlled by Google or by us or by someone else doesn't change that. The immediate, concrete result is the same. So it's not about wanting it to function differently; that's not why I care.

The difference it makes to me is about freedom, privacy, independence, self-agency. I am happy to have the choice to use any particular company's service, but I am not happy to be forced to use them, to have no choice, if I can't leave no matter how bad it gets. What if I don't like Google monitoring my notifications to know what I'm doing? What if I don't like to live in fear of offending them in some way and being cut off from their service and having no replacement option? What if I just don't want to condone their business model by using it, but I still want to be notified when I have messages?

Push notification delivery is one of the many invisible technical services that underpin our online communication systems. These kinds of services are implicitly considered to be part of the public infrastructure, something that we now assume is available to everyone.

When we allow ourselves to become dependent on any particular company's service, and yet do not regulate it as a public service provider, then we subject ourselves to the company's whims, priorities and values, which are different from ours. They will inevitably act against public interests.

Building publicly owned infrastructure based on open standards and freedom software is therefore essential to ensure the independent provision of services aligned with public needs and values.

I am one of the people who feels it is my place to use, promote and build non-proprietary public services, both for my own mental wellbeing and because I believe it is important for society.

It is the same reason why I support: open Ed-Tech, degoogled phones, #matrix, #fediverse, freedom software, open-source hardware, #rightToRepair.

Involving the OS ROM

In android-compatible OS ROM projects such as LineageOS, implementing some core support for the UnifiedPush.org standard now seems to me like the right way to go. Exactly what form of support is to be decided.

Some ways an OS like LineageOS could usefully be involved to improve the UnifiedPush experience are:

ensuring the U-P distributor app has a convenient way to be installed and permitted to run in the background, free from restrictions, because getting this right is critical and if the user installs the distributor manually it can be tricky to get right; (investigate: would it need to be a system app, or some kind of whitelisting (ugh), or be split into a system component and a user component, or what?)
providing a convenient way to let the user (or the OS distribution provider) configure the distributor's U-P server address: perhaps rather than using an ad-hoc UI provided by the distributor app, it could integrate with “accounts” settings.
potentially providing a system settings UI for monitoring the U-P connections and which apps are using them.

Thoughts on the role of microG. The purpose of microG as best I understand is to provide Google compatible APIs to apps which expect Google services. Underneath these APIs, it provides access to a mixture of actual Google services, alternative real services, and fake services. As far as I know it does not so far provide any non-Google APIs, and yet for push notifications the provision of UnifiedPush APIs might be a good fit for fulfilling its overall purpose as a compatibility layer. Or perhaps not, perhaps that is out of scope and should be in LineageOS or another add-on layer instead. I'm sure the folks involved will work out what is best.

Constraints, FCM Fallback, non-Android

Unlike the situation with some other google APIs, it is important to note that an OS compatibility layer such as microG cannot automatically divert the connections made by apps built using Google's FCM, to use U-P instead. The apps must be modified.

However, the inverse is possible: a UnifiedPush aware app can automatically “fall back” to using Google's FCM if U-P support is absent and FCM support is present. See details of the Embedded FCM Distributor in UnifiedPush documentation.

Non-Android devices can use UnifiedPush too, including Linux phones such as PinePhone and Purism Librem. The UnifiedPush D-Bus spec may be relevant. (On locked-down proprietary devices such as Apple's it is unlikely to be possible, nor to make much sense: FAQ.)

Packaging a UnifiedPush Distributor

A U-P distributor app could be built in to an OS or subsystem like microG but there is a significant down-side to that: it would support only one type, or at most a fixed small number of types, of U-P server. Choosing a distributor type is more of a whole OS packaging decision. In cases where the whole OS is related to a service provider of some kind (so not like LineageOS, but perhaps like Murena/Calyx/Graphene etc.), the service provider might choose to run a U-P server for their users and have their distributor automatically connect to it (with user consent/opt-in/opt-out). In the more generic/self-hosted case (like LineageOS) it makes more sense to leave it to the user to install their preferred U-P distributor.

I would love to see distributors of google-free phones, such as Murena, support google-free push notifications. I posted a brief sketch of a UnifiedPush Plan for Murena /e/-OS on their forum, without attempting to go into details of integrating the U-P distributor into the ROM.

History

A rough time line of UnifiedPush development. (From light research and having followed it through its development.)

2018: FOSS-Push planning
2019: OpenPush — defines the architectural concept
2020: UnifiedPush — building on OpenPush concept to a complete specification
2021: Gotify — a FOSS push system — was forked to add UnifiedPush support
2021: NextPush — NextCloud-based UnifiedPush system created
2022: ntfy — a FOSS push system — adds UnifiedPush support
now: more and more apps use UnifiedPush for google-free push support

Conclusion

Whatever the specifics of how any android-compatible OS ROM project might choose to proceed with google-free push support, the solution space enabled by UnifiedPush now exists. Speaking as one of the people who prefer our devices not to be controlled by and dependent on Google:

What do we want? UnifiedPush!

When do we want it? Now!

I would love to work on any freedom tech project bringing UnifiedPush to a wider audience.

See my other posts tagged... #unifiedPush #degoogled #awesomeFOSS

UnifiedPush notifications for your Matrix server with ntfy

Tue, 21 Jun 2022 11:00:00 +0000

I have created an installation role to add the UnifiedPush-compatible push-notification server “ntfy” to the popular Matrix server installer system “matrix-docker-ansible-deploy”.

This Ansible role named “matrix-ntfy” lets a Matrix server operator offer self-hosted Google-less push notifications for their users.

Compatible Matrix apps so far include Fluffychat and Schildichat. Announced in TWIM 2022-06-17 Element's Android app will soon be UnifiedPush-compatible too.

UnifiedPush

The ntfy server implements the UnifiedPush standard. Developed around 2021, this free and open source standard lets a user choose to receive push notifications through their own (or preferred) push server instead of through Google's FCM push-notification servers. To do so, a user first installs ntfy on their device and tells it where their preferred ntfy server is. Then any UnifiedPush-compatible matrix client apps, and indeed some non-matrix apps too, will automatically use it. This is particularly valuable for people who choose a de-Googled Android-compatible phone such as LineageOS or /e/OS, that is not connected to Google services.

Being an open standard, there are other UnifiedPush servers besides ntfy. And, while particularly useful with Android, it can also be used on other platforms.

On the server side

A matrix server such as Synapse will send push notifications via ntfy, for a given user/device/app combination, when the user's matrix client app asks it to do so. The matrix server then pushes those notifications to the ntfy server's built-in “matrix push gateway” API. The matrix server can continue to send push notifications for other users and other apps in other ways.

Development of the “matrix-ntfy” role

Developed at my trax.im, the role is now integrated in “matrix-docker-ansible-deploy” and being announced in TWIM, and in the project chat rooms mentioned below.

I am using it myself in my personal matrix server. If you are willing to use it or review it or test it and report back in the #matrix-docker-ansible-deploy:devture.com room, that would be very helpful.

Project web sites and their matrix chat rooms

the new “matrix-ntfy” role, in progress | (choose one of the project chat rooms below)
matrix-docker-ansible-deploy | #matrix-docker-ansible-deploy:devture.com
ntfy | #ntfy:matrix.org
UnifiedPush | #unifiedpush:matrix.org

ntfy — julian

UnifiedPush -- Wider Developments

How We Get There

OS Integration of the UP Distributor

Server Software

Server Hosting: Technical

Server Hosting: Social/Business Aspects

Apps

Sharing with other mobile OS projects

Standardising UP client-server protocol?

UnifiedPush: Standardise UP Client-Server Protocol?

UnifiedPush High Level Architecture

Multiple Protocols or Standardised Protocol

Going Google-Free with UnifiedPush in /e/OS

So What?

Technical

If not Google's then Whose Server?

Let's Hack In

Ntfy Not NextPush?

The Future

Conclusion

Google-Free Push Messaging for Google-Free Phones

Why Would I Care?

Involving the OS ROM

Constraints, FCM Fallback, non-Android

Packaging a UnifiedPush Distributor

History

Conclusion

UnifiedPush notifications for your Matrix server with ntfy

UnifiedPush

On the server side

Development of the “matrix-ntfy” role

Project web sites and their matrix chat rooms

Read More about UnifiedPush