degoogled — julian

UnifiedPush -- Wider Developments

Wed, 27 Nov 2024 14:48:00 +0000

An article about Building the Self-Agency Mobile Ecosystem: Push Messaging

Push messaging is the system that enables incoming messages to wake up and reach our apps, instantly and efficiently. But Who Cares Who Delivers Our Notifications? Our answer is: we care, and we do not like being locked in to depending on one mega-corp's system.

Therefore, in our libre mobile computing devices, we require a push-messaging infrastructure built from open standard technology that gives us freedom to choose our service providers and authority over them.

How We Get There

How can we build this? What is needed to get widespread UnifiedPush support in /e/ and other mobile OS?

The big picture involves several areas of work.

OS integration of the distributor, including how it's configured, auto-detected, optimised, etc.
Server Software: Making sure a suitable server-side component is available, supported, optimised for this use case.
Server hosting: making sure a server component is easily deployable by others, including making generic deployment methods (Debian, Docker, Nix, K8s, ...) and integrating it in some hosting systems.
Apps: Working with other apps to make sure they connect and work with UP out-of-the-box.
Sharing this work among other mobile OS projects (microG, CalyxOS, GrapheneOS, PostmarketOS, ...) .
Consider standardising a UP client-server protocol so that a built-in distributor can work with several different server implementations and vice-versa.

(Links in this and the following sections lead to relevant issues or discussions, where I could find any.)

OS Integration of the UP Distributor

Including how the UP distributor is configured, auto-detected, optimised, Android permissions, etc.

I recently analysed how /e/OS 2.5 has started the ball rolling

Server Software

Making sure a suitable server-side component is available, supported, optimised for this use case

Nextcloud: UnifiedPush Provider — exists
It could alternatively begin with a fork of ntfy-server for example.

Server Hosting: Technical

Making sure a server component is easily deployable by others, including making generic deployment methods (Debian, Docker, Nix, K8s, ...) and integrating it in some hosting systems, ranging from hobbyist to serious commercial systems.

Docker ntfy — exists
K8s ntfy helm-chart — exists
Nextcloud: UnifiedPush Provider — exists
Matrix-Docker-Ansible-Deploy: ntfy-server — exists
Yunohost: ntfy-server — exists
SelfHostBlocks — not yet
CoopCloud — not yet
Fediversity (an upcoming hosting project) — not yet

Work out and write up (shareable) policies about who hosts, terms and conditions, privacy, rate limiting and other special measures, etc.?

Apps

Working with other apps to make sure they connect and work with UP out-of-the-box.

Category 1: default apps in /e/OS etc., like...

Mail: K9/Thunderbird-andriod #5165 and UnifiedPush for IMAP (NGI proposal 2024-10-535)
Nextcloud -uppush #17 -notifications #1225 -android #8684
Accounts/Calendar/Contacts (DAVx5 #36 + DAV-push)

Category 2: other popular apps like...

HomeAssistant #446344
Fluffychat — exists
Element — exists
Signal forks
Telegram-FOSS #577
Conversations.im — exists
Pixelfed #86

Standardising UP client-server protocol?

Should we consider standardising the UP client-server protocol? At present, by design, one must choose a UP distributor and UP server together as a pair. If we standardised the protocol between them, so any distributor would work with any server, this would certainly have some benefits, but also down-sides, and is not necessarily the right thing to do. UP was designed this way on purpose and achieves its goals without needing to standardise there. No ordinary person needs to be able to change their UP implementation. Only techies and service providers might like to do that.

Therefore this is an exploration rather than a development. I discuss this further in a separate article.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

Follow/Feedback/Contact: RSS feed · Fedi follow this blog: @julian@wrily.foad.me.uk · matrix me · Fedi follow me · email me · julian.foad.me.uk Donate: via Liberapay All posts © Julian Foad and licensed CC-BY-ND except quotes, translations, or where stated otherwise

UnifiedPush: Standardise UP Client-Server Protocol?

Tue, 26 Nov 2024 20:00:00 +0000

This is a technical article. For a more general introduction to UnifiedPush you might read the first half of Going Google-Free with UnifiedPush in /e/OS or my other articles about UnifiedPush .

Should we consider standardising a UP client-server protocol so that a built-in distributor can work with several different server implementations and vice-versa?

Not necessarily, and I will attempt to explain why.

UnifiedPush High Level Architecture

Look at the diagrams on the unifiedpush.org home page. There are four connections involved, shown arranged in a square. UP standardises the protocol for one of the connections, the bottom one (UP distributor <–> app). It also standardises something about how the whole system works.

The protocol on the top connection (App server –> UP server) has recently been upgraded to be compatible with WebPush.

The other two (App <–> app-server and UP <–> UP-server) are two other, separate, independent protocols. The protocol the app developer uses for app <–> app-server is not relevant to UnifiedPush. At present, UnifiedPush standards also do not specify a protocol to be used for the connection between the UP server and the UP distributor (shown on the left).

The ntfy distributor requires a ntfy server, whereas the nextpush distributor requires a nextpush server, and so on. They use different protocols for that link in the chain.

Multiple Protocols or Standardised Protocol

It would be possible to make a UP server that supports (for example) both the NextPush and ntfy protocol(s). In fact ntfy supports two different protocols (the “json” one and the “websocket” one) and you choose one in the settings. The ntfy server supports both of those.

The point is, UP achieves its goals without needing to standardise there. In some respects it's beneficial that the user and/or service provider can choose those components and protocol that best suit their needs, while the end-user apps and app servers all still remain completely compatible (standardised) no matter what UP components are used.

Additionally standardising the UP <–> UP-server protocol could make different UP servers and distributors interoperable. It could potentially help with users bringing their own choice of device (assuming it has a UP distributor built in) from one service provider to another, for example.

But that's more of a deployment-level standard: something that service providers might want to standardise. Android-compatible mobile OS's (for example) might get together and declare “we” (as a loose group) like to standardise on (let's say) ntfy-websocket protocol here, because (suppose) it works best with android's OS network layer and sleep optimisations. (Just examples; I have not researched what protocol actually works best in each scenario.) Then, Linux-based mobile OS's might choose a different protocol that works better with Linux's networking and sleep optimisations. Then, if a certain libre mobile service provider wanted to support both android-compatible and linux-based devices, they might choose to support two protocols. Or would standardising on one protocol be more helpful in such a scenario? It's not clear.

No ordinary person needs to be able to change their UP implementation. Only techies and service providers might like to do that.

So, at this stage this is perhaps the least important of all the areas to work on.

First explained in #unifiedpush:matrix.org.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

Going Google-Free with UnifiedPush in /e/OS

Tue, 19 Nov 2024 12:00:00 +0000

Congratulations UnifiedPush! Congratulations Murena!

Murena's /e/OS 2.5 ships with UnifiedPush support included as announced by a small note in the 2.5-t release notes .

This exciting development brings Google-free push messaging to the regular users of an important player in the freedom mobile OS space.

So What?

Push messaging is the system that enables incoming messages to wake up and reach our apps, instantly and efficiently.

To achieve this, the phone operating system keeps a single network connection open to a push messaging server. The apps don't need to keep running in the background: they can go to sleep. When a push message arrives, the OS wakes up the relevant app and hands it the message.

All our apps share this push service — even the “private” ones — if they want push messaging.

That's great, except... Who Cares Who Delivers Our Notifications?

Until now, most people's phones run Google or Apple software. The Big Techs operate and control those phones. Along with everything else, they provide the push messaging service. And so they, Big Tech, control it.

The UnifiedPush public open standard changes this. With UnifiedPush, we are no longer merely “users” of our devices but are in charge of them. With UnifiedPush, we can choose which service provider will deliver our push messaging. Thereby we are in control of who has the ability to monitor or block our push messaging.

UnifiedPush has been available to those interested, for a while now, but until now it has required installing extra software.

Putting the support for push messaging into the operating system means ~~users~~ people who choose not to be operated by Big Tech will now be able to simply install the apps they care about and immediately have the alternative system working for them.

Technical

I'm going to delve into the technical side of this initial release, because it's a big interest area of mine. If you are a regular user, I suggest at this stage you might concentrate on switching to /e/OS or another freedom mobile OS and expect this functionality will be maturing quickly from now on and soon be seamless. If you're an early adopter or interested in its development, read on.

I am trying it out. I already run /e/OS on my main phone and upgrading to version 2.5 was simple with the built-in upgrader.

The OS embeds a custom fork of ntfy, calling itself “foundation.e.ntfy 1.17.0”. So, congratulations also to ntfy! At this stage the developer of ntfy, Philip Heckel, told me in #ntfy room he was “not yet involved” in this development and this is the “First I'm hearing about it.”

ntfy is a notification system that can also provide UnifiedPush service. It is so good at this job that it has for some time been the UnifiedPush implementation recommended for most people. One can install the ntfy app and it provides UnifiedPush service to our other apps.

When we install the ntfy app ourself, we may either let it use the creator's default server (at ntfy.sh) or configure it to use our choice of ntfy server, if we know of one or if we run our own. As with anything, using someone else's server comes with conditions and restrictions, especially if we don't pay for the service, so we need to think about that.

Let's see what /e/OS does.

If not Google's then Whose Server?

In this push service built in to /e/OS, there is no configuration UI. There is a tiny settings UI where we have to “enable” the service — see below. Technically, the full ntfy UI is also available, but not accessible through the normal system UI — neither the app launcher nor the settings. But as curious techies we can hack in to it if we like. See below.

What we can do is install and use the UP-example app. Click the “REGISTER” button, and we discover it's preconfigured to connect to... . o O (their own ntfy server?) No! It connects to ntfy.sh, the server run by the author of ntfy. Hmm.

I should say that I found this while not signed in to Murena's “e cloud” services. It's possible and would make some sense if it would point to a different server if signed in. I will be interested to find out.

The first thing to comment on, then, is Murena will surely need to run their own push server, for privacy and scaling and economic reasons, or else make an agreement with ntfy. In the short term, to get things started, it may be acceptable for everyone to be subject to the free (gratis) ntfy service's rate limits and other conditions. That's something that can be changed later, seamlessly, because UnifiedPush doesn't require app (server) developers to know or care which service provider they will eventually be connecting to.

Secondly, for privacy and self-determination reasons, there should be a way to use our own choice of server, but not necessarily as a dedicated configuration option. A better way might be to tie it to the main Nextcloud account through a single-sign-on, so that when we set up the phone we have only to make that one choice of service provider and expect it to cover all services. That would give people the advantage of simplicity which is how the Big Techs play that game, but now with open standard technologies enabling us to have a choice among service providers.

When an app registers for UnifiedPush service, if more than one distributor is installed, it asks which to use. A normal user would have only one, and would not see this. Here as an experimentor and developer I already had both ntfy and NextPush installed:

It's not ideal that it identifies as “ntfy”, exactly the same as the official ntfy app. I expect this will soon be changed to /e/OS branding.

The tiny settings UI is found a the bottom of the system settings menu.

It's disabled by default. In this state, the UnifiedPush service is still advertised to apps, and they can register, but push messages are not delivered. (I suspect this isn't how we want the disabled state to be. Probably it should not advertise as available.)

That's all the configuration we get in this version of /e/OS.

(Aside: I couldn't find this in settings, to begin with. Searching in settings for “ntfy” or “push” or “unifiedpush” doesn't return any results, for me.)

Let's Hack In

As a developer we can access the ntfy UI through the system terminal, for example through an ADB shell (running this command on a USB-connected computer with “adb” tool installed, with ADB USB debugging enabled in the phone's developer settings):

adb shell am start -n foundation.e.ntfy/io.heckel.ntfy.ui.MainActivity

Just the same as in the user-installed ntfy app. (I can compare them side by side because I also have it installed, because I was already using it before this upgrade.)

We first notice this screen prominently shows some warnings about battery optimisation and choice of connection protocol. Are these warnings relevant to this version?, I wonder. (At first it showed only the first, and later both.)

And it lets us open its settings. In the ntfy settings we can choose our own server, and I can confirm this takes effect and works.

This screen also shows the currently subscribed topics — here, the “example” one — and after configuring it to use my own server, the displayed URL confirms that. (Generally one “topic” corresponds to one app.)

Ntfy Not NextPush?

I'm a little surprised Murena chose ntfy, seeing as their system is closely coupled to Nextcloud: there is also Nextpush (source on Codeberg, yay!), a Nextcloud-hosted UnifiedPush server module and its client app, that could be used instead. Perhaps they found ntfy is more stable or refined than Nextpush.

UnifiedPush developer S1m says the “Next version of NextPush will be a lot more stable, and I've rewritten the UI.”

The Future

While UnifiedPush is pretty good standard already, it is still maturing. Some important, recent improvements are making their way into the standard and into implementations like ntfy right now. These include full compatibility with the WebPush standard, and a better way to wake an app without it needing permission for unrestricted battery usage.

An adjacent recent development is DAV Push, an open standard for pushing instant updates to our shared files, calendars and contacts through CalDAV, CardDAV and WebDAV, which can also work through UnifiedPush.

To ensure people can exercise their right NOT to depend on Google, we have wider work to do. Integrating a UnifiedPush distributor is one useful step towards it. Beyond this, we need to be working with the developers of popular apps to get them supported, sharing this development across the freedom mobile OS ecosystem, deploying servers, ensuring reliability, and more. I write about this in a follow-up post: UnifiedPush — Wider Developments .

Conclusion

The details of this initial implementation seem to me to indicate an early preview release, with significant changes still needed. But that's OK: these technical implementation details can be changed.

Leaving that aside, the social side of this is amazing: it's making UP available to main-stream users. That's a big deal for the libre mobile ecosystem. Hurray!

Add UnifiedPush to /e/OS to make it possible for developers to avoid FCM and better support F-Droid applications — an /e/ community forum thread requesting UP support.
Source code of /e/ fork of ntfy: https://gitlab.e.foundation/e/os/ntfy-android
“Integrate ntfy in /e/OS” —merge request, including a simple settings UI (just enable/disable).
Romain Hunault of Murena is presenting it at Capitole du Libre 2024 this afternoon.

#unifiedPush #degoogled #awesomeFOSS #eOS #Murena #ntfy #mobiFree

My Freedom Phone Cover Design

Fri, 20 Sep 2024 12:00:00 +0000

I designed a back cover to show people mine is no ordinary phone, it's a Freedom Phone.

Share and re-use the design if you like. Replace my face and website with your own.

Fits a OnePlus-6 phone, when printed at 76 x 152 mm. (This phone model can run a wide range of freedom software.)

Paired with the lock screen I designed last year...


Lock Screen	Back Cover

Concept Development

Began as a paper sketch, 2024-09-15.


1	2	3	4	5	to share

Finished in time for Software Freedom Day 2024-09-21.

Licence and Credits

The final design to share:

Download: freedom-phone-back-cover-julian-foad-2024-v5.0.svg
SVG format (Inkscape flavour)
Excludes my face and website
Licensed as Creative Commons CC-BY-SA 4.0 “share-alike” with attribution required
Copyright © Julian Foad 2024

Created using Inkscape (freedom software)

Attributions for art work used:

“Ada and Zangemann” by David Revoy, 2023, based on “Ada & Zangemann” written by Matthias Kirschner and illustrated by Sandra Brandstätter − CC-BY-SA 4.0
rainbow umbrella shared by Lisa Cornell, 2011 − public domain
bitten apple shared by Dan Gerhards, 2011 − public domain
type fonts: Liberation Sans, cmmi10
image of my own face, shown in derivative examples here, © Julian Foad, 2016

#fossGadgets #degoogled #lineageOS #eOS

Immich -- Awesome Open Source

Thu, 12 Sep 2024 10:00:00 +0000

In our family we look after our own photos — we don't want Google or any other company deciding what we can and can't do with them.

In the past we used various desktop/laptop based open source viewer software, with storage on local disks. More recently we have been running the awesome open source PhotoPrism, with its smartphone-compatible web interface and photo library management features. Although PhotoPrism is impressive judged on its own merits, and has indeed allowed us to manage our photos ourselves, it's just not quite as usable as we'd wish.

Next up is Immich.

Immich promises “partner sharing”, where for example husband and wife might choose to share their whole photo libraries with each other. Immich offers stronger photo library management such as tagging, metadata manipulation, and deduplication of identical photos. Immich performs face recognition. Immich viewer apps seem to be user friendly and slick.

What's Good, What's Not So Good

For viewer apps we use the official Immich app from F-Droid on our degoogled phones and tablet, and the Immich web interface which works well on both mobile and desktop. The general impression is slick. It's fun to see a selection from “N years ago”. It's nice that various features of the viewer, such as the side-bar options to view by folder and view by people, can be turned on or off in the settings, depending on how we prefer to use the system. The main “time line” view works very well. Other views such as the view by people, by tags, on a map, and so on are functional but much less developed.

The user interface could be improved a lot by allowing us to use the different selection criteria together: having begun to explore a region on the map, we then want to narrow the results by people or by tags. As it stands, the interface is an example of the all-too-common anti-pattern in which every time we select a new option the computer “forgets” what we just told it a moment ago. (Nearly every shopping web site is a dire example of that anti-pattern. We might filter our results by “category=hardware” and by “material=metal”, and then adjust our search terms, and watch the computer immediately throw away our carefully chosen filters.)

Face Recognition

Face recognition is impressive. I set Immich to scanning my existing library of thousands or tens of thousands of photos dating back twenty years, which took it a few hours on my few-years-old server. Immich created groups of faces that the algorithm considered to be the same person. These groupings were amazingly accurate. The few false matches were predominantly small blurry faces picked out of the background (but still the majority of these appeared accurate), or baby faces (where to be fair I can't tell the difference even among members of my own family), or where only a small part of a face could be seen (and even many of these were accurate).

For naming the people, Immich presented a list of faces, one from each identified “person” group. The faces it chose were unfortunately often a small blurry image. It would be helpful if it could choose a “clear” image as the representative of each “person” group. On clicking this image we get to see all the photos in which that same person was identified, and there we can set the name of that person, and we can also choose a different representative image. This view of all the photos containing that person doesn't indicate where in the photo the person was detected, which in some cases puzzles me as they don't seem to be there at all, but perhaps knowing what the algorithm “saw” would just give me somewhere to direct my scorn and derision at its folly and might not really add anything helpful to the overall purpose.

An enthusiast has created a better user interface for naming and managing the faces more quickly, all on one screen. It's the main feature so far of Varun Raj Manoharan's Immich Power Tools, and the demo video there shows it in action.

The partner sharing works by including both my photos and my wife's photos in my time line view. The same for her view, if it is set up reciprocally. That is a good start. It would be more useful if we could see which photos are “ours” and which are “theirs”. I would like to see them tagged with our initials, or perhaps distinguished by a colour or styling. I would like to be able to quickly show or hide each set, and to select by partner in the search.

I hoped the partners would be able to share the same categorisation, tagging, face recognition and so on. At present unfortunately this is not possible. It is discussed as a potential future improvement in Show Face Recognition for Partner #5089.

Because the current implementation of Partner Sharing isn't close enough, I am thinking about using just one “family” account in Immich, with both (or all) partners' photo libraries in it. That might be the best way for our family to use the existing capability. Whether this would suit us depends on whether we use the upload/sync feature from our smartphones. If we do, then Immich's sync feature would place all of our photos into the same library. (I haven't found a feature to separate or tag them on upload based on which camera-phone they came from, although there might be a way.) Currently we are not using Immich's upload: we are using 'syncthing' for that, copying our photos directly to our respective photo libraries on the server. Then in Immich we define these as “external” libraries (meaning they are outside Immich's built-in upload storage area).

Folders, Albums, Tags

I have a collection of (older) photos that I already organised into folders with descriptive names. I would like these to be shown as “Albums” or “Tags”. Immich imports tags if found in embedded metadata, and a few of my photos had these. Due to the sad state of standardisation of photo metadata, however, naming the files and folders is a more universal solution than embedding descriptions and tags in the photo metadata.

The way I would expect or wish to get albums and tags from my collection would be to specify a set of patterns that match my folder and file names, and to specify rules to derive album names and tags from these patterns.

There are open issues about potential future development of folder imports in Immich. In the meantime I might try the extension Salvoxia/immich-folder-album-creator noted below.

Extensions

Extensions I have noted to investigate further:

– immich-folder-album-creator — “Automatically create and populate albums in Immich from a folder structure in external libraries” – Immich Power Tools — alternative UI for face naming and management – Immich-Kiosk — “A web slideshow for Immich”

Freedom Software Ethics

In terms of open source “freedom software” ethics, Immich sits somewhere in the middle of the spectrum.

On the positive side, Immich is open source under the GNU AGPL v3 license: that's very good for maintaining the freedoms granted to its users. Immich has its own website at its own domain, immich.app, where it hosts its blog and documentation.

On the other hand, like too many other good projects, the development is hosted in Microsoft Github rather than in a freedom-respecting forge such as Codeberg or at its own address. Use of MS Github is a real and present danger to the open source ecosystem: freedom software lives in freedom-respecting software forges. And development discussion is invited inside the proprietary Discord instead of through a freedom-respecting method such as Matrix: see Discord vs community values. Immich in its website footer also promotes other proprietary silos (Reddit, Youtube) and does not even mention any open/freedom tech channels such as the Fediverse.

Immich advertise that their client app is available through the big silo stores (Google, Apple). They also also distribute it through the freedom-respecting app store F-Droid, for Android and android-compatible degoogled phones. They mention this in the docs but not on the main page.

#awesomeFOSS #selfHosted #degoogled

WhoBIRD -- Awesome Open Source

Thu, 27 Jun 2024 08:19:16 +0000

I recently learned something new about those tweety birds we hear in the tall trees all along our school run, cycling or walking past the golf course. Near the school end, lots of coal tits, a name with which I was barely familiar. Near our end, lots of wrens. I had an idea that wrens were rare and secretive and tiny, so it had not occurred to me they were responsible for that noisy cheerful chatter. Perhaps it's hard to spot one but they're loudly singing.

These I learnt last week helped by an app named “whoBIRD” which, as you may guess, listens for bird calls and displays the most likely matches. It's delightful to identify real birds, and surprisingly successful if they're not too far away.

It's also fun trying to imitate bird calls, and satisfyingly hard to get past's WhoBIRD's correct identification of “human whistle”.

My eye was drawn to WhoBIRD when I stumbled across it while browsing for another open-source app in F-Droid. It is a little volunteer/hobbyist app which matches the sound from the microphone against Cornell University's bird call data, displaying a name and a picture of the most likely match. That's all it does.

Another family member has been using an app named “Merlin”, I soon found out, so I tried it too. Merlin is Cornell's own app, using the same data source. By contrast with WhoBird, Merlin seems aimed at serious amateur birders for research data collection, professionally produced, multi-featured and slick — and on the other hand being a huge download and demanding sign-up into their club. (For a quick try-out I found that entering the fake email address “a@b.c” on every start suffices to access the sound identification feature with just names but no pictures or additional information.)

It's nice to have the choice, which Cornell University enabled by publishing their research results as Open Data in their BirdNET project so that others can re-use it in other ways. Birders and ornithologists interested in the sightings and soundings side of this should consider Merlin or the other BirdNET apps. Technologists and researchers on the data analysis side might like to study and modify whoBIRD. As a builder and tinkerer myself, I value open source software on principle. Already I am imagining how I or anyone could modify whoBIRD to do things like send a picture of each identification to my smart watch. Or send the identifications through my Matrix network to co-ordinate the listening among multiple devices left near open windows at home, with a separate display standing on the kitchen table.

#awesomeFOSS #freedomSoftware #degoogled

Troubleshooting UnifiedPush

Fri, 21 Jun 2024 05:39:46 +0000

Let's take charge of another part of our digital life! Must we be beholden to Google for delivering our push notifications? No! No longer! With apps that support UnifiedPush neither Google nor the app makers get to determine where our data goes and who gets to monitor it. We get the control, the freedom to choose.

To me this feels like a breath of fresh air!

You might find your way here while installing an app that uses UnifiedPush. Let's see how to make sure the notifications work properly.

The official UnifiedPush Troubleshooting page is, at the time of this writing, rather bare. (Perhaps, as an open source project, I can contribute to it. EDIT: Here's a preview I have submitted on Codeberg, that's becoming better than this article.) Meanwhile, read on.

Following my earlier articles such as Who Cares Who Delivers Our Notifications? and how to self-host UnifiedPush notifications for your Matrix server with ntfy, here are some tips for using a UnifiedPush-compatible app on your Android phone or your freedom-supporting (degoogled) android-compatible.

Install a Distributor to Enable UnifiedPush

Some U-P Distributors: ntfy, NextPush, Conversations, ...

The first step to using UnifiedPush is to get a UnifiedPush “distributor” app running and configured. A distributor is the one app that will be listening all the time on your device, on behalf of all the apps that want to receive UnifiedPush notifications. When notifications arrive, it will distribute them to the correct target apps (usually messaging apps).

Ideally, in the near future, freedom phones would include a UnifiedPush distributor built-in. For now, you have to install one.

You choose your distributor app once, the first time you want to start using UnifiedPush. The distributor works in concert with one UnifiedPush server (again, your choice, depending on what the distributor supports), so consider which server or service provider you want to use. Install the distributor app, and configure it according to its documentation. Some are preconfigured to use one particular server, while others allow or require you to choose a server.

The distributor must be working before any U-P-compatible app will be able to use UnifiedPush. If no distributor is detected, most apps will choose an alternative method, usually either polling or the proprietary Google system. However, if a distributor is installed but not working properly, notifications may not arrive promptly and consistently or at all.

Follow the instructions for your chosen combination of server and distributor, or start from the UnifiedPush Distributors page.

Troubleshooting UnifiedPush

Now we assume you have installed a U-P-compatible client app, perhaps a messaging app, and are expecting it to use UnifiedPush.

The general expectation is the app should “just work”, without even telling you, if it detects a functioning UnifiedPush distributor as described above.

How does this magic work? Three steps. First, your app asks the distributor on the same device for the details of your chosen UnifiedPush service. Second, your app passes those details on to its own server (wherever that is). Third, even after your app has disconnected from its own server or “gone to sleep”, your app's server continues to use those details to push notifications to your UnifiedPush server, which delivers them to your distributor, which delivers them to your app, waking it up.

Let's see what we can check if it does not seem to “just work”.

Check your app is supported and the right variant

First check that the client app you are using supports UnifiedPush. The UnifiedPush web site lists some of the first U-P-compatible apps. Some apps come in different variants, with for instance the variant available through the Google Play store using Google's notification system while the variant available though the F-Droid freedom-software store uses UnifiedPush. Some apps give you a choice if they detect both methods are available, and some choose one method without asking you.

Use the simple UP-Example app

Install the UnifiedPush troubleshooting app UP-Example from F-Droid. Use it to manually test UnifiedPush registration and operation: simply press the “register” button and then the “test” button, and you should immediately receive an example notification named “Test”.

If this simple test fails, you need to debug your UnifiedPush distributor app and its configuration.

(TODO: see troubleshooting for specific distributors 'ntfy', 'Conversations', 'NextPush', etc.)

Use advanced notification troubleshooting tools

You can find more advanced notification troubleshooting tools in some apps.

Element-Android and Schildichat-Android have a useful, active, troubleshooting tool in “Settings –> Notifications –> Troubleshoot Notifications”. In particular the “Gateway” and “Endpoint” steps of that troubleshooter should display your ntfy server's URL that it has discovered from the ntfy client app.

Check also “Settings –> Notifications –> Notification Targets” in Element or SchildiChat, or “Settings –> Notifications –> Devices” in FluffyChat. There should be one entry for each matrix client app that has enabled push notifications, and when that client is using UnifiedPush you should see a URL that begins with your ntfy server's URL.

In the “Notification Targets” screen in Element or SchildiChat, two relevant URLs are shown, “push_key” and “Url”, and both should begin with your ntfy server's URL. If “push_key” shows your server but “Url” shows an external server such as up.schildi.chat then push notifications are being routed through that external server before they reach your ntfy server, which might or might not be what you want.

Using your own UnifiedPush server

If you are running your own UnifiedPush server, of course you should also look at its logs and debugging capabilities. That is out of scope for this article.

These tips should help you resolve any issues

For further help, contact the most relevant or specific of the projects involved, such as your client app provider or your U-P distributor provider, or for more general help contact the UnifiedPush project, through the links provided here. (Note: The blog comments are for discussing the blog post, not for getting help.)

Once it is working, enjoy one more Google-free, self-determined aspect of your digital life!

Odin and Heimdall: Free Your Samsung Android

Fri, 24 May 2024 15:00:00 +0000

This article is about one particular step of the procedure for “flashing” a new android-compatible operating system onto a Samsung device to create a Freedom-Respecting Smart Phone or Tablet.

(For the rest of the procedure, see for example the instructions to install e-OS or install LineageOS .)

Main article: Degoogling a Samsung Galaxy Tab A7 (2020)

Tools for Flashing Samsung: Odin and Heimdall

Samsung Android devices are tightly locked down. My impression is they perhaps allow re-flashing only to fulfil some legal concession. It does not seem to be an option that they want us to use. Nevertheless, it is possible. At least on some models. (I read that some models sold in the USA are completely locked; maybe there is no legal concession to allow changing the software there.)

Samsung do not seem to make available the tools needed to “flash” the software. The instructions I find all reference either “Odin” or “Heimdall”.

Odin is apparently a Samsung internal tool that has been leaked, many years ago, as a Windows-only executable with no source code. This, widely referenced as the way to go, is useless for non-customers of Microsoft-Windows. In 2022 a leaked Odin v4 for Linux was reported, also executable only. Without source code, nobody (outside Samsung) can reasonably make improvements or adaptations. (Also, many instructions including LineageOS's say to download the Odin executable from some unattested third-party web site, not even giving a checksum to verify. This is dangerous.)

Heimdall is a cross-platform open source tool developed independently. Heimdall can replace Odin, but many instructions either do not mention it at all, or point to an out-of-date version such as v1.4.2 that lacks important updates and bug fixes.

Both Odin and Heimdall work with Samsung's “download” mode, and are needed to get an alternative (freedom) operating system onto the device.

Heimdall, Open Source Samsung Flashing Tool (Odin Alternative)

An open-source flashing tool for Samsung devices, called Heimdall, was developed by Benjamin Dobell (https://glassechidna.com.au/heimdall/) around 2010 to 2017. That version is most commonly referenced.

Some bug fixes and updates have been made by other people since then, but not merged back to that version. Some of these can be found inside Microsoft's GitHub, but the best version is not there: it rightly lives on a Freedom software forge called Source Hut.

Heimdall up to v1.4.2, the latest “original” version (from Benjamin Dobell or “glassechidna”) at the time of this writing, did not work for me. See Troubleshooting: Heimdall Fails at Initialising Protocol.

Henrik Grimler now maintains a version of Heimdall at https://git.sr.ht/~grimler/Heimdall .

I recommend using Grimler's version. It has fixes and updates needed for more recent devices.

It is available as source code. You could ask around for someone to provide a build, or build it yourself following the included instructions.

Building Heimdall

Referring to Appendix B at the bottom of the instructions page for Linux, on Ubuntu 22.04, on which I already had most of the dependencies installed, I built Heimdall like this:

$ sudo apt install libusb-1.0-0-dev  # you may need more: see below
$ mkdir build
$ cd build
$ cmake -DCMAKE_BUILD_TYPE=Release -DDISABLE_FRONTEND=ON ..
$ make
$ # The executable is then found at `./bin/heimdall`
$ ./bin/heimdall version
v2.0.2

Required dependency packages are listed in the doc linked above, and also machine-readably in the source tree in the files .builds/{alpine.yml,archlinux.yml,ubuntu.yml}.

I used 'DISABLE_FRONTEND' because I was not interested in building a GUI, and could omit installing 'qtbase5-dev' and its several sub-dependencies.

Using Heimdall in place of Odin

To flash a recovery image, Odin wants to be given a “tar” file (a similar concept to a zip file) with the image file inside it. For example, we have to create recovery.tar which contains recovery.img. By contrast, Heimdall takes the image file directly, e.g. recovery.img.

Where instructions for Odin say to select the “AP” (Android Partition) button and choose the recovery.tar file, the Heimdall translation is to run heimdall flash --RECOVERY recovery.img . For flashing a different kind of image such as “vbmeta”, we write heimdall flash --VBMETA vbmeta.img. We can give more than one pair of command-line arguments to flash more than one image.

Referring to this XDA post, the command needed for this device is:

$ heimdall flash --VBMETA vbmeta.img --RECOVERY recovery.img

Or, if we prefer not to reboot immediately (equivalent of disabling Odin's “Auto-Reboot” option), add the '—no-reboot' flag:

$ heimdall flash --VBMETA vbmeta.img --RECOVERY recovery.img --no-reboot

#fossGadgets #android #degoogled #lineageOS #eOS

Troubleshooting: Heimdall Fails at Initialising Protocol

Fri, 24 May 2024 14:00:00 +0000

This is about flashing a custom operating system ROM in a Samsung Android device using the tool named Heimdall.

We may encounter a failure mode in which heimdall detect succeeds but heimdall flash and heimdall print-pit both fail, printing this:

Initialising connection...
Detecting device...
Claiming interface...
Setting up interface...

Initialising protocol...
ERROR: Failed to send handshake!ERROR: Protocol initialisation failed!

Releasing device interface...

Additional diagnostics if we give the --verbose option:

"ERROR: libusb error -7 whilst sending bulk transfer"

(This message is repeated several times, before the final failure.)

Errors like this have been reported many times: e.g. here and duplicates, here, here

Here are two scenarios in which this error can occur.

Scenario 1: Only The First Heimdall (Flash, Print-Pit) Command Works

Some people report that some Samsung devices only accept one Heimdall command in the bootloader mode. Further attempts fail. Rebooting is required before another command can succeed.

This may be a bootloader limitation in certain devices. (I have not yet seen a report with enough detail to satisfy me it is definitely the case.) In other cases, using Heimdall's options “—no-reboot” (on each command) and “—resume” (on subsequent commands, since v1.4.0) may overcome this.

(In this description, “heimdall flash” and “heimdall print-pit” count towards the limit of one command, while “heimdall detect” doesn't count towards the limit and can be used freely.)

Scenario 2: Every Heimdall (Flash, Print-Pit) Command Fails

This happens only when running Heimdall on certain linux distributions, notably Ubuntu.

Some people fixed this in their forks of Heimdall, e.g. Jesse Chan's fix is in Grimler's version of Heimdall and in several other forks. (This fix works by resetting the USB device on the Linux end. The main work in developing the fix seems to be recorded in PR #478 .)

My suggestion: build and use Grimler's Heimdall. I explain a bit about Heimdall and how to build and use it in my article Odin and Heimdall: Free Your Samsung Android .

Scenario 3: Heimdall With Resume Option Fails First Time

If the first Heimdall command after booting into download mode is given the '—resume' option, Heimdall fails for me (on Ubuntu 22.04) like this, even when using a version with the Scenario 2 bug fixed:

$ heimdall print-pit --resume --no-reboot --verbose
...
Beginning session...
ERROR: libusb error -7 whilst sending bulk transfer. Retrying...
...
ERROR: Failed to begin session!
Releasing device interface...

From what I can tell, this seems to be expected, and so a usage error. The '—resume' option should not be used on the first command.

#fossGadgets #android #degoogled #lineageOS #eOS

Freedom-Respecting Gadget: Degoogling a Samsung Tablet

Mon, 20 May 2024 19:00:00 +0000

This is about buying a Samsung Android tablet and replacing the privacy-invading proprietary Google and Samsung software with privacy-respecting Freedom Software: “degoogling” for short.

Why? In “The Problem” section below, we take a look back at how much we're giving up when we accept Google's and Samsung's terms.

I would like ordinary people, with a little technical skill, to be able to do this. The process unfortunately is currently far too difficult, especially so on Samsung devices.

My goals:

Install a privacy-preserving freedom-software operating system on my tablet
Make it easier for others to do the same

The tablet I bought:

Name: Samsung Galaxy Tab A7 10.4 2020 (Wi-Fi)
Model: SM-T500
Device code: gta4lwifi

Yes, it's a few years old already. One of the benefits of Freedom Software is the community at large has the option to maintain up-to-date software on older devices, for as long as anyone considers it worth the effort. This is in contrast with the original maker's vested interest in stopping updates of their proprietary software after a small number of years. A nod to the right-to-repair and sustainability movements.

The Problem

Have you bought a tablet or phone recently? Perhaps even a Samsung?

IF We are buying a new or second-hand tablet,

THEN We might look at a Samsung Android device. They do make nice hardware!

BUT Oh! The set-up screens! — Screen after screen of, Yes, I agree to let Them scan my house for other devices and report back anything they find... Yes, I agree to let Them install or change or delete anything on “my” device... Yes, I agree to let Them scan all my contacts and data so they can make me feel happy by feeding it back to me in different ways, no, wait, they mean mining it for profit and trapping me in their systems. Yes, Agree To All because that's the easy option (and notice there's no “Agree to None” option).

Aaargh! No way, not me. Unfriendly system, Be Gone!

THEREFORE We want to turn it into a Freedom-Respecting Smart Phone or Tablet by “flashing” a Freedom Software operating system onto the device.

The New Software

I want to install an android-compatible, privacy-friendly, Freedom-Tech operating system. Nowadays I favour e-OS by Murena. e-OS is near main-stream user-friendly, with a nice blend of privacy by default along with a choice of optional (privacy-invading) google support for those that want it. e-OS is available on a wide range of phone hardware from cheap to premium, and you can either buy it or self-install it. If you're not wanting to develop tech skills in this area, buy it pre-installed.

Android-compatible Freedom Tech Operating Systems

Android as most people know it is a proprietary product, which contains inside it some open-source software (including some of Linux), leading to eternal confusion about whether “it” itself is open-source. No. Only the people who go looking for it get an open-source system built around the Android Open Source Project (AOSP). (That's us, yay!) Everyone who buys an Android device from the usual sort of phone maker is getting a heavily proprietary product that spies on them.

For this reason I often use long-winded descriptions like “android-compatible freedom-tech operating systems” to clearly differentiate these from proprietary Android.

Murena's “e-OS” is built around AOSP, using Freedom Tech to replace all the proprietary Google pieces, in a similar way to other android-compatibles such as LineageOS (a minimal android-compatible base OS) and LineageOS-for-MicroG (LineageOS plus F-Droid and some optional google support).

See my Freedom-Respecting Smart Phone post for other Freedom Tech alternatives you might consider.

Choosing a Well Supported Tablet

I bought this tablet after checking it was supported by e-OS and by LineageOS (and therefore also by LineageOS-for-MicroG).

What I didn't realise until too late is the installation instructions say I should download and run this, that and the other from various “random” (unofficial/unverified) sources — and it turns out those instructions are intended for running on MS Windows. Ugh.

There is a Freedom software alternative for this step: Heimdall. See Odin and Heimdall: Free Your Samsung Android.

I found this part of the procedure to be a deep rabbit-hole and spent a long time getting through it, reading a lot of dead-ends along the way.

For this reason I want to document what works, and also make it easier for others to do it, by attempting to automate this procedure in the Open Android Installer.

As the procedure is similar for many devices so working on the solution for my device should, I hope, also help others.

On Choosing a Well Supported Tablet, if you have limited time and experience for working through a complex installation procedure, then I would advise only choosing a Samsung phone or tablet in the following cases:

if the Open Android Installer or e-OS Easy Installer supports it, or
if you can get one that is already running a Freedom Software OS.

(Once it is running a Freedom Software OS, it is easier to upgrade or change it from that point.)

Installing

Installing is a challenge. The procedure is fraught with difficulties. Some people report that they “simply” followed the instructions and succeeded within an hour, while many others apparently spent days, encountered problems or gave up.

The starting point is the semi-official instructions published by the likes of LineageOS and e-OS. Note that these are basically the best efforts so far of volunteer groups, and are not necessarily accurate or complete, and far from foolproof.

See:

e-OS install guide and e-OS forum thread
LineageOS install guide and XDA forum thread (also relevant for LOS-for-microG)

After many hours of trial and research I got LineageOS installed, and some notes taken, several improvements implemented in my local copy of Open Android Installer, and a first couple of blog posts posted about parts of the procedure.

After reaching that stage, it was relatively easy to replace plain LineageOS with LineageOS-for-MicroG (both at version 21).

I also tried replacing it with e-OS (version T equivalent to 20) but that failed:

Installing update...
Comparing TZ version XF.5.1-01015-1 to XF.5.1-638149-3
ERROR: This package requires firmware from an Android 12 based stock ROM build. Please upgrade firmware and retry!
ERROR:   recovery: Error in /sideload/package.zip (status 1)

The tablet was running Samsung Android version 12 before I started. Never mind that for now. I'll come back to e-OS later.

So, I can claim success on one score: my tablet is now running LineageOS-for-microG.

Oh! You wanted all the details of how I did it and what went wrong? I'm not planning to write that up in that form (it was messy), but rather to work out improvements and write them up. I'll post them here once I've got something.

Making it easier for others to do the same

Taking hours to achieve this feat just for myself does not make me feel good.

I want to make it feasible for people with just a little technical skill to install an alternative operating system. It really should not and need not be so hard.

I can:

improve the installer apps (Open Android Installer, e-OS Easy Installer)
improve the installation instructions

In particular I want to do things that help whole classes of problems, across multiple devices and multiple operating systems, such as making the installers and instructions more automatic, more consistent, more foolproof, easier to diagnose.

To these ends, I have been keeping notes on what's missing, and have started making improvements to Open Android Installer.

#fossGadgets #android #degoogled #lineageOS #eOS

degoogled — julian

UnifiedPush -- Wider Developments

How We Get There

OS Integration of the UP Distributor

Server Software

Server Hosting: Technical

Server Hosting: Social/Business Aspects

Apps

Sharing with other mobile OS projects

Standardising UP client-server protocol?

UnifiedPush: Standardise UP Client-Server Protocol?

UnifiedPush High Level Architecture

Multiple Protocols or Standardised Protocol

Going Google-Free with UnifiedPush in /e/OS

So What?

Technical

If not Google's then Whose Server?

Let's Hack In

Ntfy Not NextPush?

The Future

Conclusion

My Freedom Phone Cover Design

Concept Development

Licence and Credits

Immich -- Awesome Open Source

What's Good, What's Not So Good

Face Recognition

Partner Sharing

Folders, Albums, Tags

Extensions

Freedom Software Ethics

WhoBIRD -- Awesome Open Source

Troubleshooting UnifiedPush

Install a Distributor to Enable UnifiedPush

Troubleshooting UnifiedPush

Check your app is supported and the right variant

Use the simple UP-Example app

Use advanced notification troubleshooting tools

Using your own UnifiedPush server

These tips should help you resolve any issues

Read More about UnifiedPush

Odin and Heimdall: Free Your Samsung Android

Tools for Flashing Samsung: Odin and Heimdall

Heimdall, Open Source Samsung Flashing Tool (Odin Alternative)

Building Heimdall

Using Heimdall in place of Odin

Troubleshooting: Heimdall Fails at Initialising Protocol

Scenario 1: Only The First Heimdall (Flash, Print-Pit) Command Works

Scenario 2: Every Heimdall (Flash, Print-Pit) Command Fails

Scenario 3: Heimdall With Resume Option Fails First Time

Freedom-Respecting Gadget: Degoogling a Samsung Tablet

The Problem

The New Software

Android-compatible Freedom Tech Operating Systems

Choosing a Well Supported Tablet

Installing

Making it easier for others to do the same