Firefox for Android doesn't deliver every push notifications

[gyaaniguy]

NOTE: Please test in a least two browsers (i.e. Chrome and Firefox). This
helps with diagnosing problems quicker.

Please confirm the following:

• I have read the README entirely
• I have verified in the issues that my problem hasn't already been resolved

Setup

Please provide the following details, the more info you can provide the
better.

• Operating System:
  PHP 5.6.30
• web-push-php Version: latest as of this date

Please check that you have installed and enabled these PHP extensions :

YUP

Please select any browsers that you are experiencing problems with:

• Firefox for Mobile
  54.0.1 - android 4.2

Problem

firefox for android doesn't work with default setting. Works when $webPush->setAutomaticPadding(false); is set .

Errors with the default setting, Haven't checked other values, but the default fails with a 410 error

["message"]=>
string(427) "Client error: POST https://updates.push.services.mozilla.com/wpush/v2/gAAAAA..........M-yXA resulted in a 410 Gone response:
{"code": 410, "errno": 999, "error": "", "more_info": "http://autopush.readthedocs.io/en/latest/http.html#error-codes",

Expected

to return true.

Features Used

• VAPID Support

Example / Reproduce Case

function webPush_Register() {
$authPush = array(
'VAPID' => array(
'subject' => 'https://coinvalue.me',
'publicKey' => PUBLICKEY,
'privateKey' => PRIVVATEKEY
),
);
$webPush = new WebPush($authPush);
// $webPush->setAutomaticPadding(false); // disable automatic padding
return $webPush;
}

Other

Thanks for the library :)

[Minishlink]

Hi, thanks for your feedback. :)

The value indicated here might be outdated. Maybe try with setAutomaticPadding(X), where X < 3052?

[gyaaniguy]

setAutomaticPadding(3050) - Returns same error

setAutomaticPadding(2000) - Sends notification. No errors.

In my case I can keep it disabled. so no issues for me.

[Minishlink]

With hindsight, I don't think the length of automatic padding is wrong here. You would have received a 413 error. It looks like a sync problem through the GCM bridge (maybe when payload is too long it takes too much time??).
Another dev had a similar issue, see #90. But it seems his bug was resolved.

Let's ping @jrconlin, he may have more insight :)

[deb00t]

Hi guys, just a bump from me. And maybe 5 cents in addition.
I'm actually not using this library, but stumbled across the same problem.
Can't get notifications for Firefox Android to work either - ending up with the same code combination - 410/999

However, there's an error message saying "GCM recipient not available."

wpush v1 url-based tests and examples seem to work both desktop and android. but current FxA and FxA beta both send v2 endpoints to our server, when registering. Same for desktop Fx. HOWEVER, desktop Fx successfully receives server-pushed messages, the android version doesn't.

So there's obviously a problem with automatic bridging of mozilla webpush to GCM/FCM (battery/optimization reasons). I wasn't able to find out, how to configure the request to https://updates.push.services.mozilla.com/wpush/v2/... to receive newly registered FCM ID, or something.

Anyone an idea how to tell Mozilla's server, which GCM it should use?

[jrconlin]

Providing a bit more feedback on this after being contacted this morning.

In short, we weren't able to reproduce the same 410 error that deb00t was seeing, however we did notice some very odd behaviors. Not sure why, but GCM was being unusually slow to deliver messages (as in 4-10 minutes to complete the Webpush QA Page). GCM may be having issues that could be percolating back to our WebPush service.

Granted, slowness like that should not cause the message to fail like what was being reported, but that also comes from GCM (the reply failed items list).

There's not been a change to our server code in quite some time. We'll keep an eye on things, but for now it just may be that sending messages to android apps may not be quite as zippy.

[chrisdeeming]

Just wanted to let you know that the issue as described in the first comment seems to still exist. If I can be of any assistance in helping to debug this @jrconlin or @Minishlink then happy to help.

If the notification is pushed to a Mozilla endpoint subscribed to via macOS, it works fine. If it's a Mozilla endpoint and Android (7) then it fails with a 410 gone err 999 message.

We're working around it by disabling automatic padding for any Mozilla endpoint right now which makes it work for Android:

$webPush->setAutomaticPadding(0);

Which is ok, but obviously we'd rather keep the padding if possible.

[tino-codes]

Same here. Firefox on Android won't work without

$webPush->setAutomaticPadding(0);

[chrisdeeming]

Sorry for the nudge @jrconlin and @Minishlink but this issue still needs attention. Let me know if I can be of assistance.

[jrconlin]

Going to reopen this to try a patch.

[jrconlin]

I have a guess as to what the problem might be, but I'd appreciate some confirmation before I arbitrarily pick a fix.

In order for messages to get to android, we have to transcode them to GCM format. GCM imposes it's own set of limits on how big a message can be. The max size I calculated we could send was 3070 octets (that's the encrypted message size, not the message body size). To be safe, I'm presuming that @Minishlink trimmed it down to 3052. Since this is a complex system made of many moving parts, things may have changed somewhere reducing the message size further.

How much further is a bit of an unknown. I'm a bit bandwidth constrained right now, so I don't have a lot of time to set up a test PHP rig and keep poking the system to determine the max size that things will allow. I can either pick a new semi-arbitrary message length (3000? 2048?) or I might ask y'all if there's a value that you find works.

Thanks!

attn: @chrisdeeming, @tino-codes

[chrisdeeming]

@jrconlin I can test this fairly easily.

If you can let me know where in the code I can trial and error some different limits, I'll let you know.

[jrconlin]

@chrisdeeming Looks like you can just use the setAutomaticPadding function before you encrypt the data. So, instead of $webpush->setAutomaticPadding(0), you'd call something like

$webpush->setAutomaticPadding(3000); $webpush->sendNotification(...)

And see if that gets through.

(heh, or possibly set up a loop to try various reductions on that and let me know what gets through.)

[chrisdeeming]

@jrconlin sorry only just had a chance to test this.

Looks like a padding value of 2848 is when Firefox stops receiving the notifications. 2847 is fine in my testing though of course I appreciate you might want to trim it down a bit further.

After all that, just realised that my version of Firefox is outdated so I'd better test again in case anything has changed on their side...

EDIT: Yep, 2848 still breaks it.

[AroundtheGlobe]

@jrconlin I tested it in Firefox 61 for mobile and I get the same results. With the option: $webPush->setAutomaticPadding(0); it seems to work fine without it the push message never arrives.

[fvildoso]

$webPush->setAutomaticPadding(0);
that's not working for me on android

[geetpurwar]

$webPush->setAutomaticPadding(0);
Thats working for me.

[Honzaik]

Bump. This issue is still happening. Firefox 68.1 for Android
I suppose it is because of this mozilla-services/autopush#748

[mpw-wwu]

Can the zero padding become default behaviour? Are there any disadvantages?

+1 for the issue and the workaround here.

[jrconlin]

Padding is a cryptographic technique to obscure the content of what's being sent. As a super simple example, if I were to send you a message asking if you wanted to go to "the park", "the cafe" or "the monkey knife fighting pit" and you sent back a string that was around 30 characters long, I might guess you have a preference for simian slicing even if I can't read the message directly.

Ideally padding should make the message length as consistent as possible. One way to do that is to use padding that ensures that messages are the maximum allowable length, since good padding should trim to just fill in everything between the actual message length and the size you're specifying.

Where things get complicated is in the fact that Autopush (the server we run to handle web push) has to talk to the various mobile push networks in order to deliver the messages (e.g. Google's Fire Cloud Messaging (FCM)). These have their own limits on message size and format. Cryptography increases the size of the message, as does Base64 encoding (which is required by FCM), as well as the envelope we have to wrap things in for delivery. All of those impact the total size you can send. Autopush does try to help and returns a 413 error if it thinks the message might be too big.

So, what are viable solutions?

1. Check to make sure that the padding function is doing the right thing. Padded messages should all come out about the same length so an encoded "hello" should be about the same length as "greetings and felicitations to all who read this".

2. It may just be that 2847 is the correct padding length and that's the default that should be used for body content. As an added bonus, it means that the max message size you send out to android clients will be a bit short of that limit as well (2832 or so, see above for reasons).

[mpw-wwu]

Thank you for that well done explanation. Makes sense. In my case, encryption is of little importance, so I'll just disable padding if ”mozilla“ is part of the endpoint.

[jrconlin]

I'd caution that encryption is of little importance to you, however you may not be able to say the same about your customers or how they're using the info you're providing. It may be worth adding even a simple padding length of 1000 to cover the majority of short messages. Of course, use the padding that you find works for you, but having seen how metadata can be used in fun, abusive ways in the past, I'd be more comfortable if you didn't make it easier for the bad guys.

[mpw-wwu]

It will still be encrypted. Just the length is exposed - as I learned today 😀. And only to Firefox users like me.

[Minishlink]

Hello, in your experience what would be the most precise default setting for padding? 2847 ?

[jrconlin]

Not sure I follow, but padding is basically adding garbage to the end of a given message so that anyone who is trying to do heuristic analysis on what messages are being delivered can't determine the type of message based on it's content length.
There are two ways to add padding, one is to add randomly sized data, the other is to ensure that all messages are the same size (e.g. if you've decided that all messages are to be 3000 octets (roughly that number of characters), and your actual messages is 1024, then you'd add 3000 - 1024 = 1976 bytes of padding. (Padding can occur either inside the encrypted block or added to the post encrypted block depending on the encryption method and one's trust in the encryption methodology.)

Picking a static padding value is probably not a good idea.

[chrisdeeming]

Six years on, we still set automatic padding to 0 for Mozilla endpoints. Though it looks like we also had to do the same for ucweb.com endpoints (UC Browser):

	protected function getEndpointPadding($endpoint): int
	{
		if (strpos($endpoint, 'mozilla') !== false)
		{
			// firefox, at least on Android, has an issue with automatic padding which
			// is used to make encryption more secure, but makes encryption slower.
			// see: https://github.com/web-push-libs/web-push-php/issues/108
			// TODO: ideally won't need this forever if Mozilla ever fix this or if library works-around it
			return 0;
		}

		if (strpos($endpoint, '.ucweb.com') !== false)
		{
			// See https://xenforo.com/community/threads/158252/
			return 0;
		}

		return Encryption::MAX_COMPATIBILITY_PAYLOAD_LENGTH;
	}

Not tested in a long while whether it's still an issue though to be honest.

[Minishlink]

@jrconlin Sure, my point was about the default in the library. I feel like the library should have the default setting that is the most secure while still retaining compatibility with every push servers. Today, we set it to 3052, because that was the best default at the time to handle Firefox for Android. I was wondering if users have found better defaults (since I saw in a couple of issues that 3052 is too long sometimes for Firefox).
Oh maybe you were confused by the naming, "setAutomaticPadding" would be better named "setMaxLengthForAutomaticPadding". 3052 is the static size of all messages, not the added padding.

@chrisdeeming Interesting, thanks! We could have a more complex default based on the push server I guess. Or provide it as an option or in the README.

[markusramsak]

I tested with Firefox on my Android phone and got the following results:

$webPush->setAutomaticPadding(2820); // still works
$webPush->setAutomaticPadding(2821); // doesn't work anymore (413 Payload Too Large)

[fvildoso]

I tested with Firefox on my Android phone and got the following results:

$webPush->setAutomaticPadding(2820); // still works
$webPush->setAutomaticPadding(2821); // doesn't work anymore (413 Payload Too Large)

do you think that I should change $webPush->setAutomaticPadding(false); to $webPush->setAutomaticPadding(2820); ?

[markusramsak]

if your encrypted string has always the same length and is longer (padded with null bytes) it is more secure therefore yes.