Updating is under/incorrectly specified

[benfrancis]

The spec says that the user agent may "periodically check if the contents of a manifest has been modified (e.g., by honoring HTTP cache directives associated with the manifest or by checking for updates after the web application has been launched)."

Firstly, being able to periodically check the contents of the manifest requires that the manifest URL does not change. If the manifest URL changes, there's no way to know that any new manifest corresponds to the same app as the old manifest and the only way for the app's metadata to be updated is for the user to re-install the app.

The removal of the same-origin restriction on manifest URLs was intended mainly to allow CDNs to host the manifest on a separate origin to the app, but the above assumes that CDNs are well behaved in that they never change the URL of the manifest, or as a minimum provide a redirect. @slightlylate assures me that CDNs these days are well behaved and do not change URLs of resources. Does anyone have any additional data on that?

So assuming the manifest URL does not change, or at least provides redirects, the user agent can "periodically check the contents" of the manifest to see whether something has been updated. Then the spec says that "in the event that the members of the manifest have been updated, as determined by running the steps for processing a manifest and seeing if anything has changed, the user agent MAY update the metadata corresponding to the web application ".

Now during the "steps for processing the manifest" the spec says that the start_url must be checked to be same origin with the document URL, that is the document the app was installed from. The same is true for the scope property. If the user agent is just "periodically checking the contents" of the manifest, does it need to also keep a record of the URL of the document the app was installed from, in order to make these checks? Note that this URL may be different for every installation as an app can be installed from any page of the app.

So assuming that CDNs behave well, the manifest URL doesn't change, and we've stored the document URL, we can then periodically check the contents of the manifest. Does this work for authenticated manifest URLs too? How does the user agent get the necessary credentials to fetch the manifest? Is the check only done by the user agent when the user logs in and uses the app?

It sounds like the implementation in Chrome may also be based on some additional assumptions regarding a registered Service Worker scope, is that the case? @mounirlamouri ?

If the assumptions above are in fact assumptions made by the spec, do we need to write something in the spec about the fact that user agents do not expect the manifest URL to change, and that therefore CDNs are expected to keep the same URL for the manifest or provide a redirect?

[marcoscaceres]

Does this work for authenticated manifest URLs too? How does the user agent get the necessary credentials to fetch the manifest? Is the check only done by the user agent when the user logs in and uses the app?

Yes. This is what I imagined. Or via background sync. The manifest cannot be treated independently of its usage context, though it can work as a static resource.

If the assumptions above are in fact assumptions made by the spec, do we need to write something in the spec about the fact that user agents do not expect the manifest URL to change, and that therefore CDNs are expected to keep the same URL for the manifest or provide a redirect?

We can add that, but I don't expect anyone will actually read that: it's not as random developers and CDN providers reads specs. Authoring notes and warnings in specs are a bit of a waste of time. The examples in the spec are really for us who are working on it, as it helps us discuss ideas.

[benfrancis]

The manifest cannot be treated independently of its usage context, though it can work as a static resource.

Can you unpack that a little, what does it mean? Will an HTTP GET request to the manifest URL always get you the latest manifest? If not, what else does the user agent need to do to get the manifest? How would "background sync" work?

[benfrancis]

Maybe every time the user agent comes across a manifest link relation pointing at an installed manifest it should check to see whether the cache for that manifest has expired, and then re-download it if it has? If the manifest URL has changed, it is just ignored, unless the user installs the new manifest by installing the app again.

Presumably all user agents will need to implement some kind of store to keep a local copy of the manifest to generate launchers etc. from, and that store will need to be updated from time to time.

I can imagine the caching and fetching being optionally mediated programmatically by a Service Worker, but I'm not sure how the user agent would know to update an icon or app name on the homescreen for example if a Service Worker discovers a newer version of the manifest.

[marcoscaceres]

Can you unpack that a little, what does it mean? Will an HTTP GET request to the manifest URL always get you the latest manifest?

Independently of the application being run, probably not. An independent process cannot just go looking for a manifest on its own - it needs to do it as part of the the application life-cycle.

If not, what else does the user agent need to do to get the manifest?

The user needs to open the application. And then the user agent needs to check if there is a manifest. If there is one, download it and apply it.

How would "background sync" work?

Maybe something like:

navigator.serviceWorker.ready.then(function(registration) {
  registration.periodicSync.register({
    tag: 'get-latest-manfiest',     
    minPeriod: 12 * 60 * 60 * 1000, 
    powerState: 'avoid-draining',  
    networkState: 'avoid-cellular',
  }).then(function(periodicSyncReg) {
    // success
  }, function() {
    // failure
  })
});

// In SW... 
self.addEventListener('periodicsync', function(event) {
  if (event.registration.tag == 'get-latest-manifest') {
    event.waitUntil(fetchAndCacheLatestManifest());
  } 
});

// User opens app... browser checks manifest... oh... it has changed, re-apply! 

There is more complexity to the above... like the actual icons might have been updated by are still named the same thing, so it might be a matter of doing a HEAD request to check those, etc.

[marcoscaceres]

Maybe every time the user agent comes across a manifest link relation pointing at an installed manifest it should check to see whether the cache for that manifest has expired, and then re-download it if it has? If the manifest URL has changed, it is just ignored, unless the user installs the new manifest by installing the app again.

We can work this out internally as we implement this.

Presumably all user agents will need to implement some kind of store to keep a local copy of the manifest to generate launchers etc. from, and that store will need to be updated from time to time.

Maybe... they might also just generate/download things only on a needs basis and compare against what they are currently displaying (i.e., no need to cache manifest)... or might even be more aggressive and just periodically update icons, etc.

I can imagine the caching and fetching being optionally mediated programmatically by a Service Worker, but I'm not sure how the user agent would know to update an icon or app name on the homescreen for example if a Service Worker discovers a newer version of the manifest.

If the user launches the app, that's a pretty good indicator that this app should probably be updated.

[benfrancis]

Note that I just saw this on theguardian.com

<link rel="manifest" href="/2015-06-24-manifest.json" crossorigin="use-credentials">

That manifest URL does not look like it's designed to be static to me! How can we better communicate to developers that the manifest URL should not change, otherwise they will not be able to update their manifest (e.g. change their icon) for users who have already installed their app?

Also, the use of crossorigin="use-credentials" is interesting.

[marcoscaceres]

😿 @benfrancis, but they are using it correctly. The declaration of using a manifest is independent of its content and the URL (hence it should be possible to update the manifest if the URL changes).

The use-credentials flag there is redundant tho.

[benfrancis]

but they are using it correctly. The declaration of using a manifest is independent of its content and the URL (hence it should be possible to update the manifest if the URL changes).

This will not work.

An app can be installed from any page of the app. How is the user agent meant to know that some new manifest URL it comes across corresponds to the same app as an already installed manifest? Different pages on the same origin could point to different manifests corresponding to different apps.

If I install a Google Maps app from a page at:

https://www.google.com/maps/place/The+Warfield/@37.782739,-122.410156,18z/data=!3m1!4b1!4m2!3m1!1s0x80858085a9d7abdb:0x70d2c0624acc91fa

which links to a manifest URL at

https://www.google.com/maps/manifest.webmanifest

how will I get updates for that manifest?

If the user then navigates to

https://www.google.com/calendar/render?pli=1#main_7

on the same origin which links to a manifest at

https://www.google.com/calendar/manifest.webmanifest

do I just assume that this new manifest URL replaces the old manifest, or do I have to wait for the user to navigate to the exact page they visited before at

https://www.google.com/maps/place/The+Warfield/@37.782739,-122.410156,18z/data=!3m1!4b1!4m2!3m1!1s0x80858085a9d7abdb:0x70d2c0624acc91fa

in order to update the Google Maps icon?

Good URLs do not change for a reason.

[jmajnert]

Doesn't scope allow the service owner to solve this?

[marcoscaceres]

Doesn't scope allow the service owner to solve this?

That's the idea.

[benfrancis]

Doesn't scope allow the service owner to solve this?

Yes scope is intended to define the URL scope to which the manifest applies. But the default scope was unfortunately decided to be unbounded, which would mean that if we used this as an indicator of which pages could potentially provide a new manifest with a new manifest URL, any page from any site on the web could overwrite the manifest for any other site...

There are just too many footguns for developers.

Again, the obvious solution is that a manifest URL should identify an app, and should not be allowed to change.

[jmajnert]

I think we already had this discussion some time ago :-)

[benfrancis]

Yes, and we will keep having this discussion until the spec is in an implementable state :)

We're trying to implement this spec for Firefox OS right now, and it just doesn't make sense. I don't think Google have come across these problems yet because they haven't implemented the updating part of the spec - icons you add to the homescreen in Chrome can currently never be updated. I'm trying to find out whether Opera's implementation deals with these issues.

[marcoscaceres]

I'm pretty sure Opera's implementation will just be whatever is in Blink. @andreasbovens can probably confirm. About updating unbounded manifests, we could make it clear that those shouldn't be updated (as they can't be anyway for the reasons you outline); but that's ok.

Right now, just getting some icons showing up on a homescreen is sufficient - even if they can't be updated.

[benfrancis]

Right now, just getting some icons showing up on a homescreen is sufficient - even if they can't be updated.

This is short sighted. We should not be building these footguns into the spec, especially when there are already tried and tested solutions from years of experience building these things.

The spec says "By using the manifest URL, a user agent MAY periodically check if the contents of a manifest has been modified" - it doesn't say anything about replacing the manifest with any other manifest linked from any other page within the same URL scope (by default the whole web) which the user happens to navigate to.

[marcoscaceres]

Yeah, that bit is buggy, but we know that... artifact of when we thought we could have a stable URL for the manifest. Will file a bug to update.

[marcoscaceres]

Ok, linked from the spec. Will update soon.

[benfrancis]

artifact of when we thought we could have a stable URL for the manifest.

We have to have a stable URL for locating the manifest resource for describing a site, or the user agent can not retrieve an updated version of the resource, that's what URLs are for.

If any manifest resource linked from any page of the site is assumed to supercede any previously retrieved manifest from any other page, then the default scope has to be the origin of the app. But that seems like a dangerous and unreliable updating strategy.

[jmajnert]

The spec says "By using the manifest URL, a user agent MAY periodically check if the contents of a manifest has been modified" - it doesn't say anything about replacing the manifest with any other manifest linked from any other page within the same URL scope (by default the whole web) which the user happens to navigate to.

OK, here's how I understood it:
Once installed, the app manifest URL is periodically checked for updates. Until it's no longer there (possibly for some time to account for server failures etc), at which point it makes sense to update/replace the installed manifest with a new version linked from the pages within the app.

[benfrancis]

linked from the pages within the app

which by default is the whole web.

[benfrancis]

We have to have a stable URL for locating the manifest resource for describing a site

OK, I chatted with @sicking about this and he has a suggestion...

If we can't guarantee that the manifest URL is stable then an alternative is to use the start_url as the stable URL instead. We can update the manifest whenever the user visits the start_url (which happens whenever they launch the app from its launcher) - if a new manifest URL is linked from the page at the start_url when it is loaded, then we can update the manifest URL for the site and download the new manifest.

@sicking suggests that we should also periodically fetch the start_url, parse its HTML and follow its manifest URL in order to update the manifest for a site, even when the user doesn't visit it. I'm not sure whether that part will work.

Theoretically you could probably still also change the start_url as long as the manifest is still linked from the old start_url and you don't try to change both at the same time, or the old start_url redirects to the new start_url, but that might get a little bit hairy. We don't think sites change their start_url very often anyway...

One slight issue with this is that the default start_url is currently defined as the document URL an app is installed from, which could be different for every install. Updating the manifest then relies on every page of the app being updated with a new manifest URL and never disappearing.

For this reason I'd like to suggest that either the start_url become a mandatory field, or the default start_url be the root of the origin.

[sicking]

The default start_url should be "nothing". I.e. if a manifest doesn't define a start_url then we simply don't know what the homepage of the website is.

[jmajnert]

start_url is purely advisory, which means that UAs don't even have to load it when launching the app.
Which come to think of it is quite ok, given the work being done in EPR.

[benfrancis]

start_url is purely advisory, which means that UAs don't even have to load it when launching the app.

I think the user agent/user being able to override the provided URL is one thing, a more sensible default is another. Otherwise you risk a web app launcher that either can't launch anything, or can't be updated (if we were to attach special meaning to the start_url for the purpose of updates).

[jmajnert]

For this reason I'd like to suggest that either the start_url become a mandatory field, or the default start_url be the root of the origin

This sounds reasonable.
How about we take scope into account for the default start_url value?

[jmajnert]

What I mean is, if the algorithm https://w3c.github.io/manifest/#dfn-steps-for-processing-the-scope-member returns scope URL (not undefined), then let start_url be scope URL, otherwise root of origin.