Auto-save

[TWOC-WHT]

I saw that #3527 is still open, but no update so far.
In the issue, it is mentioned that auto-saving revision could lead to a massive amount of revision.
I also remember discussing this in slack and there are other issues like required field that are missing.
As an additional note, I have notified the users to save often, but in the end, we can't really control what the user do, sometimes they don't want to break their writing/editing flow.

Idea 1: localStorage and storing it in browser.

I am not an expert on this and not sure localStorage is a viable option for storing the current state of the edit page. There must be some kind of problem of using this method? The idea is to "save" the content of the edit page (let's call it "recovery") into LocalStorage every X minute (Overwrite the previous recovery instead having a new recovery every time). Also, when the user actually click Save Draft/Publish button, the localStorage should be cleared? or should it be cleared upon publish only?.

Idea 2: Having a "recovery" table in the database

If say the computer crashes and the user needs to grab the "recovery" on another computer. With this, I am not sure if it will interrupt/refreshes the page upon every auto-save. I am also not sure how this will affect the performance of the database too.

Just writing out what I have in mind, I also saw that there's django-autosave, and that there's something about using Django Channels? (I haven't looked further into both yet)

Has anyone out there tried out/come up with an auto-save solution that works?

[kaedroho]

Thanks for starting this discussion! I've been thinking a bit about how we could implement autosave recently, here are my thoughts on what we could implement to support a basic (non-collaboration) autosave in Wagtail

Save draft without page reload

Currently, Wagtail reloads the page on every save. Before implementing autosave, we'll need to change it to save in the background instead.

This requires the following:

• The “Save draft” button should submit the page data in the background
• Indicators displaying when the page was last saved should update automatically
• Validation errors need to be added to the form dynamically in the browser

This on its own would actually give us a nice UX improvement as the browser will no longer scroll back to the top of the page on save making it easier to work on larger pages.

All other options such as “Publish” and “Submit to workflow” should continue to reload the page.

Autosave

After saving without reload is supported, we can start getting Wagtail to trigger save draft automatically in the background, and remove the "Save draft" button.

Things that can trigger save draft:

• 3 seconds after a user has stopped typing in a text field
• Interaction with any chooser

Revisions

To avoid bloating up the database with lots of revisions which can make page history really hard to manage, we need to rethink how revisions are created for autosave.

One approach to this could be to create a revision on the first edit of a session* and update that revision on subsequent edits.

*Where a session starts when the user opens the page editor, and finishes when they navigate away.

Note: We will need to change the revision behaviour again later on when we start allowing multiple editors to edit a page at the same time. For now, we would need to implement automatic page locking to prevent editors from overwriting each other

Creating new pages

The create page view should work the same way it does today:

• User clicks “add child page”
• Fills in data
• Clicks “create page” (renamed from “save draft”)
• After the page saves successfully, they will be taken to the edit view, and autosave will start working

Invalid data

If the page is edited in any way that makes the data invalid, the validation errors will be displayed to the user immediately and auto-save will stop saving the page until the error is corrected.

Automatic page locking

At this stage, autosave will not be able to handle more than one user at a time. So to prevent them from accidentally overwriting each other, we will implement automatic locking in the page editor. This should be separate from the existing manual locking feature.

The first user to visit the page editor will not notice any difference and be able to edit normally. If another user tries to edit the page before that first editor has left, the fields will all be locked.

The page would be automatically unlocked when:

• The first editor leaves
• The lock automatically times out when the first editor doesn’t make any changes for 30 minutes

When the page unlocks, it will automatically reload for any other users who are viewing the locked editor. This is to make sure that the user who acquires the lock is on the latest version of the page.

Note: locks can only be acquired upon reloading the page editor. Also, we will know on the back-end who has locked the page so can add some validation to prevent saves from other users from happening.

If the user who has the lock doesn’t edit the page for more than 30 minutes, the page would be unlocked and the fields would become locked for that user.

[TWOC-WHT]

Thank you for the reply! Coincidentally, the automatic page locking is also something my team have asked too!
I have several follow-ups:

1. Is removing the "Save draft" button necessary? (or more specifically, will there a problem if we leave the button there?), this is more for the case where the user like the peace-of-mind/assurance from clicking the "save draft" button.
2. The invalid data handling sounds great! For the validation errors display will it be a good idea to specifically mention "Auto-save have been stopped due to some missing required field, to resume auto-save please fill-in the required field" (or something like to that effect?) and also giving it a warning feel instead of an error feel?
3. The revisions creation sounds like it could work, I do like the idea of having the same revision being updated until the user navigate away (prevents bloating/help to distinguish different session, also helpful since I encourage my team to save regularly, as suggested in the editors guide). However I don't know why my gut feeling says there could be some kind of potential bug with this (page data failed to be submitted/incomplete page data etc.); will it be useful to mark/differentiate between an autosave-revision vs manual-revision?
4. Additionally is there a risk with providing some kind of local/browser specific backups? in the case of internet going down/connection failure/etc.

and lastly, collaboration/the ability for multiple editors to edit a page at the same time, for future feature will definitely be amazing! (if you have to make a guess, when do you think this could fit in the roadmap?)

[kaedroho]

1. We could still have it. Hopefully, autosave would happen fast enough that your changes would already be saved before you have a chance to click "Save draft". And this would be displayed at the top somewhere.
2. Yep, if someone added a new block that had required fields, need to think about when the errors should appear and how they would look. Would be quite annoying if it instantly tried to save the block just after you created it and gave you a bunch of "this field is required." errors
3. I think only the last revision would be an autosave revision. I think it would only save if the page data is valid.
4. I think one issue that could occur with that is if someone else starts editing the page, it may not be possible to reconcile those local changes with the updated pages so those changes might be lost

and lastly, collaboration/the ability for multiple editors to edit a page at the same time, for future feature will definitely be amazing! (if you have to make a guess, when do you think this could fit in the roadmap?)

Yes, this is on the roadmap, subject to us getting funding to develop it though. The description in my previous comment was taken out of a wider idea for implementing collaboration in multiple stages. I haven't fleshed out all the details of each stage yet, but I think the stages for full collaboration could be:

1. Autosave with whole page locking
   Introduces basic autosave functionality into Wagtail for a single editor and automatic page locking. This is described above

2. Page patching
   Introduces an API for updating fields on a page without updating the whole page

3. Per-field locking
   Enables multiple editors to edit the same page, with automatic locking on individual fields

4. Per-block locking
   Extends the automatic locking system to lock individual blocks, inline panels, and comments

5. Rich text field collaboration
   Enables multiple editors to edit the same rich text field at the same time

[gasman]

We could still have it. Hopefully, autosave would happen fast enough that your changes would already be saved before you have a chance to click "Save draft". And this would be displayed at the top somewhere.

There is a practical reason for keeping the Save Draft button, not just 'peace of mind' - it will ensure that a persistent revision gets saved at that point (one that doesn't get overwritten on subsequent auto-saves), allowing the possibility of rolling back to it.

[thibaudcolas]

FYI we’ve further discussed this internally – pondering how much behind-the-scenes changes would be needed, and what the experience looks like for end users beyond the simple "success path" that’s easy to cater for. We’ll likely provide more substantive updates in the near future.

If others have good examples of systems that do this well currently, please share, keeping in mind a lot of the UI considerations we need to make are around data validation (e.g. required fields), so things like Google Docs which we all love but don’t have such a concept aren’t as relevant.

[phildexter]

Two big questions we've been thinking about with Milestone 1 (which is where we're likely to be aiming initially (unsurprisingly)).

1. When creating a page, at what point do we start autosaving.
Our current thinking is...

• A user navigates to the place in the page tree and clicks to add new page (as today)
• New page editor opens up. At this point the system hasn't saved the page (as per paper.dropbox.com.new).
• User enters some content into the page. This could be some text, could be a chooser. Our expectation is that most of the time it will be the Title. At this point autosave kicks in for the first time and the new page is saved. If the user hasn't filled in the Title of the page at this point, the page is saved as "Untitled Page" (this is a bigger change as it will mean us changing page saves under the hood, as currently they need a Title).
• Once the page saves the URL will update from /new to /pagetitle. At this point also validation errors will kick in - so the page will suddenly change to show validation errors for any required fields. We need to make sure that these validation errors are visible, without being too jarring, making people think they've done something wrong.
• As users continue to edit the page, the page will continue to autosave, even with validation errors. However, the user will not be able to Publish / Submit to a workflow until the validation errors have been dealt with. If they try and publish, a more prominent error will show (as per Contentful).

2. At milestone 1, although the page is autosaving in the background, we are unable to show changes made by others in real time. How do we prevent people from consistently overwriting each other's changes.

• To elaborate a little, milestone 1 means that if I'm editing a page in Wagtail for half an hour and then I leave, we only save 1 revision for that session. Each time the page autosaves, it overrides my preview revision. This avoids endless small revisions using up lots of database space and making things hard to understand. (as per Matthew's comment above, it will also be possible to trigger a new revision save if needed). This means that if two people load the page up and start editing, they'll constantly be updating the latest revision with their own version. Whoever makes the last edit wins - the other person's changes are "lost" (or saved in a revision, but still...)
• This is true with the existing way Wagtail works - but it feels a more pronounced issue with autosave. There are two broad ways for us to tackle this. We could lock a page when someone is on it. This feels bad as people might have tabs open but not be actively editing it.
• Given we have a roadmap for where we're taking this next, we want to avoid over-engineering a solution here that will soon be thrown away. Therefore, we are thinking that when a user lands on a page that someone else is already editing, we show a warning - something like "Person X is currently editing this page. Please consider contacting them to avoid losing changes." when they load the page. It is then up to that user to figure out if they can continue with editing it, or if they want to contact the other person.

[gasman]

One approach to this could be to create a revision on the first edit of a session* and update that revision on subsequent edits.

*Where a session starts when the user opens the page editor, and finishes when they navigate away.

We need to be careful to correctly handle the case where a user has multiple tabs open editing the same page - if they edit, save, then go back to a tab containing an older version of the content and edit any field, this will trigger an autosave that replaces the latest revision by that same user - at minimum, we need to make sure that happens as a new revision, not an db update on that latest revision.

[Scotchester]

I think this has come up earlier in this discussion, but I just would like to emphasize that it would be extremely useful to have a field be required only on publishing, but not when saving a draft of a page. There are so many occasions I've seen where one would want to start drafting a page before certain required content is ready.

[TWOC-WHT]

have a field be required only on publishing

An example of this in my experience is the cover image for an article, especially where the original writers don't decide the cover image (i.e. Needs to be approved by Editorial team).
In Wagtail I made the cover image not mandatory (to make drafting/submit to workflow more convenient).
So having that final reminder of "Can't publish until this is filled" will be a unique experience! [Although from my side it's more of a "nice to have"]

[gasman]

Yep, we'd definitely hope to implement things so that validation is only enforced when the page is published or submitted to a workflow - auto-saving would be pretty clunky without that, since the expectation is that you'll frequently be saving things in an unfinished state.

This does raise the question of how to make that work when the validation constraints exist at the database level - even though draft revisions are saved as JSON in the PageRevision model where that isn't an issue, we have to save something to the actual database table in order to assign an ID. Hopefully we can solve that by changing the database representation for unpublished draft pages, so that the base Page record exists but the record for the specific page subclass doesn't.

[fabienheureux]

+1 !
As explained in the Wagtail page editor discussion, below a recent experience of mine

Apart from the auto-save feature, this could be really useful, again, for some editors I worked with.
In a recent project, there was a content agency involved that had to write a lot of content, quite early in the project, without worrying about publishing dates and a few fields that were required for the finished project.
We had to make some fields non-required while they were saving pages in order to smoothen the process.
Having the ability to save a page without mandatory fields being filled in could be awesome, while preventing the publication of such a page for obvious reasons.

[lb-]

just an idea what if there was a form class for the pre-saved state? by default this would be the same page form but with all fields non-required. Customisations could have different fields hidden or even added for this if they wanted. This form class could own the serialisation of that data and then mapping to the real fields when an actual save happens.

[thibaudcolas]

For people following this discussion - a few recent updates:

• We’re going ahead with the most basic type of auto-save as part of the page editor 2022 project.
• @allcaps shared a POC (https://github.com/allcaps/formpoc) of POSTing Django forms data on field changes (for the purpose of validation only in the POC)

We’ve been discussing UI frameworks for Wagtail in #7689 (comment) and as part of the page editor work. There is some overlap between those discussions (cc @lb- @kaedroho @mixxorz) and auto-save / collaboration features – as soon as we start syncing page updates from multiple users and reflecting other users’ updates in the browser, we’d need some form of dynamic form re-rendering, which doesn’t fit well with Django’s vanilla forms rendering.

For the page editor 2022 project’s variant of auto-save, we’re not expecting any significant re-architecture, but in the future we expect this might be needed. There are a few important considerations here:

• Re-rendering of a sub-part of the form would be server-initiated, based on a data change in one user’s browser.
• We’d likely want some form of client-side dynamic rendering for this (Telepath, React, both) so we can also retain any client-side-only state.
• We’d also have to worry about the state of support for WebSockets in the Django world – how much of a proportion of Wagtail sites we could expect to use a compatible server, and what we’d do for sites that don’t.

Short term, we’ve also been wondering how auto-save interacts with bespoke Django widgets:

• Can we rely on them firing specific DOM events?
• Should we instead rely on observing DOM mutations?
• If a field doesn’t support auto-saving for some reason – how would we signal that to users?

[lb-]

I do think it would be worthwhile seeing how this could be tackled with vanilla JS (similar to the epic POC from @allcaps ).

However, I am of the strong opinion this could be achieved well with Stimulus JS and either patching the HTML (something similar to how Turbolinks/htmx does it without going all in on those libraries also) or the JSON response approach. The nice thing about Stimulus JS is it handles all the DOMMutationObsever stuff under the hood so that when new elements are added to the DOM (e.g. InilnePanel) we can hook into those elements being added and track their onchange behaviour (or similar).

This approach would require data attributes added to fields (or their containers), but this offers a nice way to 'opt in' to the behaviour on a per field basis, for simple Django fields it would hopefully just require widget.attrs added.

Here is an example from the Rails world of the DOM attributes that could be used https://stimulus-components.com/docs/stimulus-rails-autosave/#usage

This video may be useful - htmx and Django autosave https://www.youtube.com/watch?v=orUuVcKUPqE

If we want, I would be happy to give a POC a go also with Stimulus, building on the formpoc above.

[lb-]

For more inspiration (or even usage) it would be worth looking into how these Django libraries solve the more abstract problem of making HTML from Django templates available dynamically without having to build a discrete API endpoint (or JSON representation) of each thing.

• https://sockpuppet.argpar.se/ (uses Stimulus JS for the JS side)
• https://github.com/edelvalle/reactor
• https://www.django-unicorn.com/

These all use some kind of HTML patching when the new view response is received from the server, often using https://github.com/patrick-steele-idem/morphdom to do the work there in a smart/efficient way.

This may work to avoid a whole class of problems where we are bypassing the Django HTML rendering engine to add behaviour and instead embrace it.

This video is a good starting point for an overview of these (from the author of Django Unicorn), including an intro about where VueJS can create added complexity and force more work in the JS side and less on the Django side (applies to React also).

https://www.youtube.com/watch?v=UlFcUXxYSto

[thibaudcolas]

Thank you @lb-, it’s indeed probably too early for me to say this is a "React" thing. I like the idea of doing POCs but I think it’s too early, most likely we’d want an RFC or other similar technical spec first (there are even more complexities in the backend / Django models side).

With the first iteration of autosave, so we get a sense of the complexity, all I’d expect to need dynamic rendering is the form & fields’ "error" state. At the field level, this is compromised of:

• An "error message" element displayed either below or above the fields
• Styles to denote the error state – could be as simple as using a CSS class on a container element to add a red border
• The field’s aria-invalid="true", and aria-describedby pointing at the error message

At the form level, much less likely to be an issue with how Django forms normally render but worth mentioning for reference:

• Non-field errors as a list with the right semantic markup
• Visual indicators in the page editor’s tabs, in the minimap, showing where there are errors
• (TBC) a live region announcing the form’s state

This dynamic rendering would be based on the following state (final logic TBC):

• Whether the given field has been "touched" or not
• Whether there were validation errors for the given field
• Whether the form was auto-saved, or the user attempted to publish

---

For future milestones, I’d like to consider HTML patching approaches but I’m worried they’d be too limited if the markup rendering happens server-side with the client doing a diff + swap. As I understand them, those approaches generally assume that:

• The HTML to render is derived only from the state which the server is aware of. An illustration of where this fails is… form fields’ states, such as undo/redo stacks. I believe some libraries like morphdom at least preserve the focus state between re-renders, but I don’t know how other parts of form fields’ states fare.
• The HTML is more or less stateless. It’s unclear to me how that then interacts with JS widgets which might not be able to re-initialise based on changes to their markup if they weren’t conceived to work like this from the start (for example a widget that only initialises on DOMContentLoaded, or one that renders differently depending on viewport dimensions).

[lb-]

Great points @thibaudcolas - thanks for the response. Happy to help where needed on this front if I can.

[thibaudcolas]

For people interested in this – here’s a recent update in @phildexter’s page editor build update blog post. We’re currently looking for funding to support the wider rework of the page editor, and this is one of the features that will be de-scoped if we don’t find this funding.

---

Here is the UI we’ve designed so far for reference – a top-level "auto-save indicator". Then form fields’ errors would appear similarly to they do now (just as you go through the page rather than on publish/save draft). And "Save draft" would obviously be gone – we’d only retain an option to name specific "auto-saved" versions.

Additionally we would have a dialog to help with potential conflicts, if multiple editing sessions are attempting to auto-save a page in a way that would create conflicts:

[kaedroho]

Additionally we would have a dialog to help with potential conflicts, if multiple editing sessions are attempting to auto-save a page in a way that would create conflicts:

I think that dialog wouldn't be necessary if we implemented automatic page locking.

Allowing multiple editors to edit the same page and showing a dialog whenever someone else types something might become a quite annoying experience, so I think it would be better to prevent multiple people from editing the same page at the same time.

See the description of "Automatic Page Locking" from my previous comment:

The first user to visit the page editor will not notice any difference and be able to edit normally. If another user tries to edit the page before that first editor has left, the fields will all be locked.

The page would be automatically unlocked when:

• The first editor leaves
• The lock automatically times out when the first editor doesn’t make any changes for 30 minutes

When the page unlocks, it will automatically reload for any other users who are viewing the locked editor. This is to make sure that the user who acquires the lock is on the latest version of the page.

[BrendanMartin]

Are there any updates on this?

[mattlinares]

Would be great to get an update on this!

[digicase]

I have used this fork of Django Admin locking in my projects and been happy with the results.

https://github.com/theatlantic/django-admin-locking

The features match the description @kaedroho wrote for automatic page locking, and may be of some use for reference despite not receiving regular development.