GSoC 2022: Toolkit for StreamField data migrations in Wagtail

[allcaps]

Discussion about Toolkit for StreamField data migrations in Wagtail a Google Summer of Code project.

Wagtail provides a field type for semi-freeform data, StreamField, which uses a JSON representation to store "blocks" of data. While flexible for editors, a pain point for developers has been writing data migrations when changes are made to the block definitions, due to its nested format. This project aims to provide a set of reusable utilities to allow Wagtail implementors to easily generate data migrations for changes to StreamField block structure. It could be extended to cover another common pain point when updating stored data: dealing with live and draft revisions.

[chosak]

Related issue: #2110 (Migrating a Block within a StreamField)
One previously proposed solution: #4755 (Add StreamField migration support)

At CFPB we have also developed some code for StreamField data migrations that lives here and is documented here. Our approach (which could certainly be improved upon!) tries to address migration of both the page itself as well as any historic revisions that also contain serialized versions of the StreamField data.

[thibaudcolas]

Notes from today’s meeting, based on questions from Sandil:

• This will likely start as a standalone Django management command, outside of Wagtail itself, to make it simpler to iterate on.
• This command will read configuration files, and interactively ask more details to handle specific cases, to auto-generate the migration.

[hazho]

I think it should be a functionality within the admin site, and in the page editor, this should be done as how a list of submission can be downloaded as xlsx and csv file,this should be a JSON like test file or .JSON file, this mean it will become easy for page builders to import and export parts of streamfield and even will open an opportunity for copying from a tree to another tree of streamfield

[hazho]

and should (specifically) be part of blocks.js file

[hazho]

even if we are talking about migrating streamfields of all excesting pages, the above approach with make that easy to migrate all pages by putting the whole process into stages (sub-processing)

[xgdyp]

Hello, I am interested in this idea and can you recommend to me some pre-works about this?

[jacobtoppm]

Hi @xgdyp, welcome and thanks for your interest! I don't know exactly what you mean by preworks. If you want background, I think reading through the links @chosak has added would be a good start. If you'd like to experiment yourself, start by setting up a development environment (https://docs.wagtail.org/en/stable/contributing/developing.html). If you're not familiar with Django's data migrations already, making a change to a non-Streamfield and writing a data migration (for example, making a ForeignKey non-nullable, so it needs a migration to add a default) using https://docs.djangoproject.com/en/4.0/topics/migrations/#data-migrations-1 would be beneficial. I'd then progress to making changes to StreamField and generating migrations. What do they look like? Which kind of changes cause you to lose data or break things currently? I'd then have a go at hand writing a StreamField data migration that addresses a basic change, such as renaming a block, so you don't lose data - this will give you a sense of what this project aims to make easier. Feel free to ask any questions you encounter!

[xgdyp]

Thanks a lot！I'll have a try !

…

---- 回复的原邮件 ---- | 发件人 | Jacob ***@***.***> | | 日期 | 2022年04月01日 16:40 | | 收件人 | ***@***.***> | | 抄送至 | ***@***.******@***.***> | | 主题 | Re: [wagtail/wagtail] GSoC 2022: Toolkit for StreamField data migrations in Wagtail (Discussion #8156) | Hi @xgdyp, welcome and thanks for your interest! I don't know exactly what you mean by preworks. If you want background, I think reading through the links @chosak has added would be a good start. If you'd like to experiment yourself, start by setting up a development environment (https://docs.wagtail.org/en/stable/contributing/developing.html). If you're not familiar with Django's data migrations already, making a change to a non-Streamfield and writing a data migration (for example, making a ForeignKey non-nullable, so it needs a migration to add a default) using https://docs.djangoproject.com/en/4.0/topics/migrations/#data-migrations-1 would be beneficial. I'd then progress to making changes to StreamField and generating migrations. What do they look like? Which kind of changes cause you to lose data or break things currently? I'd then have a go at hand writing a StreamField data migration that addresses a basic change, such as renaming a block, so you don't lose data - this will give you a sense of what this project aims to make easier. Feel free to ask any questions you encounter! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mr-vaibh]

I'm taking this project, and gonna collaboratively work on this 👍

[sandilsranasinghe]

Progress so Far:

For now, we have decided to develop this as a separate package. https://github.com/sandilsranasinghe/wagtail-streamfield-migration-toolkit So far we've developed an approach which could be split into 3 parts:

• A process to recurse through different block structures and map data
• Operations that carry out the altering of the relevant blocks
• Querying and updating the changed data

For the recursion process we are working with raw JSON like data. We use the current block definitions to identify different types of structures (StreamBlocks, StructBlocks, ListBlocks) when recursing (for now we have not taken block definitions which differ from the actual data into consideration).
We use a block path (similar to the block path in the code in the cfpb link mentioned by chosak) to identify the blocks which contain the child blocks which will be altered and map their data. When we reach the block at the end of the block path we pass the value of that block to an operation which will handle altering the value as necessary. For example "stream1.char1" would match,

[
    { ..., "type": "stream1", "value": [
        { ..., "type": "char1" }, # match this
        { ..., "type": "char1" }, # match this
        { ..., "type": "char2" },
        ...
    ] },
    { ..., "type": "stream1", "value": [
        { ..., "type": "char1" }, # match this
        { ..., "type": "char2" },
    ] },
    { ..., "type": "foo" },
    { ..., "type": "foo" },
    ...
]

Operations are classes which have an apply method which will be called for all end blocks matching the given block path from the recursion process. This will contain the logic for altering the block value. We will be having a few operations available for from the package for basic/common use cases. We have the following for now.

• rename blocks
• remove blocks
• combine stream children to a listblock
• combine stream children to a streamblock
• move stream child inside a structblock
• move list child inside a structblock
• alter block value

Users will be able to define custom operations for any use cases they want, and we will try to include operations requested by the community as we go on. For example, the simplest operation which changes a block value would look like,

class AlterBlockValueOperation(BaseBlockOperation):
    def __init__(self, new_value):
        self.new_value = new_value

    def apply(self, block_value):
        return self.new_value

When querying data one important consideration is avoiding validation which can remove data which is not in the structure of our current block definitions. For this we're using an annotation when querying to get the raw json data straight from the database column completely avoiding validation. Afterwards a list of operations can be applied on the data with the above recursion process for each page. Note that any non page models with streamfields can also be updated.
For each page, all revision data can be updated or if needed revisions from a given date onwards only can be updated. With the changes in wagtail 4.0, we support any models with revisions to have their revisions updated when migrating stream data. Currently, this process is done by having a MigrateStreamData class which subclasses django's migrations.RunPython and handles the querying and updating. Pages and revisions will be updated in chunks with queryset.iterator() and bulk_update to reduce memory usage, especially for cases where there are large amounts of revisions. Usage will be as simple as creating a new instance with the app, model and streamfield to which changes are to be made along with a list of operations and block paths, and adding it to the list of operations in a migration file.This is the progress so far.

What we are looking to work on next include a management command to easily generate code for migration files, an autodetect feature to detect limited changes (rename and remove)

[sandilsranasinghe]

Initial version of package 0.1.0 published at https://pypi.org/project/wagtail-streamfield-migration-toolkit/ This is still in the early stages of development, so any feedback would be most appreciated. Note that work is still underway on improving the autodetect feature.