RFC: Catalog Import API

[AydinHassan]

Change description

Effort

High

Description

Our current API for importing entities is very flexible, however:

• The documentation is lackluster and brittle due to the fact that the schema maps directly to the databases.
• This exposure of implementation details leads to issues documenting the api but also using it as certain fields and entities are just bookkeeping.
• The size and complexity of the current schema make it impossible to generate clients or mappers for it.
• Variants and Media uploads have to be fully managed by the client
• The api puts the effort of error handling onto the client

Resulting in increased effort to build and maintain middlewares.
We want to provide an HTTP API and a PHP API for importing catalog related entities independent of the database schema.
And reduce the amount of implementation details developers have to know about when integrating Shopware within existing
systems e.g., ProductVisibility, ProductMedia Entities, Media creation, errors, etc.

Benefit

• Evolve independently of the current database schema
• Be strongly typed
• Support referencing associations by different fields (eg assign tax by name instead of id)
• Be simpler to import images and generate variants
• Support asynchronous importing
• Have advanced error handling and reporting
• PHP & HTTP API
• Full Open API specifications

Break strategy

No expected breaks

Please see the full ADR for information regarding the API, specifications, error handling and so on.

Feedback

We are asking for feedback to understand the perceived value of this potentially large undertaking. Specific items we are interested in:

• Is this something you would use?
• What features would be most important to you?
• What features/data points would you absolutely need including for it to be useful in real world use cases?
• And any other thoughts/queries, eg about error handling, async nature, API, etc.

[thomaspeissl]

It would be great.
I have written many import applications, and the most challenging aspect for me is "Serialization failure" errors during big "sync" "upserts". Just look at all the data batching, "sleeps" and "retries" I had to incorporate into my Shopware Admin API client library here:
https://github.com/thomaspeissl/v-shopware-api-client/blob/e170ec24f06fbde043ee32fc379e0747259a32dd/shopware.v#L281C52-L281C52

[AydinHassan]

Great thanks for the feedback. Chunking is definitely something we will handle and retrying can also be added quite easy because we will use the message queue.

[ts-mothership]

Yes. Yes, definitively yes! The current way to import products requires too much technical shopware database knowledge. We had to implement whole flows to read different product states (stock, media, prices, variants, categories) in order to build the payload for updating a single product. While we didn't encounter too many issues, this process was still flawed, performance heavy, and it opened the door to race conditions, maintenance bugs, and other issues.

For the feature we'd expect a few cases. First and most important is the data structure. It should be possible to replace (no addition) product categories, properties, etc. with the product payload. For example:

{
  "productNumber": ...,
  "properties": ["red", "XS", "shoe"]
}

This should replace previous property assignment. No more addition. Shopware should figure out internally which entries to remove and add. Having an empty array will remove all options. Not providing the field "properties" should not touch properties.

If possible, avoid using uuids and introduce a unique code for entities. This field could be automatically set when creating the entity based on entity name (e.g. Color option "Red" => "red"). There are multiple benefits with a unique code. Payload is smaller, and monitoring, logging, error debugging is more comfortable, since we see directly which options have been used instead of finding each option via uuid first. This is currently time consuming because there are uuids in product everywhere: taxId, currencyId, deliveryTimeId, categoryIds, streamIds, optionIds, propertyIds, cmsPageId, tagIds, coverId, mediaIds, featureSetId, visibilities.

After that the next pain point would be product media. When importing a buyable product, we'd expect to have all required information in frontend inclusive images. However, managing product media is currently difficult for multiple reasons:

1. We need to create media before the actual product creation. If product creation fails, we need to deal with images. Either we delete them or we need to figure out, if this image already has been uploaded. Currently we use the latter and save an image hash into media customFields. On reimport we generate an image checksum and compare it with the hash in database to find an image which already has been uploaded.
2. One product image has been removed. Ne need to make sure the media image was removed too, not just the assignment.
3. The content of one product image has been changed. Again we use the checksum hash to make sure a new image is uploaded and the old image is deleted.
   Those issues can be addressed by providing a media array in product payload and let Shopware figure out the rest, maybe with an internal checksum hash for images to avoid doing unneccessary work. Example payload with different sources (file and url)

{
  "productNumber": ...,
  "media": [
    // existing media image exmaple. first in list should always be the cover
    "0004F1BEA7DC4959A1FB8AEE120B178A",
    // example of file reference used for import
    "var/image-import/product-back.jpg",
    // example of url reference used for import
    "https://my-domain.com/image-import/product-back.jpg",
  ]
}

I believe there can be still other improvements, for example when it comes to product type structure (standalone product, main product with variants or options, rules, etc....). But when those issues have been addressed, it would make our live a lot easier. Improved error handling and async import would be nice features too. Having this should be already way friendlier to developers who do not know the internal entity structure. Thank you for your efforts!

[micronax]

We solved the image problem by creating a small middleware companion plugin. The middleware writes the product image urls into a custom field of each product and imports the products in the first run as active = false.

The companion plugin then picks up the new product by the created event and runs a MQ processor to download and attach the images. After all images are present, we set the product to active = true.

[AydinHassan]

Regarding importing images we plan to import (download/create entities) first and then assign them to the product during product creation. This is part of the async process, where all dependencies are resolved from bottom to top with the root entities being processed last. If the product creation fails we will remove the media. Regarding the other bits we will have a think about them. Thanks for the feedback!

[micronax]

Yes, please! We built a middleware connecting different ERP systems to the Shopware API. We currently use batches of 100 products sent to the batch / sync API. The articles contain around 30 properties and each around 4-8 variants.

Sending 100 products with all properties etc. to the sync API results in devastating 34.000 MySQL queries even, when MQ-based indexing is enabled by header. The problem is that for every changes entity / relation a EntityWrittenEvent is emitted which is then received by dozens of subscribers etc etc.

Having an dedicated and easier endpoint for a lean product catalogue import with the aim to improve both performance and ease of the product sync would be great!

[florianliebig]

+1 on this. Didn't see anything regarding that in the ADR, too.

Maybe something like

$session->commit(['silent' => true'])

https://github.com/shopware/shopware/blob/trunk/adr/2023-09-22-catalog-import-api.md#api-overview

[AydinHassan]

We will use the Sync API under the hood regardless, so unfortunately this problem will still exist. At the same time it's something we could add to the Sync API and then we can also expose it here. Will discuss internally, but I can't see this as a viable option because the internal system uses those same events to calculate various important states. Thanks!

[azngeek]

We already have a working prototype for the "catalog import api". You can check it at : https://github.com/mothership-gmbh/sw6-simple-api.

Our format is pretty similar to what you have proposed:

 {
    "sku": "ms-variants.2",
    "name": {
        "de-DE": "T-shirt",
        "en-GB": "T-shirt"
    },
    "description": {
        "de-DE": "T-shirt",
        "en-GB": "Super-Shirt"
    },
    "price": {
        "EUR": {
            "regular": 20.00,
            "sale": 15.00
        }
    },
    "tax": 19.00,
    "stock": 20,
    "properties": {
        "color": [
            "green"
        ],
        "size": [
            "l"
        ]
    },
    "axis": {
        "color": [
            "green"
        ],
        "size": [
            "l"
        ]
    }
}

And for images, our implementation can also solve all current issues with the complexity of handling them. Tell us, how we can support you here :)

    "images": [
        {
            "url": "https://via.placeholder.com/50x50.png"
        },
           {
            "url": "https://via.placeholder.com/57x57.png",
            "isCover": true,
            "file_name": "placeholder_57x57.png"
        }
    ]

[kleinmann]

Linked repository seems to be private 🙂

[kleinmann]

There is some pre-existing work for resolving entities without UUIDs, see https://github.com/shopware/shopware/blob/v6.5.6.0/tests/unit/Core/Framework/Api/Sync/SyncFkResolverTest.php and https://github.com/shopware/shopware/blob/v6.5.6.0/src/Core/Framework/Api/Sync/SyncFkResolver.php.
I think it's currently just resolving product numbers to product IDs and is undocumented, so probably not ready for widespread use.
I imagine it may be one of the concepts used for the new Catalog Import API.

Liking the proposal so far!

[azngeek]

The Link to https://github.com/mothership-gmbh/sw6-simple-api/settings is public now :)
We do have an internal version which has slightly more features and is more suited to our use cases. So this implementation started as a high level idea and might not be continued in the future, depending on what Shopware wants to do :)

[AydinHassan]

Thanks for sharing @azngeek we will take a look when we figured out the lower level parts!