Bulk operations without extensions

[ben221199]

At the moment, for server-to-client documents (GET) the JSONAPI content could be:

• {"data":null} (no resource)
• {"data":{}} (one resource)
• {"data":[]} (multiple resources)

However, for client-to-server documents (POST, etc.) the JSON content seems to be only be:

• {"data":{}} (one resource)

Section 7.1 tells:

The request MUST include a single resource object as primary data.

---

However, I would suggest to add support for bulk creation/updating/deletion for the SAME type without using the Atom Operations extension. For this feature the JSONAPI document format does NOT have to change, because [] is already supported by the document parser, but only NOT added as possible value for POST requests. In this case, bulk would be possible using one request. (Eventually {"data":null} could be supported too for POST requests if for some reason.)

If there will be no errors during bulk, the server could respond with normal status code. If there will be errors, the server could choose to let all resources fail and show the errors or add the valid ones and show errors for the failing ones. In the last case, it maybe should be allowed to use data next to errors. (Maybe introduce a new errors attribute to tell what the server did; maybe: {"partly":true}. (This could also be a meta thing, so that it is up to the client to handle the partly failing operation.)

I also think this suggestion is compatible with previous versions because of the never remove, only add strategy. As already told, all JSONAPI parsers already support this format.

[jelhan]

Thanks a lot for proposing.

Is your main point about not requiring an extension at all? Or only to avoid complexity of the Atomic Operations extension?

#1380 proposes to add a bulk extension, which sounds very much like what you are asking for.

[ben221199]

I see Atomic Operations as a way to change resources with different resource types in bulk. What I meant with this issue is that it seems possible for me to add bulk for resources with the same resource type without changing the format of v1.0. So, this issue asks to leave out the restrictions of data in client-to-server documents, so that bulk is possible in a POST request, like it possible for a server to send a list when doing a GET request without id.

[ben221199]

@jelhan I think you can remove the 'extension' label, because I don't want it to be an extension. I want it to be in 1.1 (or 1.2).

[ben221199]

I want to propose an improvement:

Section 7.1 in JSONAPI 1.0; Section 9.1 in upcoming JSONAPI 1.1:

OLD:

A resource can be created by sending a POST request to a URL that represents a collection of resources. The request MUST include a single resource object as primary data. The resource object MUST contain at least a type member.

NEW:

Resources can be created by sending a POST request to a URL that represents a collection of resources. To create only one resource, the request MUST include a single resource object as primary data. If the request does contain a collection of resource objects, or if the data attribute is null or if there is no data attribute, it is up to the server how to handle the request. The resource object MUST contain at least a type member.

And as Note:

For the collection of resource objects, the server could choose to add the valid resources and return an error for the invalid ones (by adding the error attribute, or by replacing the invalid resources with null in the response). It could also choose to do it like a transaction: if one fails, all fail.

---

Above proposal will cause that the rules of the document is the same, no matter the direction (server to client or client to server). At the moment that is not the case, because a server may return a data attribute as object, array or null, but a client is only allowed to send an object, no array or null.

[jelhan]

@jelhan I think you can remove the 'extension' label, because I don't want it to be an extension. I want it to be in 1.1 (or 1.2).

The extension label express that this could be solved using an extension. It does not necessary mean that it most be solved using an extension.

V1.1 is considered feature complete. There won't be any new feature added.

So far I haven't seen a convincing argument why this needs to be added to the base spec. Going forward I would prefer having experiments in user space using extensions and profiles before adding to base spec. Such experiments should proof both: 1) That there is enough interest for the feature to be implemented in a noticeable number of libraries. 2) That it actually works in practice.

For the collection of resource objects, the server could choose to add the valid resources and return an error for the invalid ones (by adding the error attribute, or by replacing the invalid resources with null in the response). It could also choose to do it like a transaction: if one fails, all fail.

If it's not about transaction support, why not use separate requests in parallel?

[ben221199]

I will think about how to improve this. Maybe @lindyhopchris can give me some help, because he has a great library.

[lindyhopchris]

hey! so I can see the point of what @ben221199 is suggesting - i.e. Atomic operations allows you to do bulk operations for a mixture of resources, whereas what this topic suggests is the ability to do bulk operations for a specific resource type. I.e. POST /api/v1/comments could create multiple comment resources.

I suppose thinking about this as a package implementor, I'm not sure why I'd need to implement that if I'm planning to implement Atomic Operations. I.e. if I've implemented Atomic Operations, it's already possible to create multiple comments in one request, so I don't need a bulk create-comments endpoint in addition to the Atomic Operations endpoint.

I have however got a question for @jelhan though.... I do get the point of view that new features should be extensions first, before being adopted by the spec. However, the problem is once your API has implemented an extension, it's exceptionally difficult to drop it in favour of changes to the spec, because removing an extension would be a breaking change from the client's perspective - particularly when you're developing an API that third-party clients are connecting to. I.e. your content negotiation can't one day change from allowing an extension to not allowing an extension, just because that extension has been incorporated into the spec. Not sure if this has been thought about? I'm guessing in the scenario where an extension/profile has been adopted by the spec, as a server implementor I just let clients keep sending that extension in the headers even though it is technically no longer an extension.

[jelhan]

your content negotiation can't one day change from allowing an extension to not allowing an extension, just because that extension has been incorporated into the spec.

I think it is important to clarify what incorporating an extension into the base specification itself would mean. An extension can not be incorporated into the base specification without changes. An extension must namespace members and query parameters defined by it. However the base specification itself must not use a member name or query parameter with a namespace.

Incorporating an extension in the base specification would mean that the use cases supported by that extension would be supported by the base specification. The base specification might use a similar design as the extension. But it can not use the same design.

This means that the extension and the new feature supported by a new version of specification could be supported in parallel.

I'm guessing in the scenario where an extension/profile has been adopted by the spec, as a server implementor I just let clients keep sending that extension in the headers even though it is technically no longer an extension.

It would be up to the implementation to decide if an when to drop support for an extension. Regardless if it may decide to drop support because of the use case is better supported by another extension or by a new version of the base specification itself. Due to content negotiation rules for extensions a server could support different extension in parallel even if they are defining the same namespace. In that case it only needs to ensure that both are not applied to the same request in parallel.

[lindyhopchris]

@jelhan thanks for clarifying, that all makes sense!

[ben221199]

I see. Maybe if I still want this bulk feature, I should make something like bulk:data that accepts {"bulk:data":null} and {"bulk:data":[]}. Maybe even in combination with bulk:errors. because at the moment, data and errors aren't allowed together. If I see that this extension is working well in the community, it could be thought to merge this in the base spec, maybe 1.2. I will think about it all.

[jelhan]

I see. Maybe if I still want this bulk feature, I should make something like bulk:data that accepts {"bulk:data":null} and {"bulk:data":[]}. Maybe even in combination with bulk:errors. because at the moment, data and errors aren't allowed together. If I see that this extension is working well in the community, it could be thought to merge this in the base spec, maybe 1.2. I will think about it all.

I think that would be best way forward. Happy to review a draft of such an extension!

[jelhan]

I published a draft for a bulk creation extension: https://github.com/jelhan/json-api-bulk-create-extension Happy to receive feedback, questions and suggestions as issues on that GitHub repository.

I decided to limit it to resource creation only.

• Updating is more complex as soon as it comes to updating relationships. It's not obvious how to distinguish between patching relationships and doing a full-replacement. I don't see much value in supporting full-replacement only. Patching seems to be better pattern for most use cases.
• Bulk deletion seems to be an edge case. Especially if it needs to be atomic and cannot be done by many independent, parallel requests.
• Supporting creating, updating and deleting multiple resources in a single request is well supported by atomic operations extension. I don't see how it could be supported in a less complex way.

[ben221199]

Cool. This is what I meant when creating this issue, but in my case without bulk: extension namespace, so that it would be part of the base spec. Maybe we should brainstorm on this topic more to complete your draft. Afterwards we always can consider what to do with it next.

[pyvovarov-s]

Wow such a wonderful initiative, it's definitely a missing part of the existing standard, hope it will bring the schema to a new level of maturity.

[jelhan]

I don't see any need adding a feature to the base specification if it can be solved well using an extension or profile. I tend towards keeping the base spec lean.

[AlexanderKaran]

Hi Everyone,

I know this has been discussed extensively, but I have two questions.

1: Why do we need to add bulk? It is not a stretch for a POST to also accept an array? Bulk create is a very common operation; maybe there is an excellent reason for needing the bulk key, but initially, I can not see it. (Sure, someone will take me to school for that comment, lol.)

2: With bulk creation being so common, do we have to keep it as an extension? JSON:API spec is very common at Atlassian for our REST APIs, and in a new service, nearly all our POST requests support bulk creation.

[jelhan]

Great to hear that Atlassisn is using JSON:API a lot.

We can for sure consider adding bulk support to base specification at some point in time. But I'm still convinced that we should gather experience from real world usage before using extensions. What is blocking you from using an extension at Atlassian to address that common need?

[AlexanderKaran]

Nothing is stopping us from using the extension. It's more that there is consensus for it to be in the main spec and two for it not to need a bulk key.