What is the correct way of using batches? Help me understand how BentoML create batches

[maikelroennau]

Hi, I am implementing a simple face detector service with BentoML and I would like it to run as fast as possible so I am trying to use the batch processing feature.

I set batch=True in my service function with a JsonInput() as input:

@bentoml.api(input=JsonInput(), batch=True)
def predict(self, request):
    # my code here
    return results

Then I send the following request:

[
    { "client_id": "AAABBB", "image": "aaaAAA"},
    { "client_id": "CCCDDD", "image": "bbbBBB"}
]

image is base64 encoded.

While running the service I realized BentoML is delivering the request to my function inside a list:

[
    [
        { "client_id": "AAABBB", "image": "aaaAAA"},
        { "client_id": "CCCDDD", "image": "bbbBBB"}
    ]
]

What is making me confuse is the structure of the parsed request. The way it is being delivered to my function, I have only one item including my entire request, which is different from what is mentioned in the documentation, that tells me to iterate over a list with n items and produce a result with the same n length.

In my case here, I understand my input as a single request containing two different items, that should be batch processed. Am I doing/understanding this concept correctly? Please educate me on this topic.

Thank you in advance!

[yubozhao]

Hi @maikelronnau, Good question.

When you have batch mode on, BentoML will attempt to group the incoming requests together in a small batch.

For your case, if you are sending each item separately, BentoML will combine those requests together in a list.

# request 1:
{ "client_id": "AAABBB", "image": "aaaAAA"}
# request 2:
{ "client_id": "CCCDDD", "image": "bbbBBB"}

#BentoML will pass the requests data in form of a list to the function:
[
      { "client_id": "AAABBB", "image": "aaaAAA"},
      { "client_id": "CCCDDD", "image": "bbbBBB"}
]

[maikelroennau]

Thank you for the clarification!