Inference Graph supported by BentoML

[withsmilo]

Hi, BentoML team!
I'm trying to figure out how to implement an inference graph using the BentoML. We must be able to support real-time inference between several BentoServices already deployed in K8s in the form of acyclic graph. I think there are some following scenarios that can be considered when inferencing several models at the same time.

1. sequential inference : many one-model-in-one-BentoService

• This is really a bad design.
• supported by BentoML

2. sequential inference : one multiple-models-in-one-BentoService

• supported by BentoML
  • Can I invoke the service handler of model 2 from the service handler of model 1 ? #928 (comment)
  • https://docs.bentoml.org/en/latest/concepts.html#packaging-model-artifacts

3. sequential inference : many one-model-in-one-BentoService

• not yet supported by BentoML

4. parallel inference : one multiple-models-in-one-BentoService

• not yet supported by BentoML
  • User should implement directly some async operation between model1 and model2. This can sometimes be difficult for Data Scientist.
  • Support for ensemble inference #996 (comment)

5. parallel inference : many one-model-in-one-BentoService

• not yet supported by BentoML
  • User should implement directly some async operation between model2 and model3. This can sometimes be difficult for Data Scientist.

6. graph definition : many one-model-in-one-BentoService

• not yet supported by BentoML
• This is just an idea. Here is a graph definition.

inference_graph:
    endpoint: BentoService_A
    BentoService_A:
        - route: (BentoService_B, BentoService_E)
    BentoService_B:
        - model_1
        - route: (BentoService_C, BentoService_D)
    BentoService_C:
        - model_2
    BentoService_D:
        - model_3
    BentoService_E:
        - (model_4, model_5)

Can BentoML cover all scenarios here in the future? I want to discuss with you how to implement all the scenarios here to provide a more convenient interface to users. Please let me know if there is anything I am missing. Any comments are welcome.

[withsmilo]

Any thought about this suggestion? I will start to implement a prototype for this. :)

[parano]

Hey @withsmilo, thanks for this great summary on this topic!

Can BentoML cover all scenarios here in the future? I want to discuss with you how to implement all the scenarios here to provide a more convenient interface to users. Please let me know if there is anything I am missing. Any comments are welcome.

I think you have covered all the related use cases and BentoML should support all of them in the long-term. The next step would be designing a flexible API for users to define those Services and Graphs, and propose how it can be implemented/architected. Do you have a design in mind for supporting some(or all) of those use cases? Happy to take a look at the initial design and share feedback.

[ssheng]

BentoML added inference graph capability in the 1.0 release. Please read more here, https://docs.bentoml.org/en/latest/guides/graph.html