memory sharing via gunicorn workers?

[ebbunnim]

Hi, I want to ask bentoml team about two things.

1. I'm trying to use the transformers library in the new version 1.0.0-rc1(to use the runner). But it seems that i have to define a transformers pipeline and store the model artifacts. By the way, the task I want to perform is related to sentence similarity, which is not supported by the transformers pipeline. What should I do?

2. Due to OOM issue, memory sharing between gunicorn workers should be done. If i can't use the rc1 version due to the issue above, i want to use 0.13 stable version. But in this version, after increasing the number of workers to 5, i got OOM right away. (Maybe the reason is that the size of the embedding vector, which is the basis for the sentence similarity calculation, is too large.) Can i use --preload option for memory sharing between gunicorn workers?
   ( NIT : The app packed by bentoml will be on the k8s cluster)

[bojiang]

Hi @ebbunim.
For q1, the doc about transformer integration needs to be improved for custom pipelines. We are investigating that.

The rough idea for your case now is to use a custom pipeline (https://huggingface.co/docs/transformers/add_new_pipeline). For now, users have to register the custom task to src/transformers/pipelines/__init__.py:SUPPORTED_TASKS on their own as well. We may provide a better solution, or at least docs, for this use case in future releases.

[bojiang]

For q2, to be clear, most ML frameworks have an internal solution to make use of multiple CPU cores. For example, users can set intra_op_parallelism_threads/inter_op_parallelism_threads to control the threads of TensorFlow operations. By default, the number would be CPU core count. Thus in many cases, N instances of model workers mean N*CPUs threads on the system. Because of the overhead of context switching, it drags down the throughput.

[parano]

About memory sharing between unicorn workers with --preload option, in general we do not recommend this approach. it is not really memory sharing, but simply preload the model in python before forking the process. It may work in some cases, but it is tightly coupled with the extension implementation and may not be the most efficient way of accessing a shared model (as @bojiang explained above).

I'd recommend go with 1.0 release. The runner design in bentoml 1.0 is going to solve the "memory sharing" issue you were looking for and avoid the OOM issue.

There is currently an issue regarding transformer custom pipeline, we are working on a fix. see #2534

[withsmilo]

@ebbunnim
Just as workaround with v0.13.1, you could manually add the preload_app setting if you want. Please refer 1db97c3. But I'd recommend to go with the 1.0 release also.