Ex3 Version of Transformers Library

[dmay01]

I just wanted to share my experience with the Transformers library. The suggested version in the initial requirements file is 3.5.1. However, I am now using version 4.19.2, which significantly sped up the inference in part 3. Before the upgrade, it took more than 6h for the inference of 52K query, context pairs. Afterward, it took 4 minutes - using the exact same code. Since I spent hours trying to get my code to run faster, I wanted to share this.

[sophiaalthammer]

Thanks thats great!

[ConBoe]

Hi @dmay01,
I am currently still having a computation time of multiple hours while I already use the version 4.19.2. Batching has not helped so far. Could you maybe give a hint on how you speed up your performance?
Would be greatly appreciated!!
All the Best,
Con

[martinmitteregger]

Do you perform the computation on GPU or CPU? Because you have to explicitly use GPU if you want to

[ConBoe]

So I turned the GPU on as described in the top of the template. Beside that I used Batching, but i did not give other arguments to the pipeline or tokenizer function. Is something else needed?

[GitianOberhuber]

Yes, use the device argument of the pipeline() function to the ordinal number of your GPU (probably 0), else the GPU isn't used.
https://huggingface.co/transformers/v3.0.2/main_classes/pipelines.html