The engine caches the data in the translation

[sdlmw]

Hello，

When we are translating a sentence. The engine will automatically increase the memory of the graphic card.
Engine error occurs(OOM) when the memory of the video card is full.
Is it possible to turn off the engine caching mechanism？

thanks

[guillaumekln]

Hi,

During translation, the memory only increases if the batch size increases or the sentence length increases.

[sdlmw]

Hi @guillaumekln

Can I turn off the caching mechanism? We have a lot of request connections. Frequent cases of low memory.


Now I'm in a situation where the video card memory consumes much more than memory.

[guillaumekln]

Did you already set this environment variable?

THC_CACHING_ALLOCATOR=0

[sdlmw]

Hi @guillaumekln ,

I shouldn't have set it up. Excuse me, how should I do it?

[guillaumekln]

Try:

THC_CACHING_ALLOCATOR=0 th translate.lua <your options>

[sdlmw]

It doesn't look like it has any effect. @guillaumekln