can we train by Parallel Computing or Multithreading or multi-Progress

[joytianya]

can we train by Parallel Computing or Multithreading or multi-Progress?
Speed up training
thank you

[yutkin]

@joytianya Yes, we can! For example, look at YouTokenToMe. This BPE implementation quite efficiently uses parallel processing.

[taku910]

Thank you. I will take a look. Actually, the current BPE algorithm is a little conservative to find the most frequent pairs.

[taku910]

Will work on it in the next release.

[lockmatrix]

I am really looking forward to parallel training,
as running Asian language corpora on multi-core computers is extremely slow, making me feel like I am wasting my CPU...

[heyaudace]

Will work on it in the next release.

Hi @taku910 - I was wondering whether this feature was released. Thank you

[ganeshkrishnan1]

Just tagging along: is it possible to use multi threaded tokenization for multi-cpu training?