About EmbeddingBag of torch

[henryqin1997]

In pytorch, EmbeddingBag is itself a module with description "Computes sums or means of 'bags' of embeddings, without instantiating the intermediate embeddings". DLRM take use of this module, and currently colossal has not adapted it. Is EmbeddingBag from pytorch supposed to work directly with colossal layers, or it will be adapted later?

[FrankLeeeee]

@kurisusnowdeng Can you answer @henryqin1997 's query?

[kurisusnowdeng]

Hi, @henryqin1997 . Thank you for your interest. We are sorry that we do not support embeddingbag now. You can directly use the pytorch embeddingbag if no tensor parallelism involved. Or, to use tensor parallel embeddingbag, you may have to use our tensor parallel embedding and aggregate the results manually. We will add this layer into our system very soon.