Gnmt train slow

[yajiedesign]

GPU cannot be filled with train_gnmt.
GPU:GTX TITAN X
CPU:4770k
GPU LOAD: about 10%
CPU LOAD: one core full.
dataset: LCSTS
vocab size: 320783
num work: 0 (Trying to set it to 2 seems to improve, but it takes up too much memory (about six times as much as 0)).
batch size: 16

[ymjiang]

Batch size=16 seems not reasonable. Did you try to increase it?

[yajiedesign]

@ymjiang There's no way to increase it any more. Now it takes up about 10G of gpu memory.

[szha]

cc @sxjscience. One way to work around the memory limit is to perform gradient accumulation. Several model zoo scripts already implement this logic, and it might be a good idea to add it to GNMT too.

[sxjscience]

Will take a look at it.

[szhengac]

@zhreshold Using num_worker=2 requires six times more memory compared to num_worker=0 seems unreasonable. Do you have any idea?

[zhreshold]

@szhengac using multi worker will enable pre-fetching batches, default is 2*batch size, and each worker will be fetching a batch simultaneously.

[yajiedesign]

now.I saved gpu money by setting share_embed.and I increase batch size.it some improvements.

[szhengac]

@zhreshold but each input batch is simply an integer matrix, which shouldn't occupy too much gpu memory.

[zhreshold]

@szhengac Did you mean gpu memory? Dataloader should never occupy gpu memory

[szhengac]

@zhreshold Oh I mean cpu memory.