Uneven GPU utilization.

[bolli24]

Hi,
I am currently trying to train a model. But it seems to me that my GPU is not used fully all the time. As I can tell by the GPU power logging using HWiNFO the GPU is idle about 50% of the time. To a novice this looks like a bottleneck somewhere. What is causing this behaviour?

My GPU is a RTX 3060 Ti and the CPU is an AMD R5 5600X.

I am running the training on Windows with WSL2 using the following arguments:
stylegan2_pytorch --name attempt5 --data ./2022_100k --num_train_steps 10000 --image_size 64 --log --transparent --save_frames --network-capacity 32 --batch-size 8 --gradient-accumulate-every 10 --save_every 300 --evaluate_every 100 --attn-layers [1,2]

Here is a GPU power graph produced by HWiNFO. The interval are 5-7 seconds in duration.