Performance to expect

[yannik1015]

Hey,
I have no idea if my setup is performing unusually slow, as I am missing any reference to compare to. I would therefore like to know if the following number of iterations per second for training the model are normal or not. I am wondering as the speedup I get from switching training on the CPU to the GPU is only 2x.

I think it would also be interesting and beneficial if others could also state their system and performance in order to allow others verifying if their setup is performing normal.

GTX 1060 6GB during Training:

• KNRM: 7.5-10 it/s
• TK: 4.5-5.5 it/s

Greetings,
Yannik

[dblasko]

Hi Yannik,

For what it's worth, for TK, I also get about 3 to 4 it/s on CPU on a M1 Mac (can't use the GPU as Metal is supported in Pytorch 2.0 and onwards). This performance leads to a single training epoch not being finished after more than 16 hours of training, which makes it really difficult to know if the trained model actually learns well.

To the lecture team and others: Would it be possible to know if those training times are normal please, or if they indicate a mistake in our implementation?
An additional question given this speed of training: from my understanding, early stopping is usually done at the epoch level (check the stopping condition at the end of each epoch) - are we expected to do it intra-epoch here? (For example, we give feedback on evolution of the loss every 100 batches in an epoch, should we also perform early stopping at that level?)

Thank you!

Best,
Daniel

[yannik1015]

Thanks for the reference values. During the Q&A session they mentioned my speeds were okay but they also were not really sure about it. But if you get 3-4 it/s per second on a CPU and my GPU only runs marginally faster it does not really seem that these are intended values.

For your question regarding the intra-epoch stopping I think this will answer it #107

[ldosen00]

The longer times are okay, but try to reason and explain why it's happening in the report.

[yannik1015]

Okay after reaching out to the organizers I've been told that 5-10 iterations per second for a batch size of 32 seems to be enough.