Adding few crop layers to an existing model makes it much slower

[mathmanu]

I have an interesting observation. If I add a few Crop layers to the mobilenet model, it becomes quite slow - more than 3x slower. I have a multi GPU setup. Is this happening because Crop layers are not CUDNN accelerated and there are overheads in data movement between GPUs? I can't think of any other reason.

[mathmanu]

Copying the previous conversation from issue #386

mathmanu commented 23 hours ago
@drnikolaev I have an interesting observation. If I add a few Crop layers to the mobilenet model, it becomes quite slow - almost 4x slower. I have a multi GPU setup. Is this happening because Crop layers are not CUDNN accelerated and there are overheads in data movement between GPUs? I can't think of any other reason.
@RSly

RSly commented 20 hours ago • edited
@mathmanu @drnikolaev , I also confirm the problem with the crop layers.
in an example:
A. with no crop layer, it takes 2hours to train
B. with 4 crop layers it takes 3hours to train

that is 1.5x slower...
@drnikolaev

drnikolaev commented 11 hours ago
@RSly @mathmanu please try this:

transform_param {
use_gpu_transform: true
...
@mathmanu

mathmanu commented 10 hours ago
Setting that parameter for Crop layer had no impact on speed - its still slow.
@drnikolaev

drnikolaev commented 10 hours ago
@mathmanu actually yes, this setting has nothing to do with CropLayer. Could you please open a request for this? Thank you!