ResNet backbone vs. mask pooling

[hbredin]

It has been noticed that the 3.1 pipeline efficiency suffers from speaker embedding inference. With the default config, every 10s chunk has to undergo inference 3 times by the embedding model. It proves effective by separating the whole embedding model pipeline into the resnet backbone and the mask pooling. With this modification, every chunk only needs to be inferred one time through the backbone, bringing almost 3x speedup in my experiment. Furthermore, cache inference strategy helps a lot as well, given the default overlapped ratio of 90%.

Originally posted by @mengjie-du in #1621 (comment)

Hey @mengjie-du, that's a nice idea. Would you contribute this to the pyannote.audio codebase? I tried to send you an email at the address mentioned in this paper but received an error message in return -- so I am taking my chance here.

[mengjie-du]

Sorry for late response.
My experiment code is built upon a unwarpped version of 3.1 pipeline only for cpu, which includes two mentioned separated onnx model (ResNet backbone and the final FC). The musk pooling is implemented using Numpy. Thus, the code can't be embedded into the pipeline directly. I plan to test a version compatible with the pipeline soon.
I think the key point is the line 322 in pipelines/speaker_diarization.py, where the same wave data is yielded three times. It would be more effient to yield used_mask with a shape like (spk, 1, num_frames). This adjustment would allow the model to infer all of these together without some big modifications to the control flow..

[hbredin]

Thanks. Will also look into this myself.