Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the PixIT task, the ToTaToNet model, and a pipeline for joint speaker diarization/speech separation inference #1676

Open
wants to merge 79 commits into
base: develop
Choose a base branch
from
Open

Add the PixIT task, the ToTaToNet model, and a pipeline for joint speaker diarization/speech separation inference #1676

wants to merge 79 commits into from

Conversation

joonaskalda
Copy link
Contributor

The goal of this pull request is to add the new PixIT task, ToTaToNet model, and a pipeline to perform joint speaker diarization/speech separation inference.

joonaskalda and others added 30 commits March 18, 2024 12:53
@joonaskalda
Merge pull request #5 from pyannote/develop
8365802 
git: rebase to latest develop branch
@joonaskalda
add convnet layers to PyanNet
ff2c705
@joonaskalda
add stft and free encoders/decoder
28c9057
@joonaskalda
multitask learning first attempt
eca0a31
@joonaskalda
properly logging mixit loss in train/valid
1d2d1ab
@joonaskalda
add a weight to mixit_loss
4e175b6
@joonaskalda
reformulate multitask loss
7894bf7
@joonaskalda
add dprnn
9eeb29c
@joonaskalda
pair mixtures from same file with no overlapping speakers
2fb16e9
@joonaskalda
fix mixit loss for odd batch size in validation
1a83fec
@joonaskalda
make the MoM part of the original batch
3ebc46a
@joonaskalda
clean up
055744f
@joonaskalda
check that BS is divisible by 3
4af6be1
@joonaskalda
don't use MoMs with more than 3 speakers
4aa83b3
@joonaskalda
include original mixtures in separation branch training
ef14f8f
@joonaskalda
matching the order of dimensions of branch outputs
b218473
@joonaskalda
make n_sources an argument for model constructor
ebe3471
@joonaskalda
changing LSTM default num_layers to 4
df6794b
@joonaskalda
create separate tasks and models
1fe6106
@joonaskalda
Changing n_sources to 3
b3a7821
@joonaskalda
forcing alignment between separation and diarization
0c790d5
@joonaskalda
fixing edge case of 4 speakers in a second chunk
70adb7e
@joonaskalda
adding a VAD-like forced alignment loss
fcde9b8
@joonaskalda
refactor: remove vad_loss and warm_up, assume powerset everywhere
699a2fb
@joonaskalda
remove double check of num_speakers
065cde0
@joonaskalda
refactor: moved mom constrcution
3ff0ba5
@joonaskalda
remove unused mixit wrapper
8bb63d2
@joonaskalda
format with black
5bd913d
@joonaskalda
fix for last batch in validation having size 1
40c11db
@joonaskalda
adding documentation
3ee1484
joonaskalda and others added 30 commits March 18, 2024 13:39
@joonaskalda
add lstm
66146fb
@joonaskalda
add lstm back in
adcc594
@joonaskalda
fix forward
053324e
@joonaskalda
adding training on single speaker sources
c1519f4
@joonaskalda
check edge case
680e181
@joonaskalda
clean up and format
031cb9d
@joonaskalda
first commit
4d20d4a
@joonaskalda
enable finetuning wavlm with separate lr (pytorch-lightning 2.1)
21d7f40
@joonaskalda
add gradient clipping
8081bef
@joonaskalda
clean up
5104521
@joonaskalda
rename SepDiarNet and include receptive field
a8c8caa
@joonaskalda
fix rebase mistake in PyanNet
0a9597b
@joonaskalda
fix rebase mistake in segmentation mixins
ec36794
@joonaskalda
fix joint task setup
7cfdc75
@joonaskalda
fix joint task init
4b37532
@joonaskalda
fix data iteration and add docstrings
c963934
@joonaskalda
fix docstrings
658308a
@joonaskalda
remove functionality for additional noise sources
b3ba2d4
@joonaskalda
remove functionality for using original sources for separation (when …
e451a2b 
…one speaker speaking)
@joonaskalda
fixc rebase mistake in speaker diarization task
386d750
@joonaskalda
rename joint task to PixIT
60210d9
@joonaskalda
make wavlm finetuning optional
047f741
@joonaskalda
clean up ToTaToNet
110355e
@joonaskalda
add joint diarization separation pipeline
e2cdae4
@joonaskalda
fix docstrings and imports
4df4950
@joonaskalda
fix ToTaToNet behavior when WavLM not used
c848143
@joonaskalda
update requirements.txt
ffe583a
@hbredin
Merge branch 'develop' into pixit
9d79069
@hbredin
doc: update changelog
75c3c8e
@hbredin
Merge branch 'develop' into pixit
4112ef8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@joonaskalda @hbredin