sds loss in kosmos-g

[hrz2000]

Describe
Model I am using (kosmos-g):

I didn't find the code for instruction fine-tuning of kosmos-g, specifically, I was more concerned about how to optimize kosmos-g with sds loss. In my own attempt, the shape of the noise difference is not equal to the shape of the text_embedding, and it is difficult to skip the middle unet term for gradient backpassing.

Very much looking forward to someone answering and discussing this issue, thank you！