Music Reformer results (some problems)

[asigalov61]

@lucidrains Tried reformer as you have suggested...Results were also not very impressive, unfortunately... In fact, I am getting much better results with the official Google implementation of Reformer so maybe the performance problems are due to something missing in the implementation? Again, not sure...

Here are the results:

NUM_BATCHES = int(1e5)
BATCH_SIZE = 20
GRADIENT_ACCUMULATE_EVERY = 4
LEARNING_RATE = 3e-4
VALIDATE_EVERY = 100
GENERATE_EVERY = 500
GENERATE_LENGTH = 1024
SEQ_LEN = 2048

training: 1%| | 1173/100000 [3:01:10<251:14:04, 9.15s/it]training loss: 1.5287952423095703

Generation 2048 @ 0.8

See attached samples.

It lost coherence mid-way on me in each one :( Not sure why it is so bad :( Original Google implementation works fine.

Music_Reformer_MIDI_Output_Samples.zip

[KMzuka]

I found a similar problem when training with REMI, the prediction of MIDI events is not very accurate...