Validation loss fluctuating

[GauriDhande]

Iam trying to train the model_end with few hyperparameter tuning changes. Iam also training on my own dataset of 600 wavefiles, splited as 10% test dataset, 10% validation dataset and 80% training dataset.

def train_model(input_to_softmax,
pickle_path,
save_model_path,
train_json='train_corpus.json',
valid_json='valid_corpus.json',
test_json='test_corpus.json',
minibatch_size=8,
spectrogram=True,
mfcc_dim=13,
optimizer=Adam(lr=0.0001, decay=1e-6),
epochs=3000,
verbose=1,
sort_by_duration=False,
max_duration=18.0):

model summary:

---

Layer (type) Output Shape Param #

the_input (InputLayer) (None, None, 13) 0

---

layer_1_conv (Conv1D) (None, None, 100) 39100

---

conv_batch_norm (BatchNormal (None, None, 100) 400

---

rnn_1 (GRU) (None, None, 100) 60300

---

bt_rnn_1 (BatchNormalization (None, None, 100) 400

---

rnn_bi (GRU) (None, None, 100) 60300

---

bt_rnn_bi (BatchNormalizatio (None, None, 100) 400

---

time_distributed_6 (TimeDist (None, None, 29) 2929

---

softmax (Activation) (None, None, 29) 0

Total params: 163,829
Trainable params: 163,229
Non-trainable params: 600

---

None

First 20 epochs:
Epoch 1/3000
69/70 [============================>.] - ETA: 0s - loss: 828.7257 - acc: 0.0000e+00Epoch 00000: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 824.4764 - acc: 0.0000e+00 - val_loss: 1025.0829 - val_acc: 0.0000e+00
Epoch 2/3000
69/70 [============================>.] - ETA: 0s - loss: 576.0773 - acc: 0.0000e+00Epoch 00001: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 574.0380 - acc: 0.0000e+00 - val_loss: 965.9543 - val_acc: 0.0000e+00
Epoch 3/3000
69/70 [============================>.] - ETA: 0s - loss: 475.5182 - acc: 0.0000e+00Epoch 00002: saving model to results/model_end.h5
70/70 [==============================] - 71s - loss: 475.8133 - acc: 0.0000e+00 - val_loss: 840.2710 - val_acc: 0.0000e+00
Epoch 4/3000
69/70 [============================>.] - ETA: 0s - loss: 446.0698 - acc: 0.0000e+00Epoch 00003: saving model to results/model_end.h5
70/70 [==============================] - 70s - loss: 446.3768 - acc: 0.0000e+00 - val_loss: 649.6033 - val_acc: 0.0000e+00
Epoch 5/3000
69/70 [============================>.] - ETA: 0s - loss: 420.5421 - acc: 0.0000e+00Epoch 00004: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 420.1927 - acc: 0.0000e+00 - val_loss: 505.9156 - val_acc: 0.0000e+00
Epoch 6/3000
69/70 [============================>.] - ETA: 0s - loss: 412.7203 - acc: 0.0000e+00Epoch 00005: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 412.8178 - acc: 0.0000e+00 - val_loss: 450.0329 - val_acc: 0.0000e+00
Epoch 7/3000
69/70 [============================>.] - ETA: 0s - loss: 397.5730 - acc: 0.0000e+00Epoch 00006: saving model to results/model_end.h5
70/70 [==============================] - 68s - loss: 398.1866 - acc: 0.0000e+00 - val_loss: 395.4442 - val_acc: 0.0000e+00
Epoch 8/3000
69/70 [============================>.] - ETA: 0s - loss: 388.5409 - acc: 0.0000e+00Epoch 00007: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 386.8799 - acc: 0.0000e+00 - val_loss: 401.9153 - val_acc: 0.0000e+00
Epoch 9/3000
69/70 [============================>.] - ETA: 0s - loss: 389.5977 - acc: 0.0000e+00Epoch 00008: saving model to results/model_end.h5
70/70 [==============================] - 68s - loss: 389.4545 - acc: 0.0000e+00 - val_loss: 388.4293 - val_acc: 0.0000e+00
Epoch 10/3000
69/70 [============================>.] - ETA: 0s - loss: 378.1360 - acc: 0.0000e+00Epoch 00009: saving model to results/model_end.h5
70/70 [==============================] - 68s - loss: 376.7665 - acc: 0.0000e+00 - val_loss: 407.0841 - val_acc: 0.0000e+00
Epoch 11/3000
69/70 [============================>.] - ETA: 0s - loss: 374.1938 - acc: 0.0000e+00Epoch 00010: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 374.0701 - acc: 0.0000e+00 - val_loss: 361.8077 - val_acc: 0.0000e+00
Epoch 12/3000
69/70 [============================>.] - ETA: 0s - loss: 373.0912 - acc: 0.0000e+00Epoch 00011: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 373.9879 - acc: 0.0000e+00 - val_loss: 362.8776 - val_acc: 0.0000e+00
Epoch 13/3000
69/70 [============================>.] - ETA: 0s - loss: 370.4228 - acc: 0.0000e+00Epoch 00012: saving model to results/model_end.h5
70/70 [==============================] - 68s - loss: 370.9717 - acc: 0.0000e+00 - val_loss: 353.5565 - val_acc: 0.0000e+00
Epoch 14/3000
69/70 [============================>.] - ETA: 0s - loss: 363.2626 - acc: 0.0000e+00Epoch 00013: saving model to results/model_end.h5
70/70 [==============================] - 69s - loss: 364.6332 - acc: 0.0000e+00 - val_loss: 350.5256 - val_acc: 0.0000e+00
Epoch 15/3000
69/70 [============================>.] - ETA: 0s - loss: 361.7289 - acc: 0.0000e+00Epoch 00014: saving model to results/model_end.h5
70/70 [==============================] - 68s - loss: 362.7544 - acc: 0.0000e+00 - val_loss: 391.8794 - val_acc: 0.0000e+00
Epoch 16/3000
69/70 [============================>.] - ETA: 0s - loss: 360.1477 - acc: 0.0000e+00Epoch 00015: saving model to results/model_end.h5
70/70 [==============================] - 70s - loss: 358.5634 - acc: 0.0000e+00 - val_loss: 389.8897 - val_acc: 0.0000e+00
Epoch 17/3000
69/70 [============================>.] - ETA: 0s - loss: 363.4254 - acc: 0.0000e+00Epoch 00016: saving model to results/model_end.h5
70/70 [==============================] - 70s - loss: 362.3484 - acc: 0.0000e+00 - val_loss: 347.7054 - val_acc: 0.0000e+00
Epoch 18/3000
69/70 [============================>.] - ETA: 0s - loss: 358.4653 - acc: 0.0000e+00Epoch 00017: saving model to results/model_end.h5
70/70 [==============================] - 70s - loss: 357.7992 - acc: 0.0000e+00 - val_loss: 382.3785 - val_acc: 0.0000e+00
Epoch 19/3000
69/70 [============================>.] - ETA: 0s - loss: 355.4213 - acc: 0.0000e+00Epoch 00018: saving model to results/model_end.h5
70/70 [==============================] - 68s - loss: 355.0019 - acc: 0.0000e+00 - val_loss: 378.6394 - val_acc: 0.0000e+00
Epoch 20/3000
69/70 [============================>.] - ETA: 0s - loss: 356.1490 - acc: 0.0000e+00Epoch 00019: saving model to results/model_end.h5
70/70 [==============================] - 70s - loss: 355.6650 - acc: 0.0000e+00 - val_loss: 369.9678 - val_acc: 0.0000e+00

as you can see, the training and validation loss is very high an also vlidation loss is going up and down. Also, can anyone suggest how this loss can be reduced?

[Savant-HO]

Hello, I also meet this problem. Could you please tell me how to solve it? Thanks!