loss ctc fluctuates

[qkrguswn2401]

hi, I am creating an ASR model on Korean dataset.
My E2E ASR model without language model works really well, wer around 1 on dev and evaluation set.
But When I apply language model fusion (deep and cold), ctc dev loss does not converge at all and around 50% WER on eval set.
attention dev loss converges and accuracy is about 99%.
I can not find the cause of fluctuation of ctc loss and not performing well on eval set.
Thank you