Loss function

[314rated]

Hi,
Thanks for the nice and very helpful work! I am also trying to do text-to-image generation, but on tensorflow.
My loss graphs are going totally wrong:

1. The loss functions I am using are:

d_loss_real = tf.reduce_mean(disc_real_image_logits)
d_loss_fake = tf.reduce_mean(disc_fake_image_logits)
d_loss_wrong = tf.reduce_mean(disc_wrong_image_logits)
d_w_loss = d_loss_fake + d_loss_wrong - d_loss_real
g_w_loss = -1*(d_loss_fake)

2. For optimisation I am using following lines:

rms_d_optim = tf.train.RMSPropOptimizer(learning_rate=5e-5).minimize(  loss['d_loss'],var_list=variables['d_vars'])
rms_g_optim = tf.train.RMSPropOptimizer(learning_rate=5e-5).minimize(loss['g_loss'], var_list=variables['g_vars'])
d_clip = [v.assign(tf.clip_by_value(v, -args.d_clip_limit, args.d_clip_limit)) for v in variables['d_vars']]
with tf.control_dependencies([rms_d_optim]):
      rms_d_optim = tf.tuple(d_clip)
for epoch in range(100):
           for diter in range(10):
                       sess.run([rms_d_optim],feed_dict=feed)
                       sess.run(d_clip)
             sess.run([rms_g_optim],feed_dict=feed)

Could you suggest some direction for fixing this?
I went through your code (I am not well versed with PyTorch as of now), and it seems that you are also using same losses. Please correct me if I am mistaken.
Thanks