Model structure

[electronic-river]

Hello, I noticed that when running your model for prediction, the resulting output is concatenated with the optical imagery, followed by a lambda operation. In the end, the output shape of the model is (16, 27, 128, 128) instead of (None, 13, 128, 128). Is there any issue with this?

[ameraner]

Hi :) Those extra concatenate and lambda operations come from here:

dsen2-cr/Code/dsen2cr_network.py

Lines 53 to 66 in 9d6cdcf

	if use_cloud_mask:
	# the hacky trick with global variables and with lambda functions is needed to avoid errors when
	# pickle saving the model. Tensors are not pickable.
	# This way, the Lambda function has no special arguments and is "encapsulated"
	shape_n = tf.shape(input_opt)
	def concatenate_array(x):
	global shape_n
	return K.concatenate([x, K.zeros(shape=(batch_per_gpu, 1, shape_n[2], shape_n[3]))], axis=1)
	x = Concatenate(axis=1)([x, input_opt])
	x = Lambda(concatenate_array)(x)

Iirc, I had to add those extra concatenated K.zeros "placeholders" so that the output shape from the model would match with the shape of the target (this was a prerequisite of Keras). The target included an extra layer for the cloud mask, as I needed it to be able to compute the various losses metrics in https://github.com/ameraner/dsen2-cr/blob/main/Code/tools/image_metrics.py (this is also why I'm including the original optical input btw). Since I still need to compute the losses also when predicting (e.g. during the test phase), this is also implemented for the predict step.

The first dimension, 16 for you, should be the batch_per_gpu dimension, defining how many input images you send to the GPU in parallel.

[electronic-river]

When using the model for prediction, if the input size is 500x1000, and the model's input and output are 128x128, I observed in the source code that the program randomly crops the input size to 128x128 for prediction. If I want to predict on the entire 500x1000 image, what should I do?

[ameraner]

The data augmentation and random cropping are deactivated for the prediction step. However, it looks like I was supporting only square crops... so I think, atm you could change the crop_size to 500, and you would get the first half of your image predicted. With a little code modification you should probably be able to tile the input image and get a full predicted image as output.

[electronic-river]

Hello, sorry to bother you amidst your busy schedule. Due to certain reasons related to the performance, I am currently trying to use your model and its parameters to train my own dataset (consisting of Sentinel-1 and Sentinel-2 images downloaded from Google Earth Engine). I am following your approach to divide the dataset into 256x256 patches. However, during training, I have observed that the output loss becomes NaN. Could you please provide guidance on how to resolve this issue? Thank you.

[ameraner]

Hmm, that's odd, in my setup the trainings were always stable... did you tune/change the learning rate maybe?