stuck at random accuracy for larq converted model

[prapti1998]

Hi,

I am training an Xception net architecture with larq quantised layers.
I am trying to use separable conv2d operations , but when I use the following configuration, the loss is not decreasing.
kwargs_d= dict(input_quantizer="ste_sign", pointwise_quantizer="ste_sign") larq.layers.QuantSeparableConv2D(256, (3, 3), padding='same', use_bias=False,**kwargs_d,pad_values=1.0, name='block3_sepconv1')

Also for the conv2d layers I am using the following config:
kwargs = dict(input_quantizer="ste_sign", kernel_quantizer="ste_sign", kernel_constraint="weight_clip") larq.layers.QuantConv2D(64, (3, 3), use_bias=False, **kwargs,name='block1_conv2')

The snippet of logs are attached( the loss is oscillating and the accuracy is stuck at a random value):
logs_correct.txt.txt

Also, I debugged my larq model more, and found that for the same input image the larq model's forward pass is giving "all zero(0.0)" output where as for the keras model (without larq) for forward pass there is a non-zero output.

Can anyone please tell me if there is something I am doing wrong or if I have missed anything?

I am using tf2.3 and the latest version of larq.

Thanks

[AdamHillier]

@prapti1998 could you perhaps include a detailed model summary of the exact model you're using (the Larq version)? lq.models.summary(model) should generate that for you.

A fairly common mistake that leads to constant output with Larq models is when you have an output of a ReLU layer fed into a Larq binary layer. In this case, since the output of a ReLU is >= 0, the input is quantised to the constant 1.0.

[prapti1998]

@AdamHillier thanks for pointing out the information about relu layer. I had initially changed my architecture and kept relu before batchnorm as I had read it herehere, but had left one relu before a binary layer by mistake. Now I am not getting all zero values in the forward pass, but still the validation accuracy is stuck at 47.53% and the loss is not decreasing with the larq model.

Please find attached the larq model summary.
larq_model_summary.txt

Thanks

[AdamHillier]

Are you doing image classification? In which case, how many classes are to using (to put the 47.5% accuracy in context)? It's a bit hard to tell from your model summary, but in general it's advisable to not binary quantise the first layer or final layer of a image classification network, as doing so can dramatically hurt performance.

As a general strategy, I would recommend trying to work out which quantised layer is causing the problem by starting with your unquantised model, and step-by-step quantising various parts of your model and testing the training, so that you can find the part/layer that causes dramatic accuracy loss when quantised.

[prapti1998]

hi @AdamHillier . Yes, I am doing image classification, with only 2 classes. I tried both of the things that you have suggested.

When I quantize only one conv2d layer also in the whole network which is not the first or final layer( the other layers in the whole architecture being the non-quantized keras version), then also it is getting stuck at the same accuracy over the epochs.

[saman-aghazadeh]

Hi @AdamHillier . We have tried the techniques you mentioned above, but still cannot make the training phase to show some reasonable accuracy and loss values. What is your suggestion at this phase? How do you think we should approach debugging this issue?

Thanks.

[saman-aghazadeh]

Can you please elaborate a bit more on the definition of shortcut blocks? Appreciate it!

[rnusselder]

Have a look at https://docs.larq.dev/larq/guides/bnn-architecture/#high-precision-shortcuts.
If you would like to use BNNs in your product, it would be best to reach out to: https://plumerai.com/contact_us. Plumerai is the company that develops Larq and we provide highly efficient and accurate BNN models to customers.

[saman-aghazadeh]

Thanks @rnusselder . Quick question from @Tombana . Upon reading the recommended papers in the documentations sections, it looks like simply replacing convolutions with quant versions, plus adding bypass layers, most probably would not lead into a good accuracy target, due to information loss. In fact, one need to make changes into the model as well, to eliminate information loss as much as possible.

What are your thoughts on it? Do you confirm my above argument?

Thanks.

[Tombana]

It is nearly impossible to say this with certainty without trying it out, but it is indeed possible that you have to change some model parameters to keep high accuracy.

[saman-aghazadeh]

Gotcha. Sounds like a very tricky process to train BNNs, which definitely makes sense.