Support for 32-bit Raspberry Pi OS

[saman-aghazadeh]

I was recently working on deploying models using LCE on our 32-bit RPI system. I realized I'm not able to get any performance gain. It looks like it has been specifically mentioned that LCE currently supports 64-bit systems.

I'm happy to contribute into optimizations for 32-bit RPI machines. Was wondering if anyone can help me understand why current kernels do not perform well on 32-bit OS machine, and what are are the suggestions for developing new optimized kernels?

Thanks

[Tombana]

Hi @saman-aghazadeh,

Although we mainly focus on 64-bit ARM CPU's, LCE does have 32-bit optimized kernels courtesy of this community PR: #432.
Binary convolution layers with bitpacked output are not supported by this, so if your model has those then it will not run with optimized code, and we would happily welcome a contribution for this.

Could you share some more details about your model and about the latencies you obtain? Then we might be able to hunt down the problem.

Was wondering if anyone can help me understand why current kernels do not perform well on 32-bit OS machine

The code for matrix-matrix multiplication is written in assembly and uses instructions that are only available when the CPU is running in 64-bit mode. Even if a CPU supports 64-bit mode it can still run in 32-bit mode when a 32-bit OS is used.

compute-engine/larq_compute_engine/core/bgemm/kernels_aarch64.h

Lines 105 to 118 in 664aa67

	#define LCE_BMLA(Vd, Vr, Vl1, Vl2, Vl3, Vl4) \
	"eor v28.16b, " #Vr".16b, " #Vl1".16b\n" \
	"eor v29.16b, " #Vr".16b, " #Vl2".16b\n" \
	"eor v30.16b, " #Vr".16b, " #Vl3".16b\n" \
	"eor v31.16b, " #Vr".16b, " #Vl4".16b\n" \
	"cnt v28.16b, v28.16b\n" \
	"cnt v29.16b, v29.16b\n" \
	"cnt v30.16b, v30.16b\n" \
	"cnt v31.16b, v31.16b\n" \
	"addp v28.16b, v28.16b, v29.16b\n" \
	"addp v30.16b, v30.16b, v31.16b\n" \
	"addp v28.16b, v28.16b, v30.16b\n" \
	"uaddlp v28.8h, v28.16b\n" \
	"uadalp " #Vd".4s, v28.8h\n"

[saman-aghazadeh]

.

[saman-aghazadeh]

For gender model we are using the xceptionnet architecture

For the larq version I replaced the conv2d and the separableconv2d layers with quant larq versions( For the exact code see attached file).

Benchmarks for 32 bit 4-core raspberry pi (CPU supports 64-bit, but the OS is 32-bit Raspbian):

1. Original keras version( without any optimisation): 0.3 seconds on average per image
2. with larq trained binary network+ larq compute engine tflite buffer: 0.5 seconds on average per image
3. with larq trained BNN( without larq compute engine): 0.6 seconds on average per image

[Tombana]

LCE does not support binary depthwise layers. So when creating a QuantSeparableConv2D layer, then pointwise_quantizer can be ste_sign but depthwise_quantizer should be None.

Its also possible that LCE can't run the optimized version because of padding. To use the optimized kernels, you must set pad_values = 1 in the quantized layers.

For more info on it, see this page: https://docs.larq.dev/compute-engine/model_optimization_guide/

I recommend opening the converted .tflite file in https://netron.app/ to see if the layers did get converted to LCE layers.

For the exact code see attached file

I don't see an attached file. Is it possible for you to share just the snippet of a single seperable conv2d layer?

[Tombana]

Hi @saman-aghazadeh,
Did this solve your issue?

[saman-aghazadeh]

Hi @Tombana .

Before we get into the deployment, we realized we got stuck in the training phase, where we cannot see any improvement in the accuracy. We are trying to fix that issue, before any deployment.

[prapti1998]

Hi @Tombana . I am also trying to use separable conv2d operations , but when I use the following configuration, the loss is not decreasing.
kwargs_d= dict(input_quantizer="ste_sign", pointwise_quantizer="ste_sign")
larq.layers.QuantSeparableConv2D(256, (3, 3), padding='same', use_bias=False,**kwargs_d,pad_values=1.0, name='block3_sepconv1')

Also for the conv2d layers I am using the following config:
kwargs = dict(input_quantizer="ste_sign", kernel_quantizer="ste_sign", kernel_constraint="weight_clip")
larq.layers.QuantConv2D(64, (3, 3), use_bias=False, **kwargs,name='block1_conv2')

The snippet of logs are attached( the loss is oscillating and the accuracy is stuck at a random value):
logs_correct.txt.txt

[prapti1998]

@Tombana I debugged my larq model more, and found that for the same input image the larq model's forward pass is giving "all zero(0.0)" output where as for the keras model (without larq) for forward pass there is a non-zero output.

[prapti1998]

I am attaching the exact training script
github_upload.txt

[Tombana]

Hi @prapti1998,
It seems that your question is about training and not about conversion to TFLite which is what this discussion was originally about. Could you open a discussion or issue on the larq repository instead: https://github.com/larq/larq ?

[prapti1998]

yes sure. Thanks @Tombana