Replies: 1 comment
-
I am not sure if I correctly understand your question but let me share a few thoughts regarding batch normalisation:
Let me know if that helps or whether you have any follow-up questions |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am trying to understand the BinaryNet tutorial by dividing the model to many matrix operation sequentially to check the result of each layer.
"https://docs.larq.dev/larq/tutorials/binarynet_cifar10/"
Divided into layers
In the tutorial, the trained weight are quantized for the filters, and it keeps real values for the parameters of the batch normalization (BN) layer. For example, the parameters ( the mean values, standard deviation values, and beta, ... etc ) of the BN layer are all presented with real number.
Which means that the operation of convolution operation, dense layer operation can use the quantized matrix-multiplier operator (e.q. XNOR or ternary operation). And it still needs some real-number operator for the BN layer operation.
I know that we might fuse the BN layer into the followed convolution layer or dense layer, and then quantize the fused convolution (or dense) layer, but it still has some biases or shifts left after fuse the BN layer and convolution layer.
Can we either keep the BN layers and quantize all the calculations into binarized operations or remove the BN layers and keeps good training convergency and accuracy?
Beta Was this translation helpful? Give feedback.
All reactions