CNN Architectures Trained With CIFAR 10

Outline:

1. ResNet
2. VGG-16

ResNet

ResNet-20 and ResNet-50 are represented in the following table with N=3 for ResNet-20 and N=8 for ResNet-50.

input: 3x32x32 (RGB image)
Convolutional Layer				kernel: 3x16x3x3	stride=1; padding=1
Batch Normalization				features: 16
ReLU
First Group	(N)x	Basic Block	Convolutional Layer	kernel: 16x16x3x3	stride=1; padding=1
			Batch Normalization	features: 16
			ReLU
			Convolutional Layer	kernel: 16x16x3x3	stride=1; padding=1
			Batch Normalization	features: 16
			ReLU
Second Group	1x	Basic Block	Convolutional Layer	kernel: 16x32x3x3	stride=2; padding=1
			Batch Normalization	features: 32
			ReLU
			Convolutional Layer	kernel: 32x32x3x3	stride=1; padding=1
			Batch Normalization	features: 32
			(Downsample)	kernel: 16x32x1x1	stride=2; padding=0
			ReLU
	(N-1)x	Basic Block	Convolutional Layer	kernel: 32x32x3x3	stride=1; padding=1
			Batch Normalization	features: 32
			ReLU
			Convolutional Layer	kernel: 32x 3x3x32	stride=1; padding=1
			Batch Normalization	features: 32
			ReLU
Third Group	1x	Basic Block	Convolutional Layer	kernel: 32x64x3x3	stride=2; padding=1
			Batch Normalization	features: 64
			ReLU
			Convolutional Layer	kernel: 64x64x3x3	stride=1; padding=1
			Batch Normalization	features: 64
			(Downsample)	kernel: 32x64x1x1	stride=2; padding=0
			ReLU
	(N-1)x	Basic Block	Convolutional Layer	kernel: 64x64x3x3	stride=1; padding=1
			Batch Normalization	features: 64
			ReLU
			Convolutional Layer	kernel: 64x64x3x3	stride=1; padding=1
			Batch Normalization	features: 64
			ReLU
Average Pooling				kernel: 8x8	stride=8
reshape: 64x1x1 => 64x1
Fully Connected Layer				kernel: 64x10

Note: Different ResNet depths are available but not explored, for example, ResNet-56. The only criteria is to obey the constraint N=int((x-2)/6), where x is the number of layers the user intends to use, i.e., N must be an integer. The final number of effective layers will be l=N*6+2.

VGG-16

input: 3x32x32 (RGB image)
Convolutional Layer	kernel: 3x64x3x3	stride=1; padding=1
Batch Normalization	features: 64
ReLU
Convolutional Layer	kernel: 64x64x3x3	stride=1; padding=1
Batch Normalization	features: 64
ReLU
Max Pooling	kernel: 2x2	stride=2
Convolutional Layer	kernel: 64x128x3x3	stride=1; padding=1
Batch Normalization	features: 128
ReLU
Convolutional Layer	kernel: 128x128x3x3	stride=1; padding=1
Batch Normalization	features: 128
ReLU
Max Pooling	kernel: 2x2	stride=2
Convolutional Layer	kernel: 128x256x3x3	stride=1; padding=1
Batch Normalization	features: 256
ReLU
Convolutional Layer	kernel: 256x256x3x3	stride=1; padding=1
Batch Normalization	features: 256
ReLU
Convolutional Layer	kernel: 256x256x3x3	stride=1; padding=1
Batch Normalization	features: 256
ReLU
Max Pooling	kernel: 2x2	stride=2
Convolutional Layer	kernel: 256x512x3x3	stride=1; padding=1
Batch Normalization	features: 512
ReLU
Convolutional Layer	kernel: 512x512x3x3	stride=1; padding=1
Batch Normalization	features: 512
ReLU
Convolutional Layer	kernel: 512x512x3x3	stride=1; padding=1
Batch Normalization	features: 512
ReLU
Max Pooling	kernel: 2x2	stride=2
Convolutional Layer	kernel: 512x512x3x3	stride=1; padding=1
Batch Normalization	features: 512
ReLU
Convolutional Layer	kernel: 512x512x3x3	stride=1; padding=1
Batch Normalization	features: 512
ReLU
Convolutional Layer	kernel: 512x512x3x3	stride=1; padding=1
Batch Normalization	features: 512
ReLU
Max Pooling	kernel: 2x2	stride=2
Average Pooling	kernel: 1x1	stride=1
reshape: 512x1x1 => 512x1
Fully Connected layer	kernel: 512x4096
ReLU
Fully Connected layer	kernel: 4096x4096
ReLU
Fully Connected layer	kernel: 4096x10

Note: VGG-11, VGG-13, and VGG-19 are also implemented in code but not yet explored.