Lambda Mobilenets

Mobilenets with Lambda layers

Lambda Networks proposed in LambdaNetworks: Modeling Long Range Interactions without Attention.

Install

We use the implementation done by lucidrains

pip install lambda-networks

Method

We replace some layers in MobileNet-v1 architecture with lambda layer. It significantly reduces the parameters with some performance gain on cifar100 dataset.

MobileNet-v1 architecture:

The following table shows which layer to replace and remove to get the performance boost:

C: Conv layer same as in original architecture

L: Lambda layer

Blank cell represents that the layer is removed (replaced with identity layer)

	Layer number and Type
Id	0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14	Params (M)	Top-1 (%)
A1	C | C | C | C | C | C | C | C | C | C | C | C | C | C | FC	3.30	65.54
A2	C | C | C | C | C | C | C | C | | | | | L | C | FC	1.84	69.48
A3	C | C | C | C | C | C | C | C | L | C | L | C | L | C | FC	2.53	65.25
A4	C | C | C | C | C | C | C | C | L | | | | C | | FC	1.24	68.22
A5	C | C | C | C | C | C | L | | | | | | C | | FC	0.80	69.91
A6	C | C | C | | C | | L | | | | | | C | | FC	0.71	66.38

The above table shows significant gain in A5 configuration as compared to original configuration A1.

Citations

@inproceedings{
    anonymous2021lambdanetworks,
    title={LambdaNetworks: Modeling long-range Interactions without Attention},
    author={Anonymous},
    booktitle={Submitted to International Conference on Learning Representations},
    year={2021},
    url={https://openreview.net/forum?id=xTJEN-ggl1b},
    note={under review}
}

@article{DBLP:journals/corr/HowardZCKWWAA17,
     author    = {Andrew G. Howard and
                  Menglong Zhu and
                  Bo Chen and
                  Dmitry Kalenichenko and
                  Weijun Wang and
                  Tobias Weyand and
                  Marco Andreetto and
                  Hartwig Adam},
     title     = {MobileNets: Efficient Convolutional Neural Networks for Mobile Vision
                  Applications},
     journal   = {CoRR},
     volume    = {abs/1704.04861},
     year      = {2017},
     url       = {http://arxiv.org/abs/1704.04861},
     archivePrefix = {arXiv},
     eprint    = {1704.04861},
     timestamp = {Mon, 13 Aug 2018 16:46:35 +0200},
     biburl    = {https://dblp.org/rec/journals/corr/HowardZCKWWAA17.bib},
     bibsource = {dblp computer science bibliography, https://dblp.org}
   }