nips2018.md

List of commands

These are the list of commands used to generate the data used in the workshop paper submitted at NIPS 2018.

In order to ensure to execute the proper version of the code, checkout to the following commit:

git checkout f96dfa09769261b0a8cee099b79b57f8593cc520

Fig 1A (hinge)

python train.py --log_dir R5L --dataset random --depth 5 --dim $h$ --width $h$ --p $P$ --optimizer adam0 --n_steps_max 2M

For some values of $P \in (500, 60\mathrm{k})$, start at large $h$ where we reach $N_\Delta = 0$ and decrease $h$ until $N_\Delta > 0.1N$.

Fig 1B

python train.py --log_dir R62d62h2L --dataset random --depth 2 --dim 62 --width 62 --p $P$ --optimizer adam0 --n_steps_max 10M
python train.py --log_dir R51d51h3L --dataset random --depth 3 --dim 51 --width 51 --p $P$ --optimizer adam0 --n_steps_max 10M
python train.py --log_dir R40d40h5L --dataset random --depth 5 --dim 40 --width 40 --p $P$ --optimizer adam0 --n_steps_max 10M

For $L=3$ ($d=51$, $h=51$) we ran 128 training varying $P$ from 21991 to 25918. For the value of $N$ we take $7854$ that correspond to the number of parameters minus the number of neurons, per neuron there is a degree of freedom lost in a symmetry induced by the homogeneity of the ReLU function. 37 of the runs have $N_\Delta = 0$, 74 have $N_\Delta > 0.4N$. Among the 19 remaining ones, 14 of them have $N_\Delta$ between 1 and 4, we think that these runs encounter numerical precision issues, we observed that using 32 bit precision accentuate this issue. We think that the 5 left with $4 < N_\Delta < 0.4 N$ has been stoped too early. The same observation apply for the other depths.

Fig 1C

python train.py --log_dir M10d1L --dataset mnist_pca --depth 1 --dim 10 --width $h$ --p $P$ --optimizer adam0 --n_steps_max 2M
python train.py --log_dir M10d3L --dataset mnist_pca --depth 3 --dim 10 --width $h$ --p $P$ --optimizer adam0 --n_steps_max 2M
python train.py --log_dir M10d5L --dataset mnist_pca --depth 5 --dim 10 --width $h$ --p $P$ --optimizer adam0 --n_steps_max 2M

For some values of $P \in (100, 50\mathrm{k})$, start at large $h$ where we reach $N_\Delta = 0$ and decrease $h$ until $N_\Delta > 0.1N$.

Fig 1D

python train.py --log_dir M10d30h5L --dataset mnist_pca --depth 5 --dim 10 --width 30 --p $P$ --optimizer adam0 --n_steps_max 3M

With $P$ varying from 31k to 68k (using trainset and testset of MNIST).

Fig 2

python train.py --log_dir M10d5L_gen --dataset mnist_pca --depth 5 --dim 10 --width $h$ --p $P$ --optimizer adam0 --n_steps_max 500k

where $P \in {10\mathrm{k}, 20\mathrm{k}, 50\mathrm{k}}$ and $h$ varies from 1 to 3k.