Question about parameter tuning

[rafaelblevin821]

Hi, I have a question about the parameter tuning in bindsnet.

In the Diehl and Cook model, are there any methods that were used to find the optimal values for 'Norm,' 'Learning rate,' etc? I have read the Diehl & Cook paper and the BindsNet paper but cannot find anything about optimizing the parameter values (unless I missed it).

What methods were used, even if the values were achieved through trial and error? How were the starting values and the range decided?

Any information on this would be extremely helpful. At the moment, I am looking into 'grid search' and 'Bayesian optimization' as methods to assist in optimizing the parameters.

@Hananel-Hazan @djsaunde

[Hananel-Hazan]

In addition to grid search and Bayesian optimization, I suggest considering genetic algorithms and particle swarm optimization for parameter tuning, as we used these methods in our papers. Regarding the parameters, I believe the norm used was the average number of black/white pixels in the images, although I cannot guarantee it. When selecting parameter ranges for tuning algorithms, it's common to rely on intuition, domain knowledge, and educated guesses. Feel free to experiment and adjust the parameter ranges based on your understanding of the problem domain and the model's behavior.
With that note, if you found set of pentameters that works better then what we use, feel free to let us know.

[rafaelblevin821]

@Hananel-Hazan thank you for this, it was very helpful. I am looking through papers and code with PSO, and GA optimization now. Are there any papers or code you would recommend having a look at?

[rafaelblevin821]

The norm value seems to be the number of image pixels (784)/10 (78.4)

[Hananel-Hazan]

We are working on a GA version that work well with Slurm HPC, @cearlUmass Do you think the code is ready for a test run?

[cearlUmass]

Not quite yet, but I am working on it. At the moment I have a prototype working on a local machine, but nothing for SLURM or Condor. I'm in the process of cleaning it up for a move to SLURM now. I was going to ask for your assistance moving it to SLURM anyhow, so I'll keep you up to date on its progress.

[rafaelblevin821]

Thanks @Hananel-Hazan , do you have any links that might be helpful or papers?

[Hananel-Hazan]

@rafaelblevin821 Please contact me via email (hananel@hazan.org.il) and we can discuss they way to move forward with GA algorithm.

[rafaelblevin821]

Hi @Hananel-Hazan, I shot you an email there earlier.

[rafaelblevin821]

Hi @Hananel-Hazan, I was just wondering if you have a working version of the GA for parameter tuning optimization yet?

[cearlUmass]

If by plug-and-play you mean "put the model in and run the GA to train it", then yes it is. You do need to define the parameters you want to tune, but other than that it's ready out of the box for any model with a fitness function that returns a singular float value.

The library only has a handful of dependencies, and Python 3.10+. Some people were having issues with the 'pip install -e .', but if that doesn't work manually installing the libraries shouldn't be difficult (See dependencies here https://github.com/cearlUmass/Distributed_GA/blob/master/pyproject.toml).

Working with it is straight forward. All you need are:

1. Model class which runs your model and returns the fitness value
2. Genome object which defines the parameter space you are searching in
3. An optimization algorithm (ie. a genetic algorithm)
4. A running method (parallelized, or synchronized)

1 and 2 you write yourself (not a lot of scripting), 3 and 4 are provided by DGA. The example script below shows how to do this.

https://github.com/cearlUmass/Distributed_GA/blob/master/scripts/Vector%20Estimate%20Benchmark/vector_estimator.py

• Model object here is called 'Testing_Model' defined in DGA/Model.py. You can look at that object to get an idea of how to make a Model class that runs your own script.
• Genome is filled with Gene's defining the shapes/behaviors of parameter vectors that fit into your model
• Algorithm is a generic genetic algorithm defined in DGA/Algorithm.py
• Synchronized running method selected. Just pass it your model and algorithm, use the run() function, and it will start the optimizer.

[rafaelblevin821]

Thank you Chris. I'll try it out this week.

[rafaelblevin821]

@cearlUmass have you tried running this with BindsNET on MNIST for example?

Do you have an example code?

[cearlUmass]

I don't currently have one working for MNIST, but this is actually next on my todo list. I need to create one for a publication I am trying to write, so I may have it done soon. It isn't conceptually too difficult if you would like to give it a try.

[cearlUmass]

Hello,

I've just added an MNIST script in the dev branch here: https://github.com/cearlUmass/Distributed_GA/tree/Dev-branch/scripts/MNIST

It's not clean, but hopefully you can make sense of it. First you will need to install PyTorch. Run main.py to run a simple genetic algorithm on the MNIST model (simple 1-layer convolutional model, capable of about 85% accuracy using backprop), and after it's finished running use results_plotter.py to plot the learning curve. At the moment, I can get a maximum of around 0.3 accuracy (30%) with 300 iterations of the GA. The GA used currently is pretty weak, I believe that's the primary issue.