distributed-make

A distributed make using celery for task distribution.

1. Local installation
2. Running locally
3. Deploying on Grid'5000
4. Running on Grid'5000
5. Makefile options

Local installation ubuntu 14.04

• Install puppet:

sudo apt-get install puppet

• Install puppetlabs-apt:

sudo puppet module install --modulepath=/usr/share/puppet/modules puppetlabs-apt

• Apply the puppet manifest in deploy

sudo puppet apply local_intall.pp

Running locally

Assuming the following directory structure:

- src
  - celeryconfig.py
  - makeparse.py
  - logging.ini
  - master.py
  - result.py
  - work.py
  - Makefile
  - premier.c

• Create master_node with localhost as the content
• Start celery

celery worker -A work -l info 

• Launch the main program with:

python master.py -f Makefile premier

Deploying on Grid'5000

• Reserve nodes on Grid'5000 with the script reserve.sh in deploy directory.

./reserve.sh 4 1:30:00

for example, the commande below reserves 4 nodes for 1h30:

• Automatically provision the nodes with the script deploy.sh in deploy directory

./deploy.sh

This will create two files master_node and worker_nodes with respectively the addresses of the machines acting as the master and the workers.

Running on Grid'5000

Running the program on Grid'5000 assumes the deployment described earlier.

NOTE: The program uses nfs to share files between the nodes

Assuming the following directory structure:

- src
  - celeryconfig.py
  - makeparse.py
  - logging.ini
  - master.py
  - result.py
  - work.py
  - Makefile
  - premier.c
  - master_node
  - worker_nodes

• Start celery for all worker-node in worker_nodes

ssh root@worker-node
celery worker -A work -linfo 

• Launch the main program with:

python master.py -f Makefile premier

Makefile options

python master.py -h

displays the master.py command line option. By default, running python master.py without the -f switch will use GNU-Makefile or makefile or Makefile found in the current directory and will execute the first target in the makefile. The -a or --async option allows to run asynchronously all the tasks without blocking the master otherwise it will wait for the last task's completion.