HPC-Geo-Data-Processing

COMP90024 Cluster and Cloud Computing - Assignment 1 - 2020S1

Contributors

Olivia 💻

XuLinYang 💻

Repository Structure

| /data 
      - project data
  /docs 
      - documentations
  /slurm
      - slurm scripts
  /src
      - source codes
  /submit
      - files for the assignment submission
  .all-contributorsrc 
      - all contributers infomation
   requirements.txt
      - required python package

how to run

1. pip install -r requirements.txt
2. run program on computer

cd src
mpiexec -n numprocs python -m mpi4py pyfile
mpiexec -n 8 python -m mpi4py main.py -country ./language.json -data ../data/testTwitter.json
mpiexec -n 8 python -m mpi4py main.py -country ./language.json -data ../data/smallTwitter.json

3. run task on spartan

cd slurm
sbatch 1node1core.slurm > 1node1core-physical.out
sbatch 1node4core.slurm > 1node4core-physical.out
sbatch 1node8core.slurm > 1node8core-physical.out
sbatch 1node16core.slurm > 1node16core-physical.out
sbatch 1node24core.slurm > 1node24core-physical.out
sbatch 2node8core.slurm > 2node8core-physical.out

sbatch 1node1core-snowy.slurm > 1node1core-snowy.out
sbatch 1node8core-snowy.slurm > 1node8core-snowy.out
sbatch 2node8core-snowy.slurm > 2node8core-snowy.out

sbatch 1node8core-cloud.slurm > 1node8core-cloud.out
sbatch 2node8core-cloud.slurm > 2node8core-cloud.out

spartan

see jobs under execution

squeue -u xuliny

Spartan Weather Report

spartan_weather

final results

And I ask Rechard Snott why we lost 0.5 mark, in email he replied: