Skip to content

Latest commit

 

History

History

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..

fairseq_moe

fairseq_moe

 
 

README.md

README.md

 
 

__init__.py

__init__.py

 
 

figure.svg

figure.svg

 
 

helloworld.py

helloworld.py

 
 

helloworld_amp.py

helloworld_amp.py

 
 

helloworld_custom_expert.py

helloworld_custom_expert.py

 
 

helloworld_custom_expert_sharded.py

helloworld_custom_expert_sharded.py

 
 

helloworld_ddp.py

helloworld_ddp.py

 
 

helloworld_ddp_tutel.py

helloworld_ddp_tutel.py

 
 

helloworld_from_scratch.py

helloworld_from_scratch.py

 
 

helloworld_switch.py

helloworld_switch.py

 
 

moe_cifar10.py

moe_cifar10.py

 
 

moe_mnist.py

moe_mnist.py

 
 

nccl_all_gather_v.py

nccl_all_gather_v.py

 
 

nccl_all_to_all_v.py

nccl_all_to_all_v.py

 
 

README.md

# Ex1 - helloworld/helloworld_ddp: running Standard MoE using 8 GPUs, with each GPU managing 2 experts:
$ mpiexec -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.helloworld --batch_size=16 --num_local_experts=2

# Ex2 - helloworld_sharded_experts: running MoE with shared expert slices using 8 GPUs, with each expert managed by 2 GPUs:
$ mpiexec -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.helloworld --batch_size=16 --num_local_experts=-2

# Ex3 - helloworld_megatron: running Dense FFN using 8 GPUs, with all 8 GPUs managing a unique expert:
$ mpiexec -host localhost -x LOCAL_SIZE=8 python3 -m tutel.launcher.run -m tutel.examples.helloworld --batch_size=16 --num_local_experts=-8