It says TP=1 even though I set TP=2

[yurishin929]

Hi,

I ran the following code with TPDEGREE:-2 in run_gemini.sh,
and I can see the log + export TPDEGREE=2
but INFO colossalai - colossalai - INFO: Distributed environment is initialized, **data parallel size: 4, pipeline parallel size: 1, tensor parallel size: 1** too.
do I need to set something more? or does it work with TP even though it says data parallel size:4 / tensor parallel size: 1? Thank you.

https://github.com/hpcaitech/ColossalAI/blob/main/examples/language/gpt/gemini/run_gemini.sh
++
when I set TP=2, there's ring-allreduce kernel on Tensorboard, but TP=1, there's only AllGather, ReduceScatter. no ring-allreduce.

[yurishin929]

This is my run_gemini.sh

set -x
# distplan in ["CAI_ZeRO1", "CAI_ZeRO2", "CAI_Gemini", "Pytorch_DDP", "Pytorch_ZeRO"]
export DISTPLAN=${DISTPLAN:-"CAI_Gemini"}

# The following options only valid when DISTPLAN="colossalai"
export GPUNUM=${GPUNUM:-4}
export TPDEGREE=${TPDEGREE:-2}
export PLACEMENT=${PLACEMENT:-"cpu"}
export USE_SHARD_INIT=${USE_SHARD_INIT:-False}
export BATCH_SIZE=${BATCH_SIZE:-16}
export MODEL_TYPE=${MODEL_TYPE:-"gpt2_medium"}
export TRAIN_STEP=${TRAIN_STEP:-10}
# export PYTHONPATH=$PWD:$PYTHONPATH

if [ ${USE_SHARD_INIT} = "True" ]; then
  USE_SHARD_INIT="--shardinit"
else
  USE_SHARD_INIT=""
fi

mkdir -p gemini_logs

nsys profile -t cuda -d 50 torchrun --standalone --nproc_per_node=${GPUNUM} ./train_gpt_demo.py \
--tp_degree=${TPDEGREE} \
--model_type=${MODEL_TYPE} \
--batch_size=${BATCH_SIZE} \
--placement=${PLACEMENT} \
${USE_SHARD_INIT} \
--distplan=${DISTPLAN} \
--train_step=${TRAIN_STEP} \
2>&1 | tee ./gemini_logs/${MODEL_TYPE}_${DISTPLAN}_gpu_${GPUNUM}_bs_${BATCH_SIZE}_tp_${TPDEGREE}_${PLACEMENT}.log

I'm using colossalai 0.2.3, NVIDIA TITAN XP, cuda 11.6

[1SAA]

Hi @yurishin929

The output from INFO is about the distributed communication environment. It does not affect your parallelism if you are not using models in Titans. Briefly, it's normal and you do not need to care about that.