About the training time

[SpaceLearner]

Hi, thank you for your excellent job. However, when I run the code using the default setting for pretraining on OAG_CS, the time taken for an epoch is much longer than you reported. It takes around 40 minutes for an epoch and 40 * 400 / 60 = 266.7 hours for 400 epochs, which is much longer than 12 hours in the paper. My machine is Tesla P100 and 8 * Xeon(R) CPU E5-2690 v4. How can I solve this problem?

The following is the log

+--------------------+-------------------------------------------+
| Parameter | Value |
+--------------------+-------------------------------------------+
| attr_ratio | 0.500 |
+--------------------+-------------------------------------------+
| attr_type | text |
+--------------------+-------------------------------------------+
| neg_samp_num | 255 |
+--------------------+-------------------------------------------+
| queue_size | 256 |
+--------------------+-------------------------------------------+
| w2v_dir | /data/data0/gjy/dataset/OAG/w2v_all |
+--------------------+-------------------------------------------+
| data_dir | /data/data0/gjy/dataset/OAG/graph_CS.pk |
+--------------------+-------------------------------------------+
| pretrain_model_dir | /data/data0/gjy/GPT-GNN/saved/OAG/gnn.pkl |
+--------------------+-------------------------------------------+
| cuda | 7 |
+--------------------+-------------------------------------------+
| sample_depth | 3 |
+--------------------+-------------------------------------------+
| sample_width | 128 |
+--------------------+-------------------------------------------+
| conv_name | hgt |
+--------------------+-------------------------------------------+
| n_hid | 400 |
+--------------------+-------------------------------------------+
| n_heads | 8 |
+--------------------+-------------------------------------------+
| n_layers | 3 |
+--------------------+-------------------------------------------+
| prev_norm | 1 |
+--------------------+-------------------------------------------+
| last_norm | 1 |
+--------------------+-------------------------------------------+
| dropout | 0.200 |
+--------------------+-------------------------------------------+
| max_lr | 0.001 |
+--------------------+-------------------------------------------+
| scheduler | cycle |
+--------------------+-------------------------------------------+
| n_epoch | 200 |
+--------------------+-------------------------------------------+
| n_pool | 8 |
+--------------------+-------------------------------------------+
| n_batch | 32 |
+--------------------+-------------------------------------------+
| batch_size | 256 |
+--------------------+-------------------------------------------+
| clip | 0.500 |
+--------------------+-------------------------------------------+
cuda:7
Start Loading Graph Data...
Finish Loading Graph Data!
paper PP_cite
paper rev_PP_cite
venue rev_PV_Conference
venue rev_PV_Journal
field rev_PF_in_L3
field rev_PF_in_L1
field rev_PF_in_L2
field rev_PF_in_L4
author AP_write_last
author AP_write_other
author AP_write_first
Start Pretraining...
Data Preparation: 68.7s
Epoch: 1, (1 / 41) 45.3s LR: 0.00005 Train Loss: (4.773, 9.771) Valid Loss: (4.762, 8.815) NDCG: 0.314 Norm: 20.012 queue: 1
UPDATE!!!
Data Preparation: 57.1s
Epoch: 1, (2 / 41) 40.3s LR: 0.00005 Train Loss: (4.594, 8.514) Valid Loss: (4.532, 7.968) NDCG: 0.353 Norm: 20.025 queue: 1
UPDATE!!!
Data Preparation: 29.7s
Epoch: 1, (3 / 41) 38.4s LR: 0.00006 Train Loss: (4.469, 7.768) Valid Loss: (4.628, 7.167) NDCG: 0.359 Norm: 20.035 queue: 1
UPDATE!!!
Data Preparation: 17.0s
Epoch: 1, (4 / 41) 36.8s LR: 0.00006 Train Loss: (4.426, 7.283) Valid Loss: (4.453, 6.991) NDCG: 0.367 Norm: 20.043 queue: 1
UPDATE!!!
Data Preparation: 13.0s
Epoch: 1, (5 / 41) 36.8s LR: 0.00007 Train Loss: (4.375, 7.060) Valid Loss: (4.509, 6.793) NDCG: 0.365 Norm: 20.047 queue: 1
UPDATE!!!
Data Preparation: 12.3s