forked from cs5220-f20/shallow-water
-
Notifications
You must be signed in to change notification settings - Fork 0
/
profiling_1000_t2.txt
4070 lines (3906 loc) · 311 KB
/
profiling_1000_t2.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Reading Profile files in profile.*
NODE 0;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.162 1 1 162162934 .TAU application
100.0 0.914 2:42.157 1 3 162157275 main
97.7 3 2:38.395 1 211 158395774 run_sim
74.5 0.049 2:00.827 50 50 2416548 central2d_run
74.5 2 2:00.827 50 5382 2416547 central2d_xrun
73.2 195 1:58.729 2392 104785 49636 central2d_step
38.1 428 1:01.722 2392 1.80835E+06 25804 central2d_predict
36.9 59,870 59,899 1.72942E+06 50022 35 limited_deriv1
35.5 57,527 57,557 1.72942E+06 49979 33 limited_derivk
35.0 507 56,691 2392 1.75048E+06 23700 central2d_correct
11.4 1 18,425 51 4131 361291 gather_sol
11.2 4 18,182 4080 8160 4457 recv_full_u
11.0 17,831 17,841 4131 62369 4319 copy_u
8.4 13,582 13,582 51 0 266331 solution_check
3.4 5,498 5,498 51 0 107823 viz_frame
1.9 3,132 3,132 1 0 3132932 MPI_Init()
1.2 1,995 1,995 1196 0 1669 MPI_Allreduce()
0.4 627 627 1 0 627655 MPI_Finalize()
0.4 578 578 4080 0 142 MPI_Recv()
0.1 35 120 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 4 85 598 16744 142 central2d_periodic
0.1 85 85 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 68 68 2392 0 29 MPI_Sendrecv()
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 40 40 1 0 40006 viz_close
0.0 21 21 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 15 15 100001 0 0 central2d_offset [THROTTLED]
0.0 9 14 1 37632 14859 lua_init_sim
0.0 0.541 14 1196 1196 12 shallow2d_speed
0.0 14 14 1196 0 12 shallow2dv_speed
0.0 12 12 14352 0 1 copy_subgrid
0.0 1 1 2 0 868 central2d_free
0.0 0.774 0.774 1 0 774 viz_open
0.0 0.025 0.025 1 0 25 MPI_Barrier()
0.0 0.016 0.017 1 2 17 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 0, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 1;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.160 1 1 162160802 .TAU application
100.0 0.9 2:42.155 1 3 162155192 main
97.2 0.572 2:37.624 1 108 157624964 run_sim
96.5 0.058 2:36.503 50 50 3130061 central2d_run
96.5 2 2:36.503 50 5382 3130060 central2d_xrun
71.6 192 1:56.094 2392 104785 48534 central2d_step
37.3 419 1:00.458 2392 1.80835E+06 25275 central2d_predict
35.5 57,569 57,598 1.72942E+06 49979 33 limited_derivk
35.3 57,213 57,242 1.72942E+06 50022 33 limited_deriv1
34.1 502 55,326 2392 1.75048E+06 23130 central2d_correct
22.9 3 37,054 598 16744 61963 central2d_periodic
22.8 37,037 37,037 2392 0 15484 MPI_Sendrecv()
2.1 3,330 3,330 1196 0 2785 MPI_Allreduce()
1.9 3,132 3,132 1 0 3132668 MPI_Init()
0.9 1,396 1,396 1 0 1396660 MPI_Finalize()
0.5 742 742 1 0 742171 MPI_Barrier()
0.2 0.057 364 51 51 7144 gather_sol
0.2 0.042 364 51 51 7143 send_full_u
0.2 364 364 51 0 7142 MPI_Send()
0.1 33 116 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 82 82 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 57 100001 100001 1 limdiff [THROTTLED]
0.0 0.503 22 1196 1196 19 shallow2d_speed
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 21 21 1196 0 18 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14669 lua_init_sim
0.0 12 12 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.11 0.11 2 0 55 central2d_free
0.0 0.019 0.02 1 2 20 central2d_init
0.0 0.008 0.008 1 0 8 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 1, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 2;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.152 1 1 162152798 .TAU application
100.0 0.898 2:42.147 1 3 162147147 main
97.2 0.47 2:37.629 1 108 157629318 run_sim
96.4 0.056 2:36.316 50 50 3126331 central2d_run
96.4 2 2:36.316 50 5382 3126330 central2d_xrun
72.1 194 1:56.890 2392 104785 48867 central2d_step
37.8 416 1:01.295 2392 1.80835E+06 25625 central2d_predict
35.7 57,807 57,837 1.72942E+06 49979 33 limited_derivk
35.6 57,771 57,800 1.72942E+06 50022 33 limited_deriv1
34.1 500 55,282 2392 1.75048E+06 23111 central2d_correct
13.9 22,598 22,598 1196 0 18895 MPI_Allreduce()
10.4 3 16,802 598 16744 28098 central2d_periodic
10.4 16,787 16,787 2392 0 7018 MPI_Sendrecv()
1.9 3,129 3,129 1 0 3129981 MPI_Init()
0.9 1,386 1,386 1 0 1386950 MPI_Finalize()
0.5 734 734 1 0 734342 MPI_Barrier()
0.3 0.046 563 51 51 11042 gather_sol
0.3 0.049 563 51 51 11042 send_full_u
0.3 563 563 51 0 11041 MPI_Send()
0.1 34 118 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 84 84 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.531 22 1196 1196 19 shallow2d_speed
0.0 21 21 1196 0 18 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14695 lua_init_sim
0.0 11 11 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.08 0.08 2 0 40 central2d_free
0.0 0.017 0.019 1 2 19 central2d_init
0.0 0.008 0.008 1 0 8 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_rank()
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 2, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 3;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.158 1 1 162158865 .TAU application
100.0 0.913 2:42.153 1 3 162153299 main
97.2 0.519 2:37.633 1 108 157633837 run_sim
96.2 0.058 2:36.064 50 50 3121295 central2d_run
96.2 2 2:36.064 50 5382 3121293 central2d_xrun
71.6 197 1:56.117 2392 104785 48544 central2d_step
37.3 420 1:00.487 2392 1.80835E+06 25287 central2d_predict
35.5 57,557 57,587 1.72942E+06 49979 33 limited_derivk
35.3 57,226 57,255 1.72942E+06 50022 33 limited_deriv1
34.1 515 55,313 2392 1.75048E+06 23125 central2d_correct
14.1 22,801 22,801 1196 0 19065 MPI_Allreduce()
10.6 4 17,121 598 16744 28630 central2d_periodic
10.5 17,104 17,104 2392 0 7151 MPI_Sendrecv()
1.9 3,132 3,132 1 0 3132780 MPI_Init()
0.9 1,385 1,385 1 0 1385769 MPI_Finalize()
0.5 0.047 824 51 51 16162 gather_sol
0.5 0.039 824 51 51 16161 send_full_u
0.5 824 824 51 0 16160 MPI_Send()
0.4 729 729 1 0 729544 MPI_Barrier()
0.1 34 118 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 84 84 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 0.484 22 1196 1196 19 shallow2d_speed
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 22 22 1196 0 19 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14660 lua_init_sim
0.0 12 12 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.097 0.097 2 0 48 central2d_free
0.0 0.014 0.015 1 2 15 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 3, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 4;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.157 1 1 162157934 .TAU application
100.0 0.889 2:42.152 1 3 162152312 main
97.2 0.457 2:37.638 1 108 157638292 run_sim
96.1 0.055 2:35.841 50 50 3116822 central2d_run
96.1 1 2:35.841 50 5382 3116821 central2d_xrun
71.5 191 1:56.010 2392 104785 48500 central2d_step
37.3 416 1:00.423 2392 1.80835E+06 25261 central2d_predict
35.5 57,512 57,541 1.72942E+06 49979 33 limited_derivk
35.3 57,189 57,218 1.72942E+06 50022 33 limited_deriv1
34.1 503 55,279 2392 1.75048E+06 23110 central2d_correct
14.2 23,002 23,002 1196 0 19233 MPI_Allreduce()
10.4 3 16,801 598 16744 28096 central2d_periodic
10.4 16,786 16,786 2392 0 7018 MPI_Sendrecv()
1.9 3,129 3,129 1 0 3129687 MPI_Init()
0.9 1,383 1,383 1 0 1383444 MPI_Finalize()
0.7 0.054 1,057 51 51 20729 gather_sol
0.7 0.052 1,057 51 51 20728 send_full_u
0.7 1,057 1,057 51 0 20727 MPI_Send()
0.4 724 724 1 0 724746 MPI_Barrier()
0.1 34 115 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 81 81 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 0.495 23 1196 1196 20 shallow2d_speed
0.0 23 23 1196 0 19 shallow2dv_speed
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14675 lua_init_sim
0.0 11 11 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.079 0.079 2 0 40 central2d_free
0.0 0.016 0.017 1 2 17 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_rank()
0.0 0 0 1 0 0 MPI_Comm_size()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 4, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 5;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.151 1 1 162151109 .TAU application
100.0 0.908 2:42.145 1 3 162145534 main
97.2 0.477 2:37.642 1 108 157642771 run_sim
96.0 0.055 2:35.619 50 50 3112386 central2d_run
96.0 1 2:35.619 50 5382 3112385 central2d_xrun
71.5 191 1:55.975 2392 104785 48485 central2d_step
37.3 413 1:00.402 2392 1.80835E+06 25252 central2d_predict
35.5 57,494 57,523 1.72942E+06 49979 33 limited_derivk
35.3 57,166 57,195 1.72942E+06 50022 33 limited_deriv1
34.1 512 55,265 2392 1.75048E+06 23104 central2d_correct
14.1 22,822 22,822 1196 0 19082 MPI_Allreduce()
10.4 4 16,795 598 16744 28086 central2d_periodic
10.3 16,778 16,778 2392 0 7014 MPI_Sendrecv()
1.9 3,129 3,129 1 0 3129972 MPI_Init()
0.8 1,371 1,371 1 0 1371883 MPI_Finalize()
0.8 0.052 1,287 51 51 25253 gather_sol
0.8 0.052 1,287 51 51 25252 send_full_u
0.8 1,287 1,287 51 0 25251 MPI_Send()
0.4 719 719 1 0 719956 MPI_Barrier()
0.1 34 116 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 81 81 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 58 100001 100001 1 limdiff [THROTTLED]
0.0 0.456 23 1196 1196 20 shallow2d_speed
0.0 23 23 1196 0 20 shallow2dv_speed
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 15 1 37632 15028 lua_init_sim
0.0 12 12 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.078 0.078 2 0 39 central2d_free
0.0 0.017 0.018 1 2 18 central2d_init
0.0 0.008 0.008 1 0 8 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0.001 0.001 1 0 1 viz_open
0.0 0 0 1 0 0 MPI_Comm_rank()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 5, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 6;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.161 1 1 162161541 .TAU application
100.0 0.901 2:42.155 1 3 162155899 main
97.2 0.468 2:37.647 1 108 157647271 run_sim
95.8 0.047 2:35.394 50 50 3107885 central2d_run
95.8 1 2:35.394 50 5382 3107884 central2d_xrun
71.4 189 1:55.808 2392 104785 48415 central2d_step
37.2 415 1:00.296 2392 1.80835E+06 25208 central2d_predict
35.4 57,354 57,383 1.72942E+06 49979 33 limited_derivk
35.3 57,137 57,166 1.72942E+06 50022 33 limited_deriv1
34.0 515 55,206 2392 1.75048E+06 23080 central2d_correct
14.0 22,695 22,695 1196 0 18976 MPI_Allreduce()
10.4 3 16,867 598 16744 28206 central2d_periodic
10.4 16,851 16,851 2392 0 7045 MPI_Sendrecv()
1.9 3,129 3,129 1 0 3129740 MPI_Init()
0.9 0.055 1,522 51 51 29856 gather_sol
0.9 0.054 1,522 51 51 29854 send_full_u
0.9 1,522 1,522 51 0 29853 MPI_Send()
0.8 1,377 1,377 1 0 1377987 MPI_Finalize()
0.4 715 715 1 0 715159 MPI_Barrier()
0.1 34 116 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 82 82 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.507 21 1196 1196 18 shallow2d_speed
0.0 20 20 1196 0 17 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14669 lua_init_sim
0.0 12 12 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.089 0.089 2 0 44 central2d_free
0.0 0.014 0.015 1 2 15 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 6, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 7;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.163 1 1 162163791 .TAU application
100.0 0.911 2:42.158 1 3 162158207 main
97.2 0.586 2:37.651 1 108 157651809 run_sim
95.7 0.067 2:35.187 50 50 3103747 central2d_run
95.7 2 2:35.187 50 5382 3103746 central2d_xrun
71.6 191 1:56.173 2392 104785 48568 central2d_step
37.3 417 1:00.485 2392 1.80835E+06 25287 central2d_predict
35.6 57,634 57,663 1.72942E+06 49979 33 limited_derivk
35.3 57,232 57,261 1.72942E+06 50022 33 limited_deriv1
34.2 501 55,380 2392 1.75048E+06 23152 central2d_correct
13.9 22,500 22,500 1196 0 18813 MPI_Allreduce()
10.2 4 16,495 598 16744 27584 central2d_periodic
10.2 16,479 16,479 2392 0 6889 MPI_Sendrecv()
1.9 3,132 3,132 1 0 3132939 MPI_Init()
1.1 0.064 1,738 51 51 34089 gather_sol
1.1 0.041 1,738 51 51 34088 send_full_u
1.1 1,738 1,738 51 0 34087 MPI_Send()
0.8 1,372 1,372 1 0 1372548 MPI_Finalize()
0.4 710 710 1 0 710354 MPI_Barrier()
0.1 34 116 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 81 81 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 58 100001 100001 1 limdiff [THROTTLED]
0.0 21 21 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.487 16 1196 1196 13 shallow2d_speed
0.0 15 15 1196 0 13 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14862 lua_init_sim
0.0 11 11 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.063 0.063 2 0 32 central2d_free
0.0 0.02 0.02 1 2 20 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 viz_open
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 MPI_Comm_size()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 7, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 8;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.165 1 1 162165858 .TAU application
100.0 0.902 2:42.160 1 3 162160293 main
97.2 0.576 2:37.656 1 108 157656220 run_sim
95.5 0.068 2:34.802 50 50 3096042 central2d_run
95.5 2 2:34.802 50 5382 3096041 central2d_xrun
69.2 179 1:52.153 2392 104785 46887 central2d_step
37.0 420 1:00.010 2392 1.80835E+06 25088 central2d_predict
34.6 56,059 56,089 1.72942E+06 50032 32 limited_deriv1
33.8 54,792 54,822 1.72942E+06 49969 32 limited_derivk
32.0 509 51,853 2392 1.75048E+06 21678 central2d_correct
23.0 4 37,348 598 16744 62456 central2d_periodic
23.0 37,331 37,331 2392 0 15607 MPI_Sendrecv()
3.3 5,276 5,276 1196 0 4412 MPI_Allreduce()
1.9 3,129 3,129 1 0 3129653 MPI_Init()
1.3 0.042 2,134 51 51 41848 gather_sol
1.3 0.063 2,134 51 51 41847 send_full_u
1.3 2,134 2,134 51 0 41846 MPI_Send()
0.8 1,373 1,373 1 0 1373518 MPI_Finalize()
0.4 705 705 1 0 705570 MPI_Barrier()
0.1 33 110 100001 100001 1 shallow2d_flux [THROTTLED]
0.0 76 76 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 44 60 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.496 20 1196 1196 17 shallow2d_speed
0.0 20 20 1196 0 17 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 8 13 1 34944 13613 lua_init_sim
0.0 13 13 14352 0 1 copy_subgrid
0.0 5 5 34944 0 0 central2d_offset
0.0 0.082 0.082 2 0 41 central2d_free
0.0 0.016 0.017 1 2 17 central2d_init
0.0 0.009 0.009 1 0 9 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 8, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 9;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.160 1 1 162160490 .TAU application
100.0 0.893 2:42.154 1 3 162154899 main
97.2 0.608 2:37.660 1 108 157660460 run_sim
95.5 0.072 2:34.866 50 50 3097331 central2d_run
95.5 2 2:34.866 50 5382 3097329 central2d_xrun
73.2 186 1:58.665 2392 104785 49609 central2d_step
37.3 419 1:00.529 2392 1.80835E+06 25305 central2d_predict
36.3 58,856 58,885 1.72942E+06 49979 34 limited_derivk
36.1 58,504 58,533 1.72942E+06 50022 34 limited_deriv1
35.7 503 57,834 2392 1.75048E+06 24178 central2d_correct
21.3 4 34,602 598 16744 57864 central2d_periodic
21.3 34,585 34,585 2392 0 14459 MPI_Sendrecv()
1.9 3,132 3,132 1 0 3132852 MPI_Init()
1.3 0.065 2,077 51 51 40734 gather_sol
1.3 0.038 2,077 51 51 40733 send_full_u
1.3 2,077 2,077 51 0 40732 MPI_Send()
1.0 1,579 1,579 1196 0 1321 MPI_Allreduce()
0.8 1,360 1,360 1 0 1360694 MPI_Finalize()
0.4 701 701 1 0 701070 MPI_Barrier()
0.1 34 114 100001 100001 1 shallow2d_flux [THROTTLED]
0.0 79 79 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.533 16 1196 1196 14 shallow2d_speed
0.0 16 16 1196 0 14 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14702 lua_init_sim
0.0 13 13 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.087 0.087 2 0 44 central2d_free
0.0 0.016 0.017 1 2 17 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 9, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 10;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.158 1 1 162158293 .TAU application
100.0 0.903 2:42.152 1 3 162152661 main
97.2 0.527 2:37.664 1 108 157664914 run_sim
95.3 0.052 2:34.531 50 50 3090625 central2d_run
95.3 1 2:34.531 50 5382 3090624 central2d_xrun
71.5 191 1:56.022 2392 104785 48504 central2d_step
37.3 417 1:00.425 2392 1.80835E+06 25262 central2d_predict
35.5 57,498 57,527 1.72942E+06 49979 33 limited_derivk
35.3 57,200 57,229 1.72942E+06 50022 33 limited_deriv1
34.1 517 55,288 2392 1.75048E+06 23114 central2d_correct
21.9 4 35,563 598 16744 59470 central2d_periodic
21.9 35,546 35,546 2392 0 14860 MPI_Sendrecv()
1.9 3,129 3,129 1 0 3129519 MPI_Init()
1.8 2,922 2,922 1196 0 2443 MPI_Allreduce()
1.5 0.055 2,422 51 51 47507 gather_sol
1.5 0.041 2,422 51 51 47506 send_full_u
1.5 2,422 2,422 51 0 47506 MPI_Send()
0.8 1,357 1,357 1 0 1357325 MPI_Finalize()
0.4 695 695 1 0 695398 MPI_Barrier()
0.1 34 116 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 82 82 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.479 21 1196 1196 18 shallow2d_speed
0.0 21 21 1196 0 18 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14721 lua_init_sim
0.0 12 12 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.089 0.089 2 0 44 central2d_free
0.0 0.016 0.017 1 2 17 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0.001 0.001 1 0 1 viz_open
0.0 0 0 1 0 0 MPI_Comm_rank()
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 10, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 11;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.161 1 1 162161868 .TAU application
100.0 0.899 2:42.156 1 3 162156325 main
97.2 0.47 2:37.669 1 108 157669376 run_sim
95.2 0.055 2:34.419 50 50 3088390 central2d_run
95.2 2 2:34.419 50 5382 3088389 central2d_xrun
73.1 196 1:58.510 2392 104785 49544 central2d_step
38.0 416 1:01.569 2392 1.80835E+06 25740 central2d_predict
36.9 59,780 59,809 1.72942E+06 50022 35 limited_deriv1
35.4 57,397 57,426 1.72942E+06 49979 33 limited_derivk
34.9 519 56,625 2392 1.75048E+06 23673 central2d_correct
20.7 33,568 33,568 1196 0 28067 MPI_Allreduce()
1.9 3,129 3,129 1 0 3129932 MPI_Init()
1.6 0.049 2,544 51 51 49883 gather_sol
1.6 0.05 2,543 51 51 49882 send_full_u
1.6 2,543 2,543 51 0 49881 MPI_Send()
1.4 3 2,325 598 16744 3888 central2d_periodic
1.4 2,308 2,308 2392 0 965 MPI_Sendrecv()
0.8 1,356 1,356 1 0 1356118 MPI_Finalize()
0.4 690 690 1 0 690595 MPI_Barrier()
0.1 33 118 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 84 84 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14674 lua_init_sim
0.0 0.459 14 1196 1196 12 shallow2d_speed
0.0 13 13 1196 0 11 shallow2dv_speed
0.0 13 13 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.078 0.078 2 0 39 central2d_free
0.0 0.019 0.02 1 2 20 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 11, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 12;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.162 1 1 162162031 .TAU application
100.0 1 2:42.156 1 3 162156398 main
97.2 0.56 2:37.673 1 108 157673782 run_sim
95.0 0.054 2:34.089 50 50 3081790 central2d_run
95.0 2 2:34.089 50 5382 3081789 central2d_xrun
71.5 191 1:55.910 2392 104785 48457 central2d_step
37.2 418 1:00.349 2392 1.80835E+06 25230 central2d_predict
35.4 57,451 57,479 1.72942E+06 49979 33 limited_derivk
35.3 57,134 57,163 1.72942E+06 50022 33 limited_deriv1
34.1 518 55,252 2392 1.75048E+06 23099 central2d_correct
21.3 34,577 34,577 1196 0 28911 MPI_Allreduce()
2.2 4 3,577 598 16744 5983 central2d_periodic
2.2 3,557 3,557 2392 0 1487 MPI_Sendrecv()
1.9 3,129 3,129 1 0 3129345 MPI_Init()
1.8 0.061 2,882 51 51 56527 gather_sol
1.8 0.047 2,882 51 51 56526 send_full_u
1.8 2,882 2,882 51 0 56525 MPI_Send()
0.8 1,352 1,352 1 0 1352150 MPI_Finalize()
0.4 685 685 1 0 685770 MPI_Barrier()
0.1 34 116 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 82 82 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 57 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.516 21 1196 1196 18 shallow2d_speed
0.0 21 21 1196 0 18 shallow2dv_speed
0.0 15 15 14352 0 1 copy_subgrid
0.0 9 14 1 37632 14991 lua_init_sim
0.0 14 14 100001 0 0 xmin2s [THROTTLED]
0.0 5 5 37632 0 0 central2d_offset
0.0 0.089 0.089 2 0 44 central2d_free
0.0 0.013 0.014 1 2 14 central2d_init
0.0 0.008 0.008 1 0 8 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 12, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 13;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 7 2:42.163 1 1 162163896 .TAU application
100.0 0.909 2:42.156 1 3 162156657 main
97.2 0.435 2:37.678 1 108 157678408 run_sim
94.9 0.049 2:33.857 50 50 3077157 central2d_run
94.9 1 2:33.857 50 5382 3077156 central2d_xrun
71.3 189 1:55.661 2392 104785 48353 central2d_step
37.2 416 1:00.248 2392 1.80835E+06 25187 central2d_predict
35.3 57,271 57,300 1.72942E+06 49979 33 limited_derivk
35.2 57,073 57,102 1.72942E+06 50022 33 limited_deriv1
34.0 513 55,107 2392 1.75048E+06 23038 central2d_correct
21.3 34,492 34,492 1196 0 28840 MPI_Allreduce()
2.3 3 3,679 598 16744 6154 central2d_periodic
2.3 3,661 3,661 2392 0 1531 MPI_Sendrecv()
1.9 3,134 3,134 1 0 3134473 MPI_Init()
1.9 0.049 3,124 51 51 61264 gather_sol
1.9 0.035 3,124 51 51 61263 send_full_u
1.9 3,124 3,124 51 0 61262 MPI_Send()
0.8 1,342 1,342 1 0 1342867 MPI_Finalize()
0.4 680 680 1 0 680956 MPI_Barrier()
0.1 34 115 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 81 81 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 57 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.463 21 1196 1196 18 shallow2d_speed
0.0 21 21 1196 0 18 shallow2dv_speed
0.0 14 14 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14641 lua_init_sim
0.0 14 14 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.076 0.076 2 0 38 central2d_free
0.0 0.015 0.015 1 2 15 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 MPI_Comm_size()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 13, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 14;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.159 1 1 162159032 .TAU application
100.0 0.915 2:42.153 1 3 162153469 main
97.2 0.426 2:37.682 1 108 157682932 run_sim
94.8 0.046 2:33.721 50 50 3074440 central2d_run
94.8 1 2:33.721 50 5382 3074439 central2d_xrun
72.6 186 1:57.733 2392 104785 49220 central2d_step
38.3 419 1:02.069 2392 1.80835E+06 25949 central2d_predict
36.4 58,918 58,948 1.72942E+06 50022 34 limited_deriv1
35.5 57,495 57,524 1.72942E+06 49979 33 limited_derivk
34.1 517 55,363 2392 1.75048E+06 23145 central2d_correct
20.5 33,253 33,253 1196 0 27804 MPI_Allreduce()
2.0 0.044 3,269 51 51 64110 gather_sol
2.0 0.044 3,269 51 51 64109 send_full_u
2.0 3,269 3,269 51 0 64108 MPI_Send()
1.9 3,132 3,132 1 0 3132612 MPI_Init()
1.7 3 2,717 598 16744 4545 central2d_periodic
1.7 2,698 2,698 2392 0 1128 MPI_Sendrecv()
0.8 1,337 1,337 1 0 1337010 MPI_Finalize()
0.4 676 676 1 0 676154 MPI_Barrier()
0.1 34 113 100001 100001 1 shallow2d_flux [THROTTLED]
0.0 79 79 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 15 15 14352 0 1 copy_subgrid
0.0 0.427 15 1196 1196 13 shallow2d_speed
0.0 9 14 1 37632 14658 lua_init_sim
0.0 14 14 1196 0 12 shallow2dv_speed
0.0 5 5 37632 0 0 central2d_offset
0.0 0.076 0.076 2 0 38 central2d_free
0.0 0.018 0.019 1 2 19 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 14, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 15;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:42.162 1 1 162162160 .TAU application
100.0 0.919 2:42.156 1 3 162156521 main
97.2 0.551 2:37.687 1 108 157687530 run_sim
94.6 0.049 2:33.425 50 50 3068501 central2d_run
94.6 1 2:33.425 50 5382 3068500 central2d_xrun
71.5 194 1:55.930 2392 104785 48466 central2d_step
37.2 418 1:00.392 2392 1.80835E+06 25248 central2d_predict
35.5 57,489 57,518 1.72942E+06 49979 33 limited_derivk
35.2 57,126 57,155 1.72942E+06 50022 33 limited_deriv1
34.1 502 55,225 2392 1.75048E+06 23087 central2d_correct
21.2 34,360 34,360 1196 0 28729 MPI_Allreduce()
2.2 0.045 3,575 51 51 70112 gather_sol
2.2 0.036 3,575 51 51 70111 send_full_u
2.2 3,575 3,575 51 0 70110 MPI_Send()
1.9 3,129 3,129 1 0 3129423 MPI_Init()
1.9 3 3,110 598 16744 5202 central2d_periodic
1.9 3,093 3,093 2392 0 1293 MPI_Sendrecv()
0.8 1,338 1,338 1 0 1338649 MPI_Finalize()
0.4 671 671 1 0 671331 MPI_Barrier()
0.1 34 118 100001 100001 1 shallow2d_flux [THROTTLED]
0.1 84 84 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.475 21 1196 1196 18 shallow2d_speed
0.0 20 20 1196 0 17 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14795 lua_init_sim
0.0 13 13 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.088 0.088 2 0 44 central2d_free
0.0 0.012 0.014 1 2 14 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_rank()
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 15, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 16;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 6 2:41.168 1 1 161168065 .TAU application
100.0 0.884 2:41.161 1 3 161161900 main
97.8 0.445 2:37.692 1 108 157692012 run_sim
95.1 0.056 2:33.257 50 50 3065152 central2d_run
95.1 2 2:33.257 50 5382 3065151 central2d_xrun
72.8 186 1:57.270 2392 104785 49026 central2d_step
38.2 417 1:01.565 2392 1.80835E+06 25738 central2d_predict
36.2 58,361 58,390 1.72942E+06 50022 34 limited_deriv1
35.8 57,598 57,627 1.72942E+06 49979 33 limited_derivk
34.4 513 55,405 2392 1.75048E+06 23163 central2d_correct
20.9 33,736 33,736 1196 0 28208 MPI_Allreduce()
2.3 0.037 3,752 51 51 73576 gather_sol
2.3 0.048 3,752 51 51 73575 send_full_u
2.3 3,752 3,752 51 0 73574 MPI_Send()
1.4 3 2,235 598 16744 3738 central2d_periodic
1.4 2,218 2,218 2392 0 927 MPI_Sendrecv()
1.3 2,125 2,125 1 0 2125373 MPI_Init()
0.8 1,343 1,343 1 0 1343631 MPI_Finalize()
0.4 666 666 1 0 666512 MPI_Barrier()
0.1 34 113 100001 100001 1 shallow2d_flux [THROTTLED]
0.0 78 78 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14979 lua_init_sim
0.0 0.476 13 1196 1196 11 shallow2d_speed
0.0 13 13 1196 0 11 shallow2dv_speed
0.0 12 12 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.091 0.091 2 0 46 central2d_free
0.0 0.011 0.012 1 2 12 central2d_init
0.0 0.007 0.007 1 0 7 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 16, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 17;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 6 2:41.165 1 1 161165912 .TAU application
100.0 0.874 2:41.159 1 3 161159042 main
97.8 0.568 2:37.696 1 108 157696576 run_sim
94.8 0.055 2:32.781 50 50 3055621 central2d_run
94.8 1 2:32.780 50 5382 3055620 central2d_xrun
68.9 173 1:51.114 2392 104785 46452 central2d_step
35.8 419 57,756 2392 1.80835E+06 24146 central2d_predict
34.8 56,001 56,030 1.72942E+06 50032 32 limited_deriv1
33.4 53,824 53,853 1.72942E+06 49969 31 limited_derivk
32.9 509 53,078 2392 1.75048E+06 22190 central2d_correct
22.5 4 36,237 598 16744 60598 central2d_periodic
22.5 36,223 36,223 2392 0 15144 MPI_Sendrecv()
3.4 5,406 5,406 1196 0 4521 MPI_Allreduce()
2.6 0.045 4,238 51 51 83114 gather_sol
2.6 0.034 4,238 51 51 83113 send_full_u
2.6 4,238 4,238 51 0 83113 MPI_Send()
1.3 2,125 2,125 1 0 2125291 MPI_Init()
0.8 1,336 1,336 1 0 1336301 MPI_Finalize()
0.4 662 662 1 0 662034 MPI_Barrier()
0.1 34 105 100001 100001 1 shallow2d_flux [THROTTLED]
0.0 71 71 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 58 100001 100001 1 limdiff [THROTTLED]
0.0 21 21 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.52 20 1196 1196 17 shallow2d_speed
0.0 20 20 1196 0 17 shallow2dv_speed
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 8 14 1 34944 14012 lua_init_sim
0.0 10 10 14352 0 1 copy_subgrid
0.0 5 5 34944 0 0 central2d_offset
0.0 0.087 0.087 2 0 44 central2d_free
0.0 0.011 0.012 1 2 12 central2d_init
0.0 0.008 0.008 1 0 8 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 17, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 18;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 6 2:41.164 1 1 161164136 .TAU application
100.0 0.879 2:41.157 1 3 161157862 main
97.9 0.556 2:37.700 1 108 157700786 run_sim
94.9 0.052 2:32.932 50 50 3058640 central2d_run
94.9 1 2:32.931 50 5382 3058639 central2d_xrun
74.2 176 1:59.650 2392 104785 50021 central2d_step
39.7 423 1:04.031 2392 1.80835E+06 26769 central2d_predict
37.1 59,841 59,870 1.72942E+06 50022 35 limited_deriv1
36.3 58,510 58,539 1.72942E+06 49979 34 limited_derivk
34.3 509 55,334 2392 1.75048E+06 23133 central2d_correct
18.3 29,442 29,442 1196 0 24617 MPI_Allreduce()
2.5 0.051 4,095 51 51 80305 gather_sol
2.5 0.032 4,095 51 51 80304 send_full_u
2.5 4,095 4,095 51 0 80304 MPI_Send()
2.4 3 3,822 598 16744 6393 central2d_periodic
2.4 3,807 3,807 2392 0 1592 MPI_Sendrecv()
1.3 2,125 2,125 1 0 2125341 MPI_Init()
0.8 1,330 1,330 1 0 1330856 MPI_Finalize()
0.4 657 657 1 0 657852 MPI_Barrier()
0.1 33 108 100001 100001 1 shallow2d_flux [THROTTLED]
0.0 75 75 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 43 59 100001 100001 1 limdiff [THROTTLED]
0.0 22 22 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 15 15 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14685 lua_init_sim
0.0 0.5 14 1196 1196 12 shallow2d_speed
0.0 13 13 1196 0 11 shallow2dv_speed
0.0 11 11 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.09 0.09 2 0 45 central2d_free
0.0 0.011 0.012 1 2 12 central2d_init
0.0 0.008 0.008 1 0 8 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_size()
0.0 0 0 1 0 0 MPI_Comm_rank()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 18, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 19;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 5 2:41.163 1 1 161163806 .TAU application
100.0 0.879 2:41.157 1 3 161157901 main
97.9 0.538 2:37.705 1 108 157705355 run_sim
94.7 0.051 2:32.660 50 50 3053208 central2d_run
94.7 1 2:32.660 50 5382 3053207 central2d_xrun
73.6 184 1:58.572 2392 104785 49570 central2d_step
37.5 416 1:00.478 2392 1.80835E+06 25284 central2d_predict
36.5 58,812 58,841 1.72942E+06 49979 34 limited_derivk
36.3 58,457 58,486 1.72942E+06 50022 34 limited_deriv1
35.9 507 57,794 2392 1.75048E+06 24162 central2d_correct
19.4 31,317 31,317 1196 0 26186 MPI_Allreduce()
2.7 0.039 4,377 51 51 85825 gather_sol
2.7 0.04 4,377 51 51 85825 send_full_u
2.7 4,377 4,377 51 0 85824 MPI_Send()
1.7 3 2,751 598 16744 4601 central2d_periodic
1.7 2,737 2,737 2392 0 1144 MPI_Sendrecv()
1.3 2,125 2,125 1 0 2125189 MPI_Init()
0.8 1,326 1,326 1 0 1326478 MPI_Finalize()
0.4 652 652 1 0 652511 MPI_Barrier()
0.1 34 113 100001 100001 1 shallow2d_flux [THROTTLED]
0.0 79 79 100001 0 1 shallow2dv_flux [THROTTLED]
0.0 42 57 100001 100001 1 limdiff [THROTTLED]
0.0 21 21 100001 0 0 central2d_correct_sd [THROTTLED]
0.0 0.492 16 1196 1196 14 shallow2d_speed
0.0 16 16 1196 0 14 shallow2dv_speed
0.0 14 14 100001 0 0 xmin2s [THROTTLED]
0.0 9 14 1 37632 14665 lua_init_sim
0.0 10 10 14352 0 1 copy_subgrid
0.0 5 5 37632 0 0 central2d_offset
0.0 0.119 0.119 2 0 60 central2d_free
0.0 0.011 0.012 1 2 12 central2d_init
0.0 0.008 0.008 1 0 8 copy_basic_info
0.0 0.001 0.001 1 0 1 MPI_Comm_rank()
0.0 0 0 1 0 0 MPI_Comm_size()
0.0 0 0 1 0 0 viz_open
---------------------------------------------------------------------------------------
USER EVENTS Profile :NODE 19, CONTEXT 0, THREAD 0
---------------------------------------------------------------------------------------
NumSamples MaxValue MinValue MeanValue Std. Dev. Event Name
---------------------------------------------------------------------------------------
1196 4 4 4 0 Message size for all-reduce
---------------------------------------------------------------------------------------
NODE 20;CONTEXT 0;THREAD 0:
---------------------------------------------------------------------------------------
%Time Exclusive Inclusive #Call #Subrs Inclusive Name
msec total msec usec/call
---------------------------------------------------------------------------------------
100.0 6 2:41.163 1 1 161163927 .TAU application
100.0 0.865 2:41.157 1 3 161157568 main
97.9 0.437 2:37.709 1 108 157709790 run_sim
94.5 0.035 2:32.336 50 50 3046727 central2d_run
94.5 1 2:32.336 50 5382 3046727 central2d_xrun
72.0 185 1:56.067 2392 104785 48523 central2d_step
37.5 416 1:00.478 2392 1.80835E+06 25284 central2d_predict
35.7 57,577 57,606 1.72942E+06 49979 33 limited_derivk
35.5 57,171 57,200 1.72942E+06 50022 33 limited_deriv1