TreePiece replication #164
Conversation
…plica group. The requests for a key will be sent to the corresponding TreePieceReplica instead of a tree piece. This is done to prevent the case when one tree piece is getting many requests and that becomes the bottleneck
…quired information from the group instead of the tree piece
This brings tp_replication up to date with master after over a decade of changes. N.B. it does not compile because of charm interface changes over that decade.
This is not production code. But it might work for testing the concept.
|
I ran some tests on the Lamb 80 million and works for some number of tree pieces(128,1024,4096) but breaks for others (960, 16384, 2**16). Error: Reason: Ok, before it handled this, but why do we have a null pointer in the tree?!? |
|
@robertwissing see if the recent commit fixes this problem. |
|
That fixed the problem, but seems like it is slightly slower with the tree replication than without. I have not run it on the merger case, but I ran it on a refined dwarf (the one from your benchmark 8X, so 400M particles). Ran it with 1024 and 8192 cores and a bit slower on both core numbers. I saw that Idle time seem to go up in the tree replication run. Below is stats for the tenth step: Regular UCX 8N: |
|
Looking at the load balancing data, this simulation does not seem to have a difficult time load balancing, so it's not clear that tree replication is needed. Key numbers are: maxPred/avgPred is very close to 1, indicating that the balancer thinks it's about to do a very good job; final "Calulating gravity" number is slightly less than maxPred, indicating that load balancing was even better than predicted. |
|
I tried to commit, but got permission denied, the tree replication need to be added to the tree build in starform.cpp aswell:
I ran the merger case which is more clustered, and here I do get quite the improvement. As can be seen below(for 4096 CPU). I also ran this simulation with more tree pieces(42000 -> 160000), in an attempt to increase the minPiece number. but instead got minPiece: 0 in these runs. not sure why that is happening exactly..... WITH TREE REPLICATION: [Orb3dLB_notopo] sorting Orb3dLB_notopo stats: maxObjLoad 0.749472 REGULAR: [Orb3dLB_notopo] sorting Orb3dLB_notopo stats: maxObjLoad 0.762852 |
|
Note: you can always do a pull request on a pull request. |
| @@ -2059,6 +2063,9 @@ void Main::advanceBigStep(int iStep) { | |||
| /******** Tree Build *******/ | |||
| buildTree(activeRung); | |||
|
|
|||
| tpReplicaProxy.clearTable(CkCallbackResumeThread()); | |||
There was a problem hiding this comment.
A refactor needs to happen to avoid code duplication (see also @robertwissing 's latest commit in starform.cpp). I suggest we move these lines into Main::buildTree().
This is @harshithamenon 's code to replicate treepieces to improve performance by distributing cache requests over several processors.