Unequal shape partition during MPI parallel mesh.

[CosmoCloudZhang]

@rainwoodman
Hi, I get a bug when I estimate power spectra using multiple CPU cores, e.g., 8 CPUs. Could you please help me diagnose where the problem lies?

The error message reported is attached as follows.

File "CODE/L1000N0900_FIDUCIAL/PHH/DATA/DMO_FOF.py", line 71, in <module> MESH1 = mesh.catalog.CatalogMesh(source = CATALOG1, Position = ['Position'], Nmesh = N, BoxSize = L, resampler = 'tsc', compensated = True, interlaced = True) File "/cosma/home/dp203/dc-zhan7/anaconda3/envs/CosmoConda/lib/python3.11/site-packages/nbodykit/source/mesh/catalog.py", line 79, in __init__ File "/cosma/home/dp203/dc-zhan7/anaconda3/envs/CosmoConda/lib/python3.11/site-packages/pmesh/pm.py", line 1446, in __init__ MeshSource.__init__(self, source.comm, File "/cosma/home/dp203/dc-zhan7/anaconda3/envs/CosmoConda/lib/python3.11/site-packages/nbodykit/base/mesh.py", line 50, in __init__ self.pm = ParticleMesh(BoxSize=BoxSize, Nmesh=_Nmesh, File "/cosma/home/dp203/dc-zhan7/anaconda3/envs/CosmoConda/lib/python3.11/site-packages/pmesh/domain.py", line 381, in __init__ self.edges = numpy.asarray(edges) ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimension. The detected shape was (3,) + inhomogeneous part. application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3

When I do print(print(self.comm.size, partition.i_edges)) in the init of the ParticleMesh class from pmesh/pm.py, I got 8 [array([ 0, 450, 900]), array([ 0, 225, 450, 675, 900]), array([ 0, 900])]. So Pmesh did not properly produce an equal partition of the catalogue, thus producing an array of edges with unequal shapes. I guess that explains the problem. Would you happen to have any idea how to fix this?

And I wonder if there is any example script for data-based parallelisation starting from a discrete catalogue to get the eventual power spectra since the example in API documentation is incomplete.

For your information, I am using the Nbodykit with version 0.3.15, Pmesh with version 0.1.56, and pfft-python with version 0.1.21, installed via Anaconda at the cluster.