-
|
Hello! I am trying to simulate a very large model, and I am using cloud to run the forward and inverse simulations. I have seen the above error when using JUDI package. Are there any solutions? |
Beta Was this translation helpful? Give feedback.
Answered by
mloubout
Apr 16, 2024
Replies: 2 comments 18 replies
-
|
This looks like a worker died, this error message is just julia saying it lost communication with that worker, would need a complete error trace that shows where JUDI errored. side note: Also it's easier if you use code blocks rather than screenshots (i.e triple quotes) |
Beta Was this translation helpful? Give feedback.
7 replies
-
|
How can I run a complete error trace that shows where JUDI errored? |
Beta Was this translation helpful? Give feedback.
-
|
I mean copy paste the entire error log, not just the small part above |
Beta Was this translation helpful? Give feedback.
-
|
I have updated the error message |
Beta Was this translation helpful? Give feedback.
-
|
That's odd it seems to error at model creation, is there some weird values in your model parameter? |
Beta Was this translation helpful? Give feedback.
-
|
Are you running out of memory/... maybe? |
Beta Was this translation helpful? Give feedback.
-
|
I have seem some warnings during forward modeling, I will post it here.
```
From worker 16: /home/hp/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
From worker 16: setattr(self, word, getattr(machar, word).flat[0])
From worker 16: /home/hp/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
From worker 16: return self._float_to_str(self.smallest_subnormal)
From worker 18: Building forward operator
From worker 50: Building forward operator
From worker 3: /home/hp/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:500: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
From worker 3: setattr(self, word, getattr(machar, word).flat[0])
From worker 3: /home/hp/.local/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
From worker 3: return self._float_to_str(self.smallest_subnormal)
```
|
Beta Was this translation helpful? Give feedback.
11 replies
-
|
I assume that if more cores are used forward modeling and inversion of a large model will be accelerated. But the improvements of computational time is not significant. The size of my model is like 2400*3000, are there any other way to accelerate computation? |
Beta Was this translation helpful? Give feedback.
-
Could you elaborate on that? |
Beta Was this translation helpful? Give feedback.
-
|
Firstly I am using addprocs(32) to invert this model, but it is slow. Then I am increasing the number in addprocs, for example, 64, or even more. But acceleration of inversion is not significant. |
Beta Was this translation helpful? Give feedback.
-
|
So if you use
Hope this helps |
Beta Was this translation helpful? Give feedback.
Answer selected by
tjk9501
-
|
Thanks for sharing that. I have two confusions after reading the example manual: Firstly, how to balance choice between workers and threads and achieve maximum computational speed according to information output by the command and model size. For example, if lscpu tells me there are 80 cpus in the workstation, and my model size is 1000 by 2000. Is it correct to choose the number of threads so that model size can be divide exactly and ensure number of workers is less than 80 in the simulation? Secondly, I have seen that in the example script, due to the double line array setup (which is a sparse array case), imaging results of the rectangle speed anomalies is strongly distorted in shape. But using the updated projected method (bounds+TV), distortion of imaging can be suppressed and reduced. If I am using a gradient updating scheme like: How to correctly apply regularizations like bounds+TV to avoid imaging distortion in sparse array cases? |
Beta Was this translation helpful? Give feedback.
-
No there is no such constraint you can choose any number of threads, openMP will take care of the scheduling accordingly
Yes at least, if you use 80 workers you will only use one thread per shot which might be quite slow. But in general, for FWI/RTM the main constraint is gonna be the memory usage, so I would choose the number of worker such that you have enough memory for every parallel shot. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! About the second question, I read your manual and in the double line array case, I noticed there is a prj() function which is used in the double line array imaging setup. you calculated and passing mg2 into the prj function with Compared with the distortion is imporved. Why will this happen? And what is the influence of prj() function? |
Beta Was this translation helpful? Give feedback.
-
|
If you are interested in regularization/constraints I would recommend reading |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
addprocsadd julia workers so it will process shots in parallel. However, If you are on a single node now, the resources are shared by all these workers, including memory and CPU cores, and will compete for it. So it will get quite slower because now the process spends time waiting for an available core, and you will run out of memory much quicker than if you were using x32 or x64 more memory.So if you use
addprocsyou need to make sure that:GOMP_CPU_AFFINITY, you can see an example here in section 3Hope this…