TagBox::buffer very slow since commit #1258 if n_error_buf >> 1

[MSABuschmann]

Hi!
We are running some 3D simulations that require a relatively large n_error_buf value (n_error_buf ~ 10). The reason for such a large value is because we are tagging the center of an extended object and we want to make sure the complete object is refined. Since we are tagging the center of the object, n_error_buf corresponds to its size. I can provide more information on this if needed.

We had been using AMReX version 20.06 so far but recently upgraded to 21.03. We ran into an issue unrelated to this discussion with 20.06 that might have been resolved in the newer version, hence we decided to upgrade to 21.03. However, we noticed a serious performance issue in the newer version if n_error_buf is large.

I was able to trace back this performance issue to the function TagBox::buffer which had been completely rewritten in commit #1258 to make it GPU compatible (we use CPUs only). In our specific case with n_error_buf=11, this function takes only 0.07 second before commit #1258, but a stunning 99 seconds after commit #1258. If we reduce n_error_buf to 1 the runtime is about 0.05 seconds both before and after that commit. So clearly the newer version of TagBox::buffer does not do well with a large n_error_buf value and ultimately prevents us from running.

Is this know? Also, since we are not using GPUs in our specific case, is it feasible for us as a workaround to simply use the previous version of TagBox::buffer, or would that break too much of the code?

Thanks!

[WeiqunZhang]

Thanks for reporting this! We will look into it.

[WeiqunZhang]

@MSABuschman Could you give #1957 a try?

[MSABuschmann]

Thanks a lot for this quick fix @WeiqunZhang, #1957 works for us!