Particle deregistration error on ASPECT v2.6.0-pre with deal.II v9.5.1

[dyvasey]

I am currently attempting to re-run continental rifting models that were previously run with older versions of ASPECT (v2.4.0-pre, commit 70a2ec3) and deal.II (v9.3.3) with newer versions (ASPECT v2.6.0-pre, commit 845dd6d; deal.II v9.5.1) in order to take advantage of recently-added Fastscape coupling. I am finding that a model that runs successfully on the same HPC system using the old versions fails when using the newer versions with the following particle deregistration error:

terminate called after throwing an instance of 'dealii::StandardExceptions::ExcMessage'
  what():
--------------------------------------------------------
An error occurred in line <62> of file </cluster/home/dvasey01/vaseylab/shared/software/dealii/v9.5.1/candi/install/tmp/unpack/deal.II-v9.5.1/source/particles/property_pool.cc> in function
    void dealii::Particles::PropertyPool<dim, spacedim>::clear() [with int dim = 2; int spacedim = 2]
The violated condition was:
    n_open_handles == 0
Additional information:
    This property pool currently still holds 1 open handles to memory that
    was allocated via allocate_properties_array() but that has not been
    returned via deregister_particle().

Stacktrace:
-----------
#0  /cluster/home/dvasey01/vaseylab/shared/software/dealii/v9.5.1/candi/install/deal.II-v9.5.1/lib/libdeal_II.so.9.5.1: dealii::Particles::PropertyPool<2, 2>::clear()
#1  /cluster/home/dvasey01/vaseylab/shared/software/dealii/v9.5.1/candi/install/deal.II-v9.5.1/lib/libdeal_II.so.9.5.1: dealii::Particles::ParticleHandler<2, 2>::clear_particles()
#2  /cluster/home/dvasey01/vaseylab/shared/software/dealii/v9.5.1/candi/install/deal.II-v9.5.1/lib/libdeal_II.so.9.5.1: dealii::Particles::ParticleHandler<2, 2>::~ParticleHandler()
#3  /cluster/home/dvasey01/vaseylab/shared/software/aspect/v2.6.0-pre-845dd6d/aspect/build/./aspect: aspect::Particle::World<2>::~World()
#4  /cluster/home/dvasey01/vaseylab/shared/software/aspect/v2.6.0-pre-845dd6d/aspect/build/./aspect: aspect::Particle::World<2>::~World()
#5  /cluster/home/dvasey01/vaseylab/shared/software/aspect/v2.6.0-pre-845dd6d/aspect/build/./aspect: aspect::Simulator<2>::~Simulator()
#6  /cluster/home/dvasey01/vaseylab/shared/software/aspect/v2.6.0-pre-845dd6d/aspect/build/./aspect: void run_simulator<2>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool)
#7  /cluster/home/dvasey01/vaseylab/shared/software/aspect/v2.6.0-pre-845dd6d/aspect/build/./aspect: main
--------------------------------------------------------

SIGABRT received

This error occurs after the model has been running on a single node for a couple hours (after ~3 Myr of model time), so it does not initiate immediately. Since the same model runs successfully on the same cluster with the older ASPECT/deal.II versions, it does not appear to be an issue (at first glance) with the system. @naliboff and I discussed this briefly and he has not seen this error before, so any insight into what is going on would be helpful. Perhaps I should be using a different version of deal.II?

[gassmoeller]

Hi Dylan, can you check how much your problem overlaps with the one discussed in #5594? They may be related or the same, in which case either one of the following options should fix the crash:

• running the model in release mode should work just fine (the problem is an actual bug, but it doesnt affect the solution of the model)
• using a recent development version of deal.II should fix it in both debug and release mode

[dyvasey]

Hi Rene, thanks for pointing me to that issue - the error certainly looks very similar, and the version of deal.II I am using predates the pull request you put in to address this (dealii/dealii#16709). I will try compiling a recent deal.II and see if that fixes the problem. This error is actually occurring in release mode (I haven't tried to reproduce it in debug mode yet). Thanks!

[dyvasey]

It looks like the pull request mentioned above fixes the issue. I was unable to easily compile a development version of deal.II (I am using recent Intel compilers but got an error about GNU compilers being out of date that I don't fully understand, but that is a separate issue), but I was able to install v9.5.1 with dealii/dealii@9d90510 cherry-picked. Thanks for the help!