Possible bug in using getTagAllocator() in GriddingAlgorithm.C when enabling CUDA

[ctian282]

When compiling with CUDA, RAJA and umpire, in my environment with gcc-7, CUDA 10, I noticed that the usage of getTagAllocator() in GriddingAlgorithm.C to initialize CellVariables will cause CUDA illegal memory access when filling those tag variables with GriddingAlgorithm::fillTags(). This causes the test sets failure as I mentioned in a previous issue. #175

The issue is caused by initialing tag data with tagAllocator, but filling it as a shared memory variable. Replacing getTagAllocator() with getDefaultAllocator() will temporary resolve this issue and pass all the tests, but I am not sure if this will bring some potential problems.

[PhilipDeegan]

to force SAMRAI to use host allocation for GriddingAlgorithm you can run something like this on program startup (or sometime before the AllocatorDatabase is initialized)

    auto& rm = umpire::ResourceManager::getInstance();

    { // initialize samrai internals allocator to HOST
        assert(!rm.isAllocator("samrai::data_allocator"));
        [[maybe_unused]] auto samrai_allocator = rm.makeAllocator<umpire::strategy::DynamicPool>(
            "samrai::data_allocator", rm.getAllocator(umpire::resource::Host));
        assert(samrai_allocator.getPlatform() == umpire::Platform::host);
    }

edit: this will only work after #180 is merged

[nselliott]

The recent changes in #180 should fix the issue with the allocator you receive from getTagAllocator, as that will be a host allocator by default, and the kernels for CellData will stay on the host for the tag variables. In general you can reassign the resource for the allocators provided in tbox::AllocatorDatabase by using the syntax suggested here by @PhilipDeegan .