Add separate workflow and dockerfile for conda build

[bquan0]

Description

This is a continuation of the closed PR #1516. In the dockerfile of that PR, we had one stage for installing dependencies via apt and one stage for installing dependencies via conda. However, we decided that it would be best to split up the apt build and the conda build for a few reasons

• the builds weren't working.
• When we had apt and conda in one dockerfile, we weren't able to take advantage of the fact that we could install moab, dagmc, and openmc via conda. This structural difference means that it's better to have 2 separate workflows and dockerfiles.

In this PR, I have reverted ubuntu_22.04-dev.dockerfile and docker_publish.yml to what they were 7 months ago, right before any of the conda changes were made. The only change I made was to remove the cython restriction in the dockerfile.

I also added a new conda_build.dockerfile and conda_docker.yml to build and test PyNE by installing packages via conda. It's still a work in progress and I'm open to any feedback about how I can change and improve it.

Motivation and Context

This PR will be the new PR to close #1515.

Changes

This should be a fix to the current situation on develop where conda builds weren't working properly at all. This should also be a better way to implement what #1516 was trying to accomplish.

Behavior

Current behavior is that we try to build PyNE via conda dependencies or apt dependencies in the same workflow/dockerfile. New behavior is that there will be separate workflows/dockerfiles.

[gonuke]

Thanks for kicking this off

[ahnaf-tahmid-chowdhury]

I have a suggestion to make. Recently, while working on the DAGMC CI, I found its implementation to be quite well. Would it be possible for us to apply a similar method here as well? It could make it easier for us to understand all the projects as they would be configured in the same way. In the case of PyNE, we may need to add a few steps to make it compatible with the CI. What do you think, @gonuke? Do you agree, or do you have a better alternative in mind?

[gonuke]

I think we first need to get some version of this working, and then we can look into improvements. The PyNE build process is different from the DAGMC one for a few reasons and we need to make sure that our CI is meeting our overall needs.

[ahnaf-tahmid-chowdhury]

Not required I suppose. We may set pyne install prefix to conda dir.

[ahnaf-tahmid-chowdhury]

Here is the issue. This should be RUN mkdir -p $HOME/opt/pyne

[ahnaf-tahmid-chowdhury]

Please use COPY command here and copy the source to the dockerfile.

[bquan0]

To apply this change, I have removed the pyne-dev stage. Now, we will build the existing pyne stage in docker_publish.yml since that uses the COPY command.

[ahnaf-tahmid-chowdhury]

MOAB v5.3 is old and doesn't support Cython v3. We have a separate PR #1528 for this. Please consider using Cython v0.29 for now.

[gonuke]

The limitation isn't from MOAB. Conda-forge has MOAB 5.5.1

[ahnaf-tahmid-chowdhury]

Unfortunately, we still need to consider staying with Cython 0.29 for this current PR, as PyNE has issues with Cython 3. This may be resolved through a separate PR #1528.

[ahnaf-tahmid-chowdhury]

It seems, the action workflows, share similarities. I'm curious if we could potentially use a single action file to select different Docker files, if feasible. I'd like to pass this decision over to @gonuke.

[gonuke]

This is failing tests because of the inconsistent resolution of $HOME

[bquan0]

After running into a bunch of errors (which were fixed by applying the solutions I previously found in the conda-build-fix branch that I forgot about), I'm now running into a "file not found" error when running the tests, which is strange. I did a quick vimdiff between the conda setup that I currently have and the conda setup in conda-build-fix but there weren't any differences other than order of dependencies and not having hdf5<1.14 (since having hdf5 breaks the build).

[gonuke]

I think something changed with how pytest handles a setup function. @bennibbelink seemed to notice this with Cyclus a few days ago and made a simple fix there that avoided the issue rather than corrected it....

The missing file is downloaded in a setup method that maybe is not being run by pytest?

[ahnaf-tahmid-chowdhury]

I'm not sure if this is the problem. I've been working on scikit-build-core and have fixed all the test issues; now all tests are passing. However, I've noticed some UnicodeDecodeError instances and resolved the use of the setup function in two files. @bquan0, could you please try running pytest -ra in the tests folder locally and see what happens?

[bquan0]

I noticed that the tests were also not working for the original docker_publish.yml where we install dependencies with apt. I think this has to do with the pytest version since they deprecated the setup() and teardown() functions in version 7.2.0. From some initial testing on my laptop with the pytest 8.1.1, it seems like we need to rename these functions into setup_function() and teardown_function().

[bquan0]

The change to the setup() function seems to have fixed a few of the errors. Also, the most recent commit isn't necessary, I just added it because I thought the setup() function still wasn't running after the rename to setup_function(). In actuality, it was running, but the failed workflow made me think otherwise.

The error that's causing the workflow to fail now has to do with the pyne.ace.ascii_to_binary() function.

The file that is missing is the C12-binary.ace file created by pyne.ace.ascii_to_binary() from C012-n.ace (file imported in the setup() function which now actually runs).

[gonuke]

I suggested the change to setup_module() in once place and it should probably propagate to the others.

Our previous usage of setup() assumed that it was called once per module, so we want to ensure that same behavior now, and not once per function. I think (speculation) that using setup_function() and teardown_function() results in teardown happening after each test, thus removing a file generated in one test that is needed for another test. (this practice may have other flaws)

[bquan0]

After applying that change and removing the dec import in test_chainsolve.py (since it is deprecated in newer numpy versions and isn't used in the file), the first BuildTest (only base_python) is passing.

The other BuildTests error out in test_activation_responses.py in the test_response_hdf5_to_mesh() test due to an error with creating a mesh from PyMOAB, which I don't know how to fix.

[gonuke]

Somethings to try in test_mesh.py:test_structured_mesh_from_coords()

1. force ranges to be lists in assert_array_equal
2. loop over elements in list:

for obs, exp in zip(sm.structured_coords, [range(1, 5), range(1, 4), range(1, 3)]):
    assert_array_equal(obs, exp)

[gonuke]

For conda builds, let's pin the moab version to 5.3.0

[bquan0]

All of the tests in docker_publish.yml are now passing except for the dagmc + hdf5 test. It's failing in test_activation_responses.py in the test_activation_responses_script() function due to a FileNotFoundError for alara_inp. It seems like a mismatch in hdf5 version might be the cause.

[ahnaf-tahmid-chowdhury]

Please consider pinning HDF5 to 1.10 not <1.14

[bquan0]

Big news: the original docker_publish.yml is now working!
Also, I propose that the github workflows for each dockerfile only runs when its respective dockerfile changes (and not when anything in the docker/ folder changes).

Also, I just realized that the 10 in update-alternatives --install /usr/bin/python python /usr/bin/python3 10 refers to the priority of the install, not the version of python. So, I have pinned the python version in conda to <3.10 because moab 5.3.0 doesn't work with python 3.10.

[bquan0]

After pinning "python<3.10", the moab install works.
After updating conda and using mamba to install dagmc and openmc, those installations also work.

The moab tests are now passing. However, the dagmc and openmc tests are failing due to the same ValueError: Cannot create cython.array from NULL pointer in test_response_hdf5_to_mesh() of test_activation_responses.py that we had with the moab test of the conda build previously (which we solved by pinning the moab version to 5.3.0).

[gonuke]

The moab tests are now passing. However, the dagmc and openmc tests are failing due to the same ValueError: Cannot create cython.array from NULL pointer in test_response_hdf5_to_mesh() of test_activation_responses.py that we had with the moab test of the conda build previously (which we solved by pinning the moab version to 5.3.0).

This may be because the DAGMC conda package is out of date. I'll have to see to that.

[ahnaf-tahmid-chowdhury]

I think it's not a good idea to set Python < 3.10 since ubuntu 22.04 comes with Python 3.10.6 which is old now. I will check this on the scikit-build-core branch and try to resolve that there.