How to build a conda-package for ABACUS with CUDA GPU implementation?

[Flashbac09]

Details

I 've been able to install conda-package abacus 3.1.3 (https://anaconda.org/deepmodeling/abacus) to create environment and

test ABACUS on the machine with NVIDIA GPU.

While a beginner on conda & conda-build, I tried to build my own conda-package of abacus with the change of compiler

setting of USE_CUDA=ON. However, conda-build failed to solve environment setting, and the building process both failed on

the circumstances of USE_CUDA=ON and USE_CUDA=OFF.

I sincerely ask for certain guidance on how to build abacus package with given meta.yaml (and building script in it). Most

importantly, how should we compile the GPU version of ABACUS with conda-build.

Case 1(trying to build a default version): meta.yaml was given in abacus-develop/conda.

jzw2@hhz:~/jzw2sda/files/abacus-develop-0303/conda$ conda build .
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
Copying /home/jzw2/jzw2sda/files/abacus-develop-0303 to /home/jzw2/jzw2sda/ana/conda-bld/abacus_1678276688186/work/
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /home/jzw2/jzw2sda/files/abacus-develop-0303/conda/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /home/jzw2/jzw2sda/files/abacus-develop-0303/conda/conda_build_config.yaml
Attempting to finalize metadata for abacus
INFO:conda_build.metadata:Attempting to finalize metadata for abacus
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

Leaving build/test directories:
  Work:
 /home/jzw2/jzw2sda/ana/conda-bld/work
  Test:
 /home/jzw2/jzw2sda/ana/conda-bld/test_tmp
Leaving build/test environments:
  Test:
source activate  /home/jzw2/jzw2sda/ana/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_
  Build:
source activate  /home/jzw2/jzw2sda/ana/conda-bld/_build_env


Traceback (most recent call last):
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/environ.py", line 799, in get_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/common/io.py", line 84, in decorated
    return f(*args, **kwds)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/plan.py", line 470, in install_actions
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 132, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 175, in solve_for_diff
    final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 293, in solve_final_state
    ssc = self._add_specs(ssc)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 590, in _add_specs
    explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/resolve.py", line 553, in _get_package_pool
    pool = self.get_reduced_index(specs)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/common/io.py", line 84, in decorated
    return f(*args, **kwds)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/resolve.py", line 574, in get_reduced_index
    explicit_specs, features = self.verify_specs(explicit_specs)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/resolve.py", line 286, in verify_specs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound:
  - elpa=[build=mpi*]
  - openblas=[build=openmp*]
  - fftw=[build=mpi*]
  - libxc
  - cereal

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jzw2/jzw2sda/ana/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/cli/main_build.py", line 496, in main
    execute(sys.argv[1:])
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/cli/main_build.py", line 485, in execute
    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/build.py", line 3091, in build_tree
    packages_from_this = build(metadata, stats,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/build.py", line 2054, in build
    output_metas = expand_outputs([(m, need_source_download, need_reparse_in_env)])
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 788, in expand_outputs
    for (output_dict, m) in deepcopy(_m).get_output_metadata_set(permit_unsatisfiable_variants=False):
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/metadata.py", line 2121, in get_output_metadata_set
    conda_packages = finalize_outputs_pass(ref_metadata, conda_packages, pass_no=0,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/metadata.py", line 782, in finalize_outputs_pass
    fm = finalize_metadata(om, parent_metadata=parent_metadata,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 546, in finalize_metadata
    build_unsat, host_unsat = add_upstream_pins(m,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 408, in add_upstream_pins
    host_deps, host_unsat, extra_run_specs_from_host = _read_upstream_pin_files(m, 'host',
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 373, in _read_upstream_pin_files
    deps, actions, unsat = get_env_dependencies(m, env, m.config.variant,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 138, in get_env_dependencies
    actions = environ.get_install_actions(tmpdir, tuple(dependencies), env,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/environ.py", line 801, in get_install_actions
    raise DependencyNeedsBuildingError(exc, subdir=subdir)
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'fftw=[build=mpi*]', 'openblas=[build=openmp*]', 'cereal', 'elpa=[build=mpi*]', 'libxc'}

Case 2(meta.yaml changed):
cmake -B conda_build ${CMAKE_ARGS} -DCMAKE_BUILD_TYPE=Release -DENABLE_DEEPKS=1 -DENABLE_LIBXC=1 -DUSE_CUDA=1

jzw2@hhz:~/jzw2sda/files/abacus-develop-0303/conda$ conda build .
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.16
Copying /home/jzw2/jzw2sda/files/abacus-develop-0303 to /home/jzw2/jzw2sda/ana/conda-bld/abacus_1678277484450/work/
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /home/jzw2/jzw2sda/files/abacus-develop-0303/conda/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /home/jzw2/jzw2sda/files/abacus-develop-0303/conda/conda_build_config.yaml
Attempting to finalize metadata for abacus
INFO:conda_build.metadata:Attempting to finalize metadata for abacus
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

Leaving build/test directories:
  Work:
 /home/jzw2/jzw2sda/ana/conda-bld/work
  Test:
 /home/jzw2/jzw2sda/ana/conda-bld/test_tmp
Leaving build/test environments:
  Test:
source activate  /home/jzw2/jzw2sda/ana/conda-bld/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_
  Build:
source activate  /home/jzw2/jzw2sda/ana/conda-bld/_build_env


Traceback (most recent call last):
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/environ.py", line 799, in get_install_actions
    actions = install_actions(prefix, index, specs, force=True)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/common/io.py", line 84, in decorated
    return f(*args, **kwds)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/plan.py", line 470, in install_actions
    txn = solver.solve_for_transaction(prune=prune, ignore_pinned=not pinned)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 132, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 175, in solve_for_diff
    final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 293, in solve_final_state
    ssc = self._add_specs(ssc)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/core/solve.py", line 590, in _add_specs
    explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/resolve.py", line 553, in _get_package_pool
    pool = self.get_reduced_index(specs)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/common/io.py", line 84, in decorated
    return f(*args, **kwds)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/resolve.py", line 574, in get_reduced_index
    explicit_specs, features = self.verify_specs(explicit_specs)
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda/resolve.py", line 286, in verify_specs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound:
  - libxc
  - elpa=[build=mpi*]
  - cereal
  - fftw=[build=mpi*]
  - openblas=[build=openmp*]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jzw2/jzw2sda/ana/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/cli/main_build.py", line 496, in main
    execute(sys.argv[1:])
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/cli/main_build.py", line 485, in execute
    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/build.py", line 3091, in build_tree
    packages_from_this = build(metadata, stats,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/build.py", line 2054, in build
    output_metas = expand_outputs([(m, need_source_download, need_reparse_in_env)])
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 788, in expand_outputs
    for (output_dict, m) in deepcopy(_m).get_output_metadata_set(permit_unsatisfiable_variants=False):
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/metadata.py", line 2121, in get_output_metadata_set
    conda_packages = finalize_outputs_pass(ref_metadata, conda_packages, pass_no=0,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/metadata.py", line 782, in finalize_outputs_pass
    fm = finalize_metadata(om, parent_metadata=parent_metadata,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 546, in finalize_metadata
    build_unsat, host_unsat = add_upstream_pins(m,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 408, in add_upstream_pins
    host_deps, host_unsat, extra_run_specs_from_host = _read_upstream_pin_files(m, 'host',
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 373, in _read_upstream_pin_files
    deps, actions, unsat = get_env_dependencies(m, env, m.config.variant,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/render.py", line 138, in get_env_dependencies
    actions = environ.get_install_actions(tmpdir, tuple(dependencies), env,
  File "/home/jzw2/jzw2sda/ana/lib/python3.9/site-packages/conda_build/environ.py", line 801, in get_install_actions
    raise DependencyNeedsBuildingError(exc, subdir=subdir)
conda_build.exceptions.DependencyNeedsBuildingError: Unsatisfiable dependencies for platform linux-64: {'fftw=[build=mpi*]', 'openblas=[build=openmp*]', 'libxc', 'cereal', 'elpa=[build=mpi*]'}

[caic99]

Hi @Flashbac09 ,
Thank you for trying this. Please note that indicating the conda-forge channel is required for elpa and other packages: conda build . -c conda-forge. Please refer to:

abacus-develop/conda/meta.yaml

Lines 6 to 9 in 47afeb1

	# This conda package can also be built locally with
	# conda build . -c conda-forge
	# And install with
	# conda create -n my_abacus_env abacus -c local -c conda-forge

Ref: https://abacus.deepmodeling.com/en/latest/quick_start/easy_install.html#install-by-conda

---

Regarding compiling the GPU version of ABACUS with conda-build, we have not tried it before. IMHO you can try adding a cudatoolkit in the requirements and build ABACUS with GPU support. Please let me know if you have any further questions!

[Flashbac09]

Currently, I've figured out a way to modify the meta data to build the GPU version of ABACUS, which can utilize the GPU and accelerate performance. However, it lacks complete test and requires long time to build & create environment. Therefore, my issue is done here for running the whole process. Further development needs more rounded analysis for conda employment, and it's my pleasure if developers can benefit from this document then.
conda Package for ABACUS-GPU.pdf

[caic99]

@Flashbac09 Thanks for your effort! Now the conda package with GPU support of ABACUS is available by conda-forge: https://anaconda.org/conda-forge/abacus