conda packaging (experimental WIP) #1131
Conversation
Robadob
force-pushed
the
rob_conda
branch
2 times, most recently
from
9a961ec
to
edf85da
Compare
|
Via Note |
|
Conda post-build log of first successful package build. |
Summary of warnings (Linux)CMake isn't removing all RPaths? Disabling the CMake rpath command did not change this warning, nor add anything related (assuming i updated git right) Overlinking (resolved)> [During the post-build phase, the shared libraries in the newly created package are examined for linkages which are not provided by the package's requirements or a predefined list of system libraries.](https://docs.conda.io/projects/conda-build/en/stable/resources/define-metadata.html#whitelisting-shared-libraries)I've resolved this by moving cuda dependencies with required requirements:
build:
...
- libcurand-dev # [linux] Build as no .so export (we don't use host curand)
- swig ==4.0.2 # Have to manually direct swig to find this
- git
- cmake
host:
- cuda-cudart-dev # [linux] Host as require .so export
- cuda-nvrtc-dev # [linux] Host as require .so exportHowever, I'm concerned this will make an implicit requirement on the build:
...
- libcurand-dev # [linux] Build as no .so export (we don't use host curand)
- cuda-cudart-dev # [linux] Host as require .so export
- cuda-nvrtc-dev # [linux] Host as require .so export
host:
...
run:
...
- cuda-cudart # [linux]
- cuda-nvrtc # [linux]Additionally, build:
missing_dso_whitelist:
- $RPATH/libcuda.so.1 # Ignore as this comes from CUDA driverPossible we can force an error for these warnings with No idea how to resolve these, we know them to be required but for whatever reason it thinks otherwise. The suggested build:
ignore_run_exports:
- python # Not clear why conda thinks this isn't used
- astpretty # Not clear why conda thinks this isn't usedPackage Verification (no longer present?)``` Package verification results: ----------------------------- /tmp/rob/tmp9cb176fx/pyflamegpu-2.0.0rc1-hbd1bc14_0.tar.bz2: C1141 Found python file "lib/python3.11/site-packages/pyflamegpu/codegen/codegen.py" without a corresponding pyc file ``` Conda is supposed to automatically compile all `.py`, i've only seen guides for exempting files.This may have self resolved, however I can't even see the "Package verification results" title anymore so unclear. Not too sure what clobber is, some kind of path rewrite? This would suggest it's trying to package the license of various cuda packages into our package though which seems wrong. |
-dev might be able to go in build and have the non-dev versions in host / run? Not sure about the rest of the things to resolve, might just have to be more trial and error. |
|
The currently built linux conda package appears to work, however it's a bit awkward to install it with dependencies.
|
|
Windows log from first successful conda packaging https://gist.github.com/Robadob/9e3b7a94991762f76a22d864d727d89a |
Summary of warnings (Windows)Missing dlls, both CUDA should probably be provided by driver/CUDA, so will add to DSO whitelist. |
|
Windows vis build now seems to work. Linux is stuck on CMake finding OpenGL. This guide suggests that core dependency tree (CDT) for mesa should be used, This conda recipe includes it's cdt There isn't much documentation about the COS6/7 feature, conda forge states that More research and testing required. |
|
trying to get cos7 to work cdt cos7 seems alot like a conda-forge feature.
https://conda-forge.org/docs/maintainer/knowledge_base.html#using-centos-7 Doesn't make a difference, maybe conda-forge specific.
Doesn't make a difference, maybe conda-forge specific.
Doesn't work, adding pkg |
|
Shall see how far I can work through this list for publishing on conda-forge this week. |
This is required in pyflamegpu for conda distribution. Add FLAMEGPU_BUILD_PYTHON_CONDA CMake option
Have been iterating build.sh Current issue is that when a venv is being setup to build the wheel, setuptools cannot be found despite it being installed by conda via requirements. Trying to install setuptools via pip fails too. Similarly, cmake can't find conda installed SWIG, however requires it be conda installed to allow build to suceed. May be able to address this one by manually locating paths to pass to CMake.
…ne for some reason
…a fresh version of the repo with a specific commit
Disable examples.
Need to dig through the log to check warnings, and do cleanup in the script. However, this is first success.
Sadly had to dso whitelist paths that should be found via run reqs
CUDA can be found, however build is failing because multiple CUDA include paths from diff versions being used at build time
…ckage on a diff machine.
This appears to work on my local machine (i renamed cuda12 bin to hide actual dlls)
Add vis feature track and string and dependency reqs Will require some cmake fiddling to use these deps rather than internet
|
conda-forge/staged-recipes#24483 Working on this here. |
| - $RPATH/cudart64_12.dll # [win64] Conda-build can't find this in run reqs cuda-nvrtc?? | ||
| ignore_run_exports: | ||
| - python # [linux] Not clear why conda thinks this isn't used | ||
| - astpretty # [linux, win64] Not clear why conda thinks this isn't used |
There was a problem hiding this comment.
Just leaving this here as a future reminder for when conda picks back up - astpretty is only a dependency of a test, not actually a pyflamegpu, so conda is correct (see #1164)
Very much work in progress
Changes
FLAMEGPU_BUILD_PYTHON_CONDAset(CMAKE_SKIP_RPATH TRUE)set(CMAKE_CUDA_RUNTIME_LIBRARY shared)placement, so that executables and pyflamegpu link against shared cuda libstodo
--error-overlinkingtoconda buildcommand, to werror overlinking.