Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ci] Run python-so-copying.yml on Linux and Mac both [WIP] #2208

Closed
wants to merge 2 commits into from
Closed

[ci] Run python-so-copying.yml on Linux and Mac both [WIP] #2208

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b725bf
[ci] Run `python-so-copying.yml` on Linux and Mac both
johnkerl Mar 5, 2024
df00bac
typofix?
johnkerl Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
38 changes: 22 additions & 16 deletions .github/workflows/python-so-copying.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,19 @@ defaults:

jobs:
build:
runs-on: ubuntu-latest
name: "TILEDB_EXISTS: ${{ matrix.TILEDB_EXISTS }} TILEDBSOMA_EXISTS: ${{ matrix.TILEDBSOMA_EXISTS }}"
strategy:
fail-fast: false
matrix:
os: [ubuntu-22.04, macos-12]
TILEDB_EXISTS: ["no", "yes"]
TILEDBSOMA_EXISTS: ["no", "yes"]
exclude:
- TILEDB_EXISTS: "no"
TILEDBSOMA_EXISTS: "yes"
runs-on: ${{ matrix.os }}
container:
image: ubuntu:22.04
image: ${{ matrix.os }}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the tricky part. This needs to be a Docker image, not the name of the GitHub Actions runner.

The linux job runs inside of a Docker container (this made it easier for me to test locally). Thus it's not as easy as adding macos-12 to the build matrix. We need to either 1) make the container directive only apply to the ubuntu job (probably possible, but not immediately clear to me how), or 2) create a second job that runs similar code on macOS (which might be worth it given all the changes to accommodate running the same steps on linux and macOS)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jdblischak I'm totally happy with well-documented code-duplication here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did find some workarounds to create a build matrix that builds natively on macOS but in Docker on Ubuntu. There is no if, so the key is to pass null to container for the macOS job. Overall the readability is not great though.

Also, I realized that we'll also need to download a different pre-built libtiledb binary for the macOS build, which makes the option to have a completely separate macOS build all the more appealing.

steps:
- name: Docker image info
run: |
Expand Down Expand Up @@ -87,39 +88,44 @@ jobs:
cd apis/python
../../venv-soma/bin/python setup.py bdist_wheel
- name: Inspect wheel
run: unzip -l apis/python/dist/tiledbsoma-*.whl | grep '\.so'
- name: Confirm libtiledb.so is copied
run: unzip -l apis/python/dist/tiledbsoma-*.whl | egrep '\.so|\.dylib'
- name: Confirm libtiledb.so/dylib is copied
if: ${{ matrix.TILEDB_EXISTS == 'no' }}
run: unzip -l apis/python/dist/tiledbsoma-*.whl | grep -q libtiledb.so
- name: Confirm libtiledb.so is **not** copied when using external shared object
run: unzip -l apis/python/dist/tiledbsoma-*.whl | egrep -q 'libtiledb.so|libtiledb.dylib'
- name: Confirm libtiledb.so/dylib is **not** copied when using external shared object
if: ${{ matrix.TILEDB_EXISTS == 'yes' }}
run: |
if unzip -l apis/python/dist/tiledbsoma-*.whl | grep -q libtiledb.so
if unzip -l apis/python/dist/tiledbsoma-*.whl | egrep -q 'libtiledb.so|libtiledb.dylib'
then
echo "libtiledb.so was copied into the wheel when it was built against an external shared object"
echo "libtiledb.so / libtileb.dylib was copied into the wheel when it was built against an external shared object"
exit 1
fi
- name: Confirm libtiledbsoma.so is copied
- name: Confirm libtiledbsoma.so/dylib is copied
if: ${{ matrix.TILEDBSOMA_EXISTS == 'no' }}
run: unzip -l apis/python/dist/tiledbsoma-*.whl | grep -q libtiledbsoma.so
- name: Confirm libtiledbsoma.so is **not** copied when using external shared object
run: unzip -l apis/python/dist/tiledbsoma-*.whl | egrep -q 'libtiledbsoma.so|libtiledbsoma.dylib'
- name: Confirm libtiledbsoma.so/dylib is **not** copied when using external shared object
if: ${{ matrix.TILEDBSOMA_EXISTS == 'yes' }}
run: |
if unzip -l apis/python/dist/tiledbsoma-*.whl | grep -q libtiledbsoma.so
if unzip -l apis/python/dist/tiledbsoma-*.whl | egrep -q 'libtiledbsoma.so|libtiledbsoma.dylib'
then
echo "libtiledbsoma.so was copied into the wheel when it was built against an external shared object"
echo "libtiledbsoma.so / libtliedbsoma.dylib was copied into the wheel when it was built against an external shared object"
exit 1
fi
- name: Install wheel
run: ./venv-soma/bin/python -m pip install --prefer-binary apis/python/dist/tiledbsoma-*.whl
- name: Check linking and RPATH
- name: Check linking and RPATH, Linux
if: ${{ inputs.os == 'ubuntu-22.04' }}
run: |
ldd ./venv-soma/lib/python*/site-packages/tiledbsoma/pytiledbsoma.*.so
readelf -d ./venv-soma/lib/python*/site-packages/tiledbsoma/pytiledbsoma.*.so | grep R*PATH
- name: Check linking and RPATH, MacOS
if: ${{ inputs.os == 'macos-12' }}
run: |
otool -L ./venv-soma/lib/python*/site-packages/tiledbsoma/pytiledbsoma.*.so
- name: Runtime test
run: ./venv-soma/bin/python -c "import tiledbsoma; print(tiledbsoma.pytiledbsoma.version())"
- name: Confirm linking to installed shared objects
run: |
rm -fr build/ build-libtiledbsoma/ dist/ apis/python/build apis/python/src/tiledbsoma/*tile*.so*
find . -name '*tile*.so*' # should only show shared objects installed in virtual env
rm -fr build/ build-libtiledbsoma/ dist/ apis/python/build apis/python/src/tiledbsoma/*tile*.so* apis/python/src/tiledbsoma/*tile*.dylib
find . -name '*tile*.so*' -o -name '*tiledb*.dylib' # should only show shared objects installed in virtual env
./venv-soma/bin/python -c "import tiledbsoma; print(tiledbsoma.pytiledbsoma.version())"