Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Packaging] Publish AdaptiveCpp package for Ubuntu #1344

Open
wants to merge 155 commits into
base: develop
Choose a base branch
from

Conversation

normallytangent
Copy link
Collaborator

@normallytangent normallytangent commented Feb 7, 2024

This PR adds a Github workflow for publishing AdaptiveCpp packages on every git-push.

  • Currently the debian directory is handled manually: ideally I would want to automate this
  • Publish the resulting artifact on Github
  • Package for architecture any
  • Use custom compiler
  • Separate packages for cpu/sscp and cpu/cuda/rocm/ze backends

@normallytangent normallytangent marked this pull request as draft February 16, 2024 13:26
@normallytangent normallytangent changed the title WIP: Publish AdaptiveCpp package for Ubuntu WIP: [Packaging] Publish AdaptiveCpp package for Ubuntu Feb 16, 2024
@normallytangent normallytangent changed the title WIP: [Packaging] Publish AdaptiveCpp package for Ubuntu [Packaging] Publish AdaptiveCpp package for Ubuntu Feb 16, 2024
@normallytangent normallytangent force-pushed the package-on-push branch 2 times, most recently from 722a629 to 7541653 Compare February 28, 2024 16:38
@DhruvDh
Copy link

DhruvDh commented Mar 6, 2024

I tried adding a job for a rocm backend, but it runs out of storage space when installing the rocm ubuntu package.
In theory, it should work with a runner that has more storage.

Here it is, in case it helps -

  packaging-acpp-hip-backend:
    name: Packaging AdaptiveCpp, clang ${{matrix.clang}}, ${{matrix.os}}, ROCm ${{matrix.rocm}}
    runs-on: ${{matrix.os}}
    strategy:
      matrix:
        clang: ["17"]
        os: [ubuntu-22.04]
        rocm: [6.0.2]
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive
      - name: install rocm
        run: |
          wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
              gpg --dearmor | sudo tee /etc/apt/keyrings/rocm.gpg > /dev/null
          echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.0.2 jammy main" \
              | sudo tee --append /etc/apt/sources.list.d/rocm.list
          echo -e 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' \
              | sudo tee /etc/apt/preferences.d/rocm-pin-600
          sudo apt update
          sudo apt install -y rocm${{matrix.rocm}}
      - name: install LLVM
        run: |
          wget https://apt.llvm.org/llvm.sh
          chmod +x llvm.sh
          sudo ./llvm.sh ${{matrix.clang}}
          sudo apt install -y libclang-${{matrix.clang}}-dev clang-tools-${{matrix.clang}} libomp-${{matrix.clang}}-dev llvm-${{matrix.clang}}-dev liblld-${{matrix.clang}}-dev
      - name: install dependencies
        run: |
          sudo apt install -y cmake libboost-all-dev wget gpg curl
      - name: setup build environment
        run: |
          export CXXFLAGS="$CXXFLAGS"
          if [[ "${{matrix.clang}}" != "11" && "${{matrix.clang}}" -lt "16" ]]; then
            export OMP_CXX_FLAGS="$CXXFLAGS -fexperimental-new-pass-manager"
            export CC=clang-${{matrix.clang}}
            export CXX=clang++-${{matrix.clang}}
          fi
          echo "CC=${CC}" >> $GITHUB_ENV
          echo "CXX=${CXX}" >> $GITHUB_ENV
          echo "OMP_CXX_FLAGS=${OMP_CXX_FLAGS}" >> $GITHUB_ENV
      - name: install packaging helpers
        run: |
          sudo  apt install -y build-essential
          sudo  apt install -y devscripts
          sudo  apt install -y debhelper
      - name: package AdaptiveCpp
        run: |
          cd ..
          cp -R AdaptiveCpp acpp-23.10.0
          tar -cf acpp_23.10.0.orig.tar.gz acpp-23.10.0/
          cd acpp-23.10.0/
          sed -i "s/-DWITH_ROCM_BACKEND=.*/-DWITH_ROCM_BACKEND=ON \\\/" debian/rules
          debuild -us -uc
      - name: Install the package
        run: |
          cd ${GITHUB_WORKSPACE}/../
          sudo dpkg -i acpp_23.10.0-1_amd64.deb
      - name: Release Packages
        uses: actions/upload-artifact@v4
        with:
          name: rocm-backend
          path: /home/runner/work/AdaptiveCpp/acpp_23.10.0-1_amd64.deb
      - name: build rocm tests
        run: |
          mkdir ${GITHUB_WORKSPACE}/build/tests-rocm
          cd ${GITHUB_WORKSPACE}/build/tests-rocm
          cmake -DAdaptiveCpp_DIR=${GITHUB_WORKSPACE}/build/install/lib/cmake/AdaptiveCpp ${GITHUB_WORKSPACE}/tests -DCMAKE_EXE_LINKER_FLAGS="-L/usr/lib/llvm-${{matrix.clang}}/lib"
          make -j2

@illuhad
Copy link
Collaborator

illuhad commented Mar 6, 2024

@DhruvDh Thanks. I think the space problem can be solved by thinning out the default runner image a little, as we already do here:

- name: Free Disk Space (Ubuntu)

However, I don't think that we need a separate ROCm package. We can just have one that supports everything :) Then what it can offload to only depends on whether users have CUDA/ROCm/OpenCL etc installed. I think the lack of ROCm support in the current version of this PR is just due to it being WIP.
(and probably we shouldn't target ROCm 6.0, as it has some issues)

@illuhad
Copy link
Collaborator

illuhad commented May 2, 2024

Do I understand correctly that the packages are now ready for testing? If so, I call on everybody reading here to give them a try and see if they work :)

@normallytangent
Copy link
Collaborator Author

The packages at this stage in this branch are ready. Would be great to hear some feedback, especially if the configuration even works.
I am currently working on a way to introduce a testing-friendly naming convention for the binaries. Which is on a separate branch on my fork for reasons that I keep needing to make multiple commits with tiny changes to make the CI happy.

normallytangent and others added 26 commits May 2, 2024 14:53
Trim the debian directory and rely entirely on the debian packaging toolchain to create the required files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants