Skip to content
This repository has been archived by the owner on Aug 9, 2022. It is now read-only.

Short-circuit builders for C Extension wheels not using the default imageset #11

Open
natefoo opened this issue Aug 28, 2018 · 1 comment
Labels
enhancement New feature or request

Comments

@natefoo
Copy link
Member

natefoo commented Aug 28, 2018

For wheels that need an imageset that is a subset (e.g. #5), we can't head off builds in the before_script step as we do with starforge wheel_type, since wheel_type only inspects a package to determine whether it's universal, purepy, or c-extension. We either need a separate command that allows Travis to determine whether a given image is in a wheel's configured imageset, or one super-command that does both wheel_type and the image membership test. Currently, the membership test is done in starforge wheel itself, which can't be run until the slow/expensive image setup steps (building a Docker image from manylinux1 (Linux) or installing Python (macOS)) are complete. Avoiding those costly steps when a particular builder wouldn't even be building anything was the whole reason starforge wheel_type was implemented in the first place.

This complexity is due in part to a Starforge design decision - prior to using CI, Starforge was intended to be run on a single host, so it was fully responsible for controlling the virtualization. With Travis there's a mixed state where some virtualization is still managed by Starforge (such as on Linux, where both 32- and 64-bit wheels are built on a single Travis builder, and in the case of Python 2.7, a total of 4 wheels due to cp27m and cp27mu ABIs), but some is not (such as on macOS). In this hybrid environment, Starforge needs to know what images the current builder "supports" so that it knows what to/not to build. But Travis needs to know what images a wheel should be built on to supply these to Starforge...

One possible solution to this would be to combine all C Extension builds in to two builders: one for Linux, and one for macOS. This is discussed (with some drawbacks) in #12.

@natefoo
Copy link
Member Author

natefoo commented Aug 28, 2018

Another solution would be to have Starforge inspect certain environment variables (configured in an image config in starforge.yml) to determine whether any image out of the configured imageset applies, for example:

.travis.yml:

env:
  global:
    - WHEEL_BUILDER_TYPE="c-extension"
  matrix:
    - PY="2.7"
    - PY="3.4"
    - PY="3.5"
    - PY="3.6"
    - PY="3.7"

matrix:
  include:
    - name: "Universal Pure Python wheels"
      os: linux
      language: python
      python: "3.6"
      env:
        - WHEEL_BUILDER_TYPE="universal"
        - PY="3.6"
      install: &purepy_install
        - pip install $STARFORGE

    - name: "Pure Python 2 wheels"
      os: linux
      language: python
      python: "2.7"
      env:
        - WHEEL_BUILDER_TYPE="purepy"
        - PY="2.7"
      install: *purepy_install

    - name: "Pure Python 3 wheels"
      os: linux
      language: python
      python: "3.6"
      env:
        - WHEEL_BUILDER_TYPE="purepy"
        - PY="3.6"
      install: *purepy_install

starforge.yml:

images:
  ci/linux-3.6:universal:
    build_when:
      env:
        WHEEL_BUILDER_TYPE: universal
  # ...    
  ci/linux-2.7:i686:
    build_when:
      env:
        WHEEL_BUILDER_TYPE: c-extension
        TRAVIS_OS_NAME: linux
        PY: "2.7"
  # ...
  ci/osx-3.7:
    build_when:
      env:
        WHEEL_BUILDER_TYPE: c-extension
        TRAVIS_OS_NAME: osx
        PY: "2.7"

And then starforge wheel_type is replaced with starforge wheel --check or something like that, which exits without building, returning 0 if any images in a wheel's imageset match the current environment, and 1 if no images match. This also takes care of wheel_type since, for wheels where a type is not explicitly set in the wheel config, starforge wheel determine which of the default imagesets (universal-wheels, purepy-wheels, or default-wheels) a wheel should build on.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant