REF: Split libraries into separate outputs #74

carterbox · 2024-03-05T22:51:37Z

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

Hi maintainers!

It looks like this feedstock is still shipping static libraries. It has been our channel's policy since 2020 that static libraries are not shipped or are separated into their own output. For more information about our static library policy, please read CFEP-18.

In this PR, I have restructured the recipe into multiple outputs including separating the static libraries and dev files from necessary run_exports (the shared objects). The method I am using is to "build everything, then split". Everything is built in the top-level section and installed to a staging directory. Then a custom python install script is used to selectively install the artifacts based on the name of each output.

My motivation for this PR is to reduce the size of artifacts in my packages dependency trees starting from packages with the largest artifacts to the smallest.

Checklist for multi-output transition

Compare info/has_prefix for changes
Compare info/files for changes
Choose package names to prevent breaks of existing builds

…nda-forge-pinning 2024.03.05.22.33.10

conda-forge-webservices · 2024-03-05T22:51:44Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

The extra section contained an unexpected subsection name. feestock-name is not a valid subsection name.

For recipe:

It looks like the '_libcairo_api' output doesn't have any tests.

conda-forge-webservices · 2024-03-05T22:53:21Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the '_libcairo_api' output doesn't have any tests.

carterbox · 2024-03-05T22:55:43Z

recipe/meta.yaml

+outputs:
+
+  # Everything needed to build against libcairo
+  - name: cairo


Because the "everything minus static libs" package is named "cairo" existing downstream packages should not break. Newly built packages will only recieve exports of the numbered shared libraries. (I believe cairo-trace is a dev package that does not need to be exported).

conda-forge-webservices · 2024-04-02T23:45:44Z

Hi! This is the friendly automated conda-forge-linting service.

I was trying to look for recipes to lint for you, but it appears we have a merge conflict.
Please try to merge or rebase with the base branch to resolve this conflict.

Please ping the 'conda-forge/core' team (using the @ notation in a comment) if you believe this is a bug.

conda-forge-webservices · 2024-04-03T15:36:58Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

I do have some suggestions for making it better though...

For recipe:

It looks like the '_libcairo_api' output doesn't have any tests.

carterbox · 2024-04-03T15:37:20Z

@conda-forge-admin, please rerender

…nda-forge-pinning 2024.04.03.08.04.29

xhochy · 2024-04-16T11:44:39Z

@conda-forge-admin please restart ci

recipe/meta.yaml

xhochy · 2024-04-16T12:04:03Z

recipe/meta.yaml

+        - pkg-config --cflags cairo-gobject
+
+  # Only what's needed for run_exports downstream
+  - name: libcairo{{ so_name_major }}


If it isn't co-installable, why include the major version in the name?

I assume you are referring to the Windows build variant because the unix packages would be coinstallable with other ABI major version.

The reason is to keep consistency between Windows and Unix package names.

xhochy · 2024-04-16T12:06:08Z

recipe/install.py

@@ -0,0 +1,167 @@
+# This is free and unencumbered software released into the public domain.


You have used this probably before/somewhere else? I'm a bit confused we need such a lengthy script. In most other cases, I either installed the shared libraries explicitly or had a script which deletes some files from the PREFIX again and then uses conda-build's host deduplication.

You have used this probably before/somewhere else? I'm a bit confused we need such a lengthy script.

Yes. The length is because I want to use this same script in multiple feedstocks without changing it too much, so it needs to be generic (long).

In most other cases, I either installed the shared libraries explicitly or had a script which deletes some files from the PREFIX again and then uses conda-build's host deduplication.

Could you please explain better the approach you are describing here? I don't see how deleting files from the prefix is helpful for package splitting.

See the script in https://github.com/conda-forge/glib-feedstock/blob/2896c82bbbe09c898d3d1bd9f5668bfb22571b00/recipe/install.sh#L1 There we install everything as usual and remove afterwards the files that should not go into the respective output.

This is an interesting approach which we can call split-by-subtraction. It is the opposite of my approach which is split-by-addition. split-by-subtraction relies on a behavior of conda-build that the state of PREFIX is reset between generating each output (i.e. that the files which have been deleted are restored). The advantage is that hardcoded PREFIX paths in the installed artifacts do not need patching because they are installed directly to the PREFIX instead of to a staging area as in split-by-addition. I could modify my install script to use split-by-subtraction instead since the main purpose of my script is to enable easy splitting of the artifacts using glob.

However, this restoration behavior between generating each output is undocumented, so relying on it as the primary package splitting method is (in my opinion) not a good idea because it could break at any time without warning. In fact, the only documented methods are pattern matching via glob and moving files into the prefix (split-by-addition).

I have an open PR to improve the glob (pattern matching) functionality in conda-build which will make both split-by-subtraction and split-by-addition obsolete, so I can refactor this PR to use that feature if/when it is merged and released.

split-by-subtraction relies on a behavior of conda-build that the state of PREFIX is reset between generating each output (i.e. that the files which have been deleted are restored).

However, this restoration behavior between generating each output is undocumented

The behaviour is clearly documented. Each output has its own set of host and build dependencies and thus gets its own environments during the build. We also never delete any files during a build phase that we have no copied into the PREFIX some lines above in the build script.

My mistake, I misread the glib install scripts. I thought that meson install was called inside build.sh, but that is not the case.

I was thinking this:

build.sh : build everything; install everything install.sh : for each output, depending on the package name, remove things

but it's actually this:

build.sh : build everything; no installing install.sh : for each output, install everything, then depending on the package name, remove things

I'll have to think about how to adapt this approach to make it more generic.

Co-authored-by: Uwe L. Korn <xhochy@users.noreply.github.com>

carterbox added 3 commits March 5, 2024 16:47

REF: Separate artifacts into multiple outputs

b3e9e0b

MNT: Re-rendered with conda-build 24.1.2, conda-smithy 3.31.1, and co…

c8bbfec

…nda-forge-pinning 2024.03.05.22.33.10

DEV: Store build artifacts for debugging

0e96499

carterbox changed the title ~~Multi output~~ REF: Split libraries into separate outputs Mar 5, 2024

BUG: Fix spelling error for extra feedstock-name

eaf6de8

carterbox commented Mar 5, 2024

View reviewed changes

carterbox added 3 commits March 6, 2024 15:27

BUG: No cairo-trace for Windows

5169edb

Update bld.bat

1039029

BLD: pkg-config requires all the deps at build time

b222c88

Merge branch 'main' into multi-output

cd3aaa3

MNT: Re-rendered with conda-build 24.3.0, conda-smithy 3.34.1, and co…

52772af

…nda-forge-pinning 2024.04.03.08.04.29

carterbox marked this pull request as ready for review April 4, 2024 20:57

carterbox requested review from ccordoba12, jakirkham, ocefpaf, pkgw and tschoonj as code owners April 4, 2024 20:57

conda-forge-webservices bot closed this Apr 16, 2024

conda-forge-webservices bot reopened this Apr 16, 2024

xhochy reviewed Apr 16, 2024

View reviewed changes

Use correct syntax for feedstock-name

5c12060

Co-authored-by: Uwe L. Korn <xhochy@users.noreply.github.com>

carterbox marked this pull request as draft April 30, 2024 16:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REF: Split libraries into separate outputs #74

REF: Split libraries into separate outputs #74

carterbox commented Mar 5, 2024 •

edited

conda-forge-webservices bot commented Mar 5, 2024

conda-forge-webservices bot commented Mar 5, 2024

carterbox Mar 5, 2024

conda-forge-webservices bot commented Apr 2, 2024

conda-forge-webservices bot commented Apr 3, 2024

carterbox commented Apr 3, 2024

xhochy commented Apr 16, 2024

xhochy Apr 16, 2024

carterbox Apr 17, 2024 •

edited

xhochy Apr 16, 2024

carterbox Apr 17, 2024

xhochy Apr 29, 2024

carterbox Apr 29, 2024 •

edited

xhochy Apr 30, 2024

carterbox Apr 30, 2024

		@@ -0,0 +1,167 @@
		# This is free and unencumbered software released into the public domain.

REF: Split libraries into separate outputs #74

Are you sure you want to change the base?

REF: Split libraries into separate outputs #74

Conversation

carterbox commented Mar 5, 2024 • edited

Checklist for multi-output transition

conda-forge-webservices bot commented Mar 5, 2024

conda-forge-webservices bot commented Mar 5, 2024

carterbox Mar 5, 2024

Choose a reason for hiding this comment

conda-forge-webservices bot commented Apr 2, 2024

conda-forge-webservices bot commented Apr 3, 2024

carterbox commented Apr 3, 2024

xhochy commented Apr 16, 2024

xhochy Apr 16, 2024

Choose a reason for hiding this comment

carterbox Apr 17, 2024 • edited

Choose a reason for hiding this comment

xhochy Apr 16, 2024

Choose a reason for hiding this comment

carterbox Apr 17, 2024

Choose a reason for hiding this comment

xhochy Apr 29, 2024

Choose a reason for hiding this comment

carterbox Apr 29, 2024 • edited

Choose a reason for hiding this comment

xhochy Apr 30, 2024

Choose a reason for hiding this comment

carterbox Apr 30, 2024

Choose a reason for hiding this comment

carterbox commented Mar 5, 2024 •

edited

carterbox Apr 17, 2024 •

edited

carterbox Apr 29, 2024 •

edited