Build non-conda-forge dependencies with vcpkg.

[teo-tsirpanis]

Checklist

• Used a personal fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

This PR enables TileDB's support for acquiring dependencies with vcpkg. To prevent vcpkg for building even dependencies already provided by conda-forge, I added empty overlay ports for these dependencies as recommended by the vcpkg team. These ports are in a recipe/tiledb-patches folder that gets copied to the downloaded sources.

While there is a vcpkg Conda package we cannot use it for two reasons. First, that package does not provide the vcpkg executable in a way CMake can find it. Second the vcpkg folder must be a cloned git repository to allow getting the requested port version from past commits. For these reasons we rely on TileDB's automatic mechanism that clones vcpkg.

I also added a document that explains how these port overlays get added.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[teo-tsirpanis]

libxml2 is not used on Windows.

[teo-tsirpanis]

CI on linux-arm64 fails when vcpkg tries to find a compiler, with:

CMake Error at CMakeLists.txt:11 (enable_language):
  The CMAKE_C_COMPILER:

    aarch64-linux-gnu-gcc

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
  the compiler, or to the compiler name if it is in the PATH.

[jdblischak]

That's a strange error. CC should be set to $RECIPE_DIR/cc_wrap.sh

tiledb-feedstock/recipe/build.sh

Line 8 in c44154a

export CC=$RECIPE_DIR/cc_wrap.sh

Is there any reason why this export wouldn't work on linux-aarch64?

[teo-tsirpanis]

Vcpkg overrides the compiler paths without considering the CC/CXX environment variables. I forced it to consider them, let's see…

[teo-tsirpanis]

CI is green! 🎉

[jdblischak]

Now that #244 was merged, could you please rebase your changes onto main and then rerender?

[teo-tsirpanis]

@conda-forge-admin, please rerender

[jdblischak]

From a conda-build perspective, this looks good to me

[jdblischak]

This documentation is great. Thanks!

[jdblischak]

And to make sure I understand, currently the only dependencies being installed with vcpkg are capnproto and libwebp, right? And all the other dependencies are installed from conda-forge. Are there known issues with migrating to the conda-forge versions of capnproto and libwebp?

The following packages will be built and installed:
    abseil[core,cxx17]:x64-linux -> system -- $SRC_DIR/system-ports/abseil
    aws-sdk-cpp[core,identity-management,s3,sts]:x64-linux -> system -- $SRC_DIR/system-ports/aws-sdk-cpp
    azure-storage-blobs-cpp:x64-linux -> system -- $SRC_DIR/system-ports/azure-storage-blobs-cpp
    bzip2:x64-linux -> system -- $SRC_DIR/system-ports/bzip2
    capnproto:x64-linux -> 1.0.1 -- $SRC_DIR/build/_deps/vcpkg-src/buildtrees/versioning_/versions/capnproto/4a615543c6406b84fc52a931335d7fdb70037627
    curl[core,zstd]:x64-linux -> system -- $SRC_DIR/system-ports/curl
  * fmt:x64-linux -> 9.1.0#1 -- $SRC_DIR/ports/fmt
    google-cloud-cpp[core,storage]:x64-linux -> system -- $SRC_DIR/system-ports/google-cloud-cpp
    libmagic:x64-linux -> 5.40#1 -- $SRC_DIR/ports/libmagic
    libwebp[core,libwebpmux,nearlossless,simd]:x64-linux -> 1.3.2 -- $SRC_DIR/build/_deps/vcpkg-src/buildtrees/versioning_/versions/libwebp/0b981028589375097039d5e39e7d87659cdfa824
    libxml2[core,lzma,zlib]:x64-linux -> system -- $SRC_DIR/system-ports/libxml2
    lz4:x64-linux -> system -- $SRC_DIR/system-ports/lz4
    openssl:x64-linux -> system -- $SRC_DIR/system-ports/openssl
  * pcre2[core,jit,platform-default-features]:x64-linux -> 10.42#1 -- $SRC_DIR/ports/pcre2
    spdlog:x64-linux -> 1.11.0 -- $SRC_DIR/ports/spdlog
  * vcpkg-cmake:x64-linux -> 2022-10-30 -- $SRC_DIR/ports/vcpkg-cmake
  * vcpkg-cmake-config:x64-linux -> 2022-02-06#1 -- $SRC_DIR/ports/vcpkg-cmake-config
    zlib:x64-linux -> system -- $SRC_DIR/system-ports/zlib
    zstd:x64-linux -> system -- $SRC_DIR/system-ports/zstd

[teo-tsirpanis]

There is also libmagic, whose feedstock is unmaintained and does not support Windows. I took a look at capnproto and libwebp. The former does not support Linux on arm64 and ppcle64 (conda-forge/capnproto-feedstock#40), and the latter does not provide CMake targets on non-Windows (I can fix it). Either way I would prefer to address these dependencies in a subsequent PR.

[jdblischak]

Either way I would prefer to address these dependencies in a subsequent PR.

Absolutely

There is also libmagic

Oh, I didn't realize the $SRC_DIR/ports/ directory also meant installation by vcpkg. If that's the case, that means fmt, pcre2, and spdlog are also being installed by vcpkg, right? I know we are already depending on fmt and spdlog from conda-forge in our tiledbsoma recipe:

https://github.com/TileDB-Inc/tiledbsoma-feedstock/blob/6e28425785020e98e088390bec528a13eeea2dc6/recipe/meta.yaml#L46

And I know pcre2 is a dependency of r-base, so it should be available for all platforms:

https://github.com/conda-forge/r-base-feedstock/blob/09919aa4e7b87e850e7405a6b9d3b1074c95527c/recipe/meta.yaml#L135

[teo-tsirpanis]

Oh I forgot about spdlog, and thanks for letting me know that it exists on conda-forge. fmt is a transitive dependency of spdlog, and pcre2 is a transitive dependency of libmagic, which will go away with TileDB-Inc/TileDB#4673.

the $SRC_DIR/ports/ directory also meant installation by vcpkg

A better indicator to see if a dependency is provided by Conda is if the version is system.

[teo-tsirpanis]

We can use the upstream libwebp after conda-forge/libwebp-base-feedstock#20 is merged.

[dudoslav]

LGTM, if the pipelines pass.

[isuruf]

I don't think it's a good idea to use vcpkg. Let's use conda-forge packages for all dependencies.

[jdblischak]

I don't think it's a good idea to use vcpkg. Let's use conda-forge packages for all dependencies.

@isuruf our goal is to use conda-forge packages as much as possible. And when a conda-forge package is not suitable for our needs, we plan to make a good faith effort to update it, eg conda-forge/libwebp-base-feedstock#20. Using vcpkg is the last-resort, fallback option.

Currently the following dependencies are installed via vcpkg: capnproto, fmt, libmagic, libwebp, pcre2, spdlog

I propose we merge this PR as is and then update the dependencies to use conda-forge according to these criteria:

• spdlog & fmt: ASAP
  • Use upstream spdlog. #254
• pcre2: drop after libmagic is updated (Update libmagic to version 5.45 TileDB-Inc/TileDB#4673)
• libwebp: after it provides CMake targets (Build with CMake on all platforms. libwebp-base-feedstock#20)
  • Use upstream libwebp. #273
• capnproto: after it is migrated to aarch64/ppc64le (Arch Migrator capnproto-feedstock#40)
• libmagic: after it is built for Windows (seems unlikely to happen anytime soon given that the feedstock is archived)

[teo-tsirpanis]

@conda-forge-admin, please rerender

[github-actions]

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

This message was generated by GitHub actions workflow run https://github.com/conda-forge/tiledb-feedstock/actions/runs/8204058822.

[teo-tsirpanis]

CI failures are transient. I don't know how to rerun it.

[xylar]

@teo-tsirpanis, the better way is to get a maintainer to hit the rerun button in Azure (which I have just done). The less preferred way is to do (at)conda-forge-admin, please restart ci. The latter reruns everything not just the failed runs.

[teo-tsirpanis]

Thanks!

[jdblischak]

Merging. We'll proceed with the plan in #242 (comment) to continue migrating as many dependencies to use conda-forge as they become available

[jdblischak]

Note to self: this PR didn't bump the build number, so no new binaries will be uploaded for this change (which is fine, the purpose of this PR was to improve the build process). The vcpkg-enabled builds will be uploaded with the next version or build number bump