Update TileDB-VCF to 0.29.2

[awenocur]

No description provided.

[shortcut-integration]

This pull request has been linked to Shortcut Story #43338: Release VCF 0.29.1 with Core 2.20.1 and INFO END handling.

[jdblischak]

• reset build number to 0
• bump tiledb pins in host/run to 2.21 (in fact you could simply delete tiledb from run. It's redundant because the tiledb recipe uses run exports, ie it automatically creates a runtime requirement if it is a host requirement)

[jdblischak]

Recipe looks good. Unclear why all the jobs are failing due to an error while building libtiledbvcf

[awenocur]

Recipe looks good. Unclear why all the jobs are failing due to an error while building libtiledbvcf

I'm seeing spdlog attempting to use libfmt for a C++20 build, when it should be using std::fmt.

[jdblischak]

I figured out the problem.

The last successful build was from ~2 weeks ago from commit 2ec4de4 (when I added pyarrow-hotfix to the requirements)

The linux-64 job installed fmt as an external project

-- Could NOT find spdlog (missing: spdlog_DIR)
-- Adding spdlog as an external project
-- Found TileDB: /home/conda/feedstock_root/build_artifacts/tiledb-vcf_1710174242588/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libtiledb.so.2.20

Because it installed tiledb 2.20.1-h7c1ff67_3, which doesn't pin fmt or spdlog

wget https://anaconda.org/conda-forge/tiledb/2.20.1/download/linux-64/tiledb-2.20.1-h7c1ff67_3.conda
conda inspect hash-inputs tiledb-2.20.1-h7c1ff67_3.conda
## {'tiledb-2.20.1-h7c1ff67_3.conda': {'recipe': {'aws_crt_cpp': '0.26.2',
##                                                'aws_sdk_cpp': '1.11.267',
##                                                'bzip2': '1',
##                                                'channel_targets': 'conda-forge '
##                                                                   'main',
##                                                'curl': '8',
##                                                'cxx_compiler': 'gxx',
##                                                'cxx_compiler_version': '12',
##                                                'libabseil': '20240116',
##                                                'libcurl': '8',
##                                                'libgoogle_cloud_devel': '2.22',
##                                                'libgoogle_cloud_storage_devel': '2.22',
##                                                'libxml2': '2',
##                                                'lz4_c': '1.9.3',
##                                                'openssl': '3',
##                                                'target_platform': 'linux-64',
##                                                'zlib': '1.2',
##                                                'zstd': '1.5'}}}

In contrast, the latest linux-64 build from this PR installed tiledb 2.20.1-h99d63bd_4, which pins fmt 10 and spdlog 1.12

wget https://anaconda.org/conda-forge/tiledb/2.20.1/download/linux-64/tiledb-2.20.1-h99d63bd_4.conda
conda inspect hash-inputs tiledb-2.20.1-h99d63bd_4.conda
## {'tiledb-2.20.1-h99d63bd_4.conda': {'recipe': {'aws_crt_cpp': '0.26.3',
##                                                'aws_sdk_cpp': '1.11.267',
##                                                'bzip2': '1',
##                                                'channel_targets': 'conda-forge '
##                                                                   'main',
##                                                'curl': '8',
##                                                'cxx_compiler': 'gxx',
##                                                'cxx_compiler_version': '12',
##                                                'fmt': '10',
##                                                'libabseil': '20240116',
##                                                'libcurl': '8',
##                                                'libgoogle_cloud_devel': '2.22',
##                                                'libgoogle_cloud_storage_devel': '2.22',
##                                                'libxml2': '2',
##                                                'lz4_c': '1.9.3',
##                                                'openssl': '3',
##                                                'spdlog': '1.12',
##                                                'target_platform': 'linux-64',
##                                                'zlib': '1.2',
##                                                'zstd': '1.5'}}}

Which resulted in fmt 10.2.1-h00ab1b0_0 and spdlog 1.12.0-hd2e6256_2 getting installed from conda-forge and causing the failure.

So the change was the PR conda-forge/tiledb-feedstock#254, also from ~2 weeks ago, to properly depend on fmt and spdlog from conda-forge

Note that this same change also recently caused problems for tiledbsoma-feedstock (TileDB-Inc/tiledbsoma-feedstock#101 (comment), TileDB-Inc/tiledbsoma-feedstock#106)

There are various ways around this. We could pin the recipe to the version of tiledb 2.20 prior to its installing fmt/spdlog from conda-forge (this would be tedious to maintain and wouldn't work long-term, but it would allow us to get 0.29.1 shipped). Another option would be to do something similar to TileDB-Inc/tiledbsoma-feedstock#106 and depend on the dependency-free "for-cloud" tiledb binary.

However, if it's possible to update TileDB-VCF to avoid these fmt/spdlog issues entirely (ie TileDB-Inc/TileDB-VCF#682), that would be ideal.

[ihnorton]

Another option would be to do something similar to TileDB-Inc/tiledbsoma-feedstock#106 and depend on the dependency-free "for-cloud" tiledb binary.

+1, IMO let's keep it simple to ship this ASAP without a dependency chain/overhead.

[jdblischak]

Summary from discussion with @ihnorton:

• Plan A is to migrate TileDB-VCF to C++20 (Move to c++20 TileDB-VCF#682) and release 0.29.2, which should hopefully fix the build error related to the version of fmt
• Plan B is to build against the "for-cloud" tiledb binary, which doesn't depend on conda-forge::fmt

Hopefully one of the two above plans will work. There is the possibility that libtiledbvcf will have trouble at runtime if it is built against fmt 10 and spdlog 1.12 in this feedstock but then deployed to TileDB Cloud which has fmt 9 and spdlog 1.11 installed. If that happens, then:

• Plan C is to build separate variants aimed at conda-forge versus TileDB Cloud like I did for SOMA in Build two variants: one for cloud and one for conda-forge tiledbsoma-feedstock#106

[jdblischak]

From the errors, it looks like it is still having trouble properly linking against fmt:

[100%] Linking CXX executable tiledbvcf
/home/conda/feedstock_root/build_artifacts/tiledb-vcf_1711152722224/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/tiledbvcf-bin.dir/cli/tiledbvcf.cc.o: in function `spdlog::details::fmt_helper::pad2(int, fmt::v10::basic_memory_buffer<char, 250ul, std::allocator<char> >&)':
tiledbvcf.cc:(.text._ZN6spdlog7details10fmt_helper4pad2EiRN3fmt3v1019basic_memory_bufferIcLm250ESaIcEEE[_ZN6spdlog7details10fmt_helper4pad2EiRN3fmt3v1019basic_memory_bufferIcLm250ESaIcEEE]+0xf7): undefined reference to `void fmt::v10::detail::vformat_to<char>(fmt::v10::detail::buffer<char>&, fmt::v10::basic_string_view<char>, fmt::v10::detail::vformat_args<char>::type, fmt::v10::detail::locale_ref)'
/home/conda/feedstock_root/build_artifacts/tiledb-vcf_1711152722224/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: CMakeFiles/tiledbvcf-bin.dir/cli/tiledbvcf.cc.o: in function `spdlog::logger::sink_it_(spdlog::details::log_msg const&)':
tiledbvcf.cc:(.text._ZN6spdlog6logger8sink_it_ERKNS_7details7log_msgE[_ZN6spdlog6logger8sink_it_ERKNS_7details7log_msgE]+0x180): undefined reference to `fmt::v10::vformat[abi:cxx11](fmt::v10::basic_string_view<char>, fmt::v10::basic_format_args<fmt::v10::basic_format_context<fmt::v10::appender, char> >)'

[awenocur]

From the errors, it looks like it is still having trouble properly linking against fmt:

One possibility is that a template is being substituted with a std::string_view argument, but not instantiated, resulting in a symbol being expected but not generated. I'm trying a forced update of spdlog; if that doesn't have the desired effect, I can look for reasons why the templates may be used in a manner the authors hadn't anticipated.

[gspowley]

The latest errors show improvement after (TileDB-Inc/TileDB-VCF#684) and (TileDB-Inc/TileDB-VCF#686).

One linker error remains for Linux:

/home/conda/feedstock_root/build_artifacts/tiledb-vcf_1711384793823/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/12.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: tiledbvcf: hidden symbol `_ZN6spdlog6logger12err_handler_ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE' in CMakeFiles/tiledbvcf-bin.dir/cli/tiledbvcf.cc.o is referenced by DSO

@ihnorton, @jdblischak any ideas on how to resolve this error? Should we fall back to Plan B?

[jdblischak]

any ideas on how to resolve this error? Should we fall back to Plan B?

Something weird is going on. spdlog and fmt are installed in the conda env, and ideally libtiledbvcf would link against these (since these are also what libtiledb linked against). That is what I thought 0.29.1 was doing. However, now with 0.29.2, it is back to installing spdlog as an external project.

-- Could NOT find spdlog (missing: spdlog_DIR)
-- Adding spdlog as an external project
-- Found TileDB: /home/conda/feedstock_root/build_artifacts/tiledb-vcf_1711384793823/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_/lib/libtiledb.so.2.20

Can either the source file FindSpdlog.cmake or the build script be updated so that libtiledbvcf builds against the conda-installed spdlog and fmt?

[gspowley]

There's a new cmake option to control how spdlog is handled, similar to htslib.

This change to the build script should use the conda installed spdlog, if that's what we need:

cmake \
  -DCMAKE_INSTALL_PREFIX:PATH="${PREFIX}" \
  -DOVERRIDE_INSTALL_PREFIX=OFF \
  -DCMAKE_BUILD_TYPE=Release \
  -DFORCE_EXTERNAL_HTSLIB=OFF \
  -DFORCE_EXTERNAL_SPDLOG=OFF \
  ../libtiledbvcf

[jdblischak]

We have a different problem now. CMake can't find spdlog in the conda environment, which is causing the build to fail on every platform.

https://dev.azure.com/TileDB-Inc/CI/_build/results?buildId=38681&view=results

CMake Error at cmake/Modules/FindSpdlog.cmake:93 (message):
  Unable to find spdlog
Call Stack (most recent call first):
  src/CMakeLists.txt:9 (find_package)


-- Could NOT find spdlog (missing: spdlog_DIR)
-- Configuring incomplete, errors occurred!
make[2]: *** [CMakeFiles/libtiledbvcf.dir/build.make:93: libtiledbvcf-prefix/src/libtiledbvcf-stamp/libtiledbvcf-configure] Error 1
make[1]: *** [CMakeFiles/Makefile2:141: CMakeFiles/libtiledbvcf.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

0.29.1 was able to find spdlog in the conda environment, so I assume some recent change to FindSpdlog.cmake caused the change in behavior

TileDB-Inc/TileDB-VCF@0.29.1...release-0.29

[awenocur]

We need to figure out how to remove the arg -Wl,--allow-shlib-undefined from the final linker command for the tiledbvcf executable.

[awenocur]

Now I'm running into a wheel/packaging problem:

2024-03-28T02:28:58.6026304Z LookupError: setuptools-scm was unable to detect version for '/home/conda/feedstock_root/build_artifacts/tiledb-vcf_1711592366035/work'.
2024-03-28T02:28:58.6026660Z 
2024-03-28T02:28:58.6027180Z Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work.

How did this ever work, given that we always use tarballs? I'll try creating an archive with a numeric filename to see whether that fixes it.

[awenocur]

@ihnorton can you bypass the Windows CI after the remaining checks pass?