Does Flux-core build via Spack on Arm? I can't get it to work

[christopherwharrop-noaa]

I am building flux-core (and flux-sched) with the latest spack. I can build it no problem on amd64 (x86). But when I build on ARM I get:

14 10029.6 ==> Installing flux-core-0.53.0-5orr5yuidoebifb2sottreksoqarnxnp [261/266]
#14 10029.6 ==> No binary for flux-core-0.53.0-5orr5yuidoebifb2sottreksoqarnxnp found: installing from source
#14 10031.4 ==> Fetching https://mirror.spack.io/_source-cache/archive/2f/2f14d032a2d54f34e066c8a15c79917089e9f7f8558baa03dbfe63dbf56918b7.tar.gz
#14 10031.4 ==> No patches needed for flux-core
#14 10031.5 ==> flux-core: Executing phase: 'autoreconf'
#14 10031.6 ==> flux-core: Executing phase: 'configure'
#14 10040.1 ==> Error: ProcessError: Command exited with status 1:
#14 10040.1     '/tmp/root/spack-stage/spack-stage-flux-core-0.53.0-5orr5yuidoebifb2sottreksoqarnxnp/spack-src/configure' '--prefix=/opt/software/linux-ubuntu20.04-neoverse_n1/gcc-9.4.0/flux-core-0.53.0-5orr5yuidoebifb2sottreksoqarnxnp' '--enable-pylint=no' '--disable-docs'
#14 10040.6
#14 10040.6 1 error found in build log:
#14 10040.6      222    checking for systemd/sd-bus.h... no
#14 10040.6      223    checking for HWLOC... yes
#14 10040.6      224    checking for LZ4... yes
#14 10040.6      225    checking for SQLITE... yes
#14 10040.6      226    checking for LIBUUID... yes
#14 10040.6      227    checking for CURSES... yes
#14 10040.6   >> 228    checking for LIBARCHIVE... configure: error: Package requirements (
#14 10040.6             libarchive) were not met:
#14 10040.6      229
#14 10040.6      230    Package 'iconv', required by 'libarchive', not found
#14 10040.6      231
#14 10040.6      232    Consider adjusting the PKG_CONFIG_PATH environment variable if you
#14 10040.6      233    installed software in a non-standard prefix.
#14 10040.6      234
#14 10040.6
#14 10040.6 See build log for details:
#14 10040.6   /tmp/root/spack-stage/spack-stage-flux-core-0.53.0-5orr5yuidoebifb2sottreksoqarnxnp/spack-build-out.txt
#14 10040.6

I know for a fact that libarchive is being built with iconv, and that variant is shown in the Spack concretization. I've done this build on a clean EC2 ARM instance that had no previous cache (and no build cache was used). I do not know how to solve this issue so would appreciate ideas.

The concretization looks like:

14 43.23 ==> Concretized flux-core@0.53.0
#14 43.23  -   5orr5yu  flux-core@0.53.0%gcc@9.4.0~cuda~docs~security build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   h6tfiw5      ^czmq@4.1.1%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   47wq7cq          ^autoconf@2.69%gcc@9.4.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   laz75ri              ^m4@1.4.19%gcc@9.4.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   4aj5q3z                  ^libsigsegv@2.14%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   yqu4gw3              ^perl@5.38.0%gcc@9.4.0+cpanm+opcode+open+shared+threads build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   xjy7yhu                  ^berkeley-db@18.1.40%gcc@9.4.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   qmw7zk5          ^automake@1.16.5%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   n4iib62          ^docbook-xml@4.5%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   2r5vk5g          ^docbook-xsl@1.79.2%gcc@9.4.0 build_system=generic patches=a92c397 arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   6nlhndv          ^libtool@2.4.7%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   bldt4bf          ^util-linux-uuid@2.38.1%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   qi2eo75      ^gmake@4.4.1%gcc@9.4.0~guile build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   3y562rm      ^gnuconfig@2022-09-17%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   jydyg6v      ^hwloc@2.9.1%gcc@9.4.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   dnvs4vs          ^libpciaccess@0.17%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   3vsguew              ^util-macros@1.19.3%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   z6jawgu          ^libxml2@2.10.3%gcc@9.4.0+pic~python+shared build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   htaxrlz      ^jansson@2.14%gcc@9.4.0~ipo+shared build_system=cmake build_type=Release generator=make arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   ed4r3pv          ^cmake@3.23.1%gcc@9.4.0~doc+ncurses~owncurl+ownlibs build_system=generic build_type=Release arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   4eizceq      ^libarchive@3.7.1%gcc@9.4.0+iconv build_system=autotools compression=bz2lib,lz4,lzma,lzo2,zlib,zstd crypto=openssl libs=shared,static programs=none xar=libxml2 arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   hfh7ogq          ^bzip2@1.0.8%gcc@9.4.0~debug~pic+shared build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23 [e]  m36cbty              ^diffutils@3.7%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   5ux3e7r          ^libiconv@1.17%gcc@9.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   gh34f73          ^lzo@2.10%gcc@9.4.0 build_system=autotools libs=shared,static arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   xeqyu5x          ^openssl@3.1.3%gcc@9.4.0~docs+shared build_system=generic certs=mozilla arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   zdz4lpf              ^ca-certificates-mozilla@2023-05-30%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   cj2acyt          ^xz@5.4.1%gcc@9.4.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   wworcav          ^zlib@1.2.13%gcc@9.4.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   evk6vdk          ^zstd@1.5.2%gcc@9.4.0+pic+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   3aoafby      ^libzmq@4.3.5%gcc@9.4.0~docs~drafts+libbsd+libsodium~libunwind build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   fwjucnv          ^libbsd@0.11.7%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   wnqj6zw              ^libmd@1.0.4%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   6yewba5          ^libsodium@1.0.18%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   jl3ctno      ^lua@5.3.6%gcc@9.4.0~pcfile+shared build_system=makefile fetcher=curl arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   3ts6nvj          ^curl@8.4.0%gcc@9.4.0~gssapi~ldap~libidn2~librtmp~libssh~libssh2+nghttp2 build_system=autotools libs=shared,static tls=openssl arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   ggjtqap              ^nghttp2@1.57.0%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   xhqd7o7          ^readline@8.2%gcc@9.4.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   gguiloc          ^unzip@6.0%gcc@9.4.0 build_system=makefile arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   hmsgnlf      ^lua-luaposix@36.1%gcc@9.4.0 build_system=lua arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   es2k2tu      ^lz4@1.9.4%gcc@9.4.0+pic build_system=makefile libs=shared,static arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   a2pfv4i      ^ncurses@6.4%gcc@9.4.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   f7iyvuw      ^pkg-config@0.29.2%gcc@9.4.0+internal_glib build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   juk2hq7      ^py-cffi@1.15.1%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   b5jgpvo          ^libffi@3.4.4%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   3iptdck          ^py-pip@23.1.2%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   mwfs4ei          ^py-pycparser@2.21%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   ifzgfam          ^py-setuptools@63.4.3%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   2t7q333          ^py-wheel@0.37.1%gcc@9.4.0 build_system=generic arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   lovdxyt      ^py-jsonschema@4.17.3%gcc@9.4.0~format-nongpl build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   b6ktyju          ^py-attrs@21.4.0%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   bz66553          ^py-hatch-fancy-pypi-readme@23.1.0%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   qjbqz7d              ^py-tomli@2.0.1%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   mlchklp                  ^py-flit-core@3.8.0%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   zkdo32q          ^py-hatch-vcs@0.3.0%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   c6no345              ^py-setuptools-scm@7.1.0%gcc@9.4.0+toml build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23 [e]  mucrphy                  ^git@2.25.1%gcc@9.4.0+man+nls+perl+subtree~svn~tcltk build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   4gsn5qt                  ^py-typing-extensions@4.8.0%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   x3qgoki          ^py-hatchling@1.18.0%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   wlu4r2i              ^py-editables@0.3%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   jijxxih              ^py-packaging@23.1%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   36owrbn              ^py-pathspec@0.11.1%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   b226sv4              ^py-pluggy@1.0.0%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   i7myaui              ^py-trove-classifiers@2023.8.7%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   6cog66w                  ^py-calver@2022.6.26%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   3bgdt4d          ^py-pyrsistent@0.19.3%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   26ivktn      ^py-ply@3.11%gcc@9.4.0 build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   ytplibu      ^py-pyyaml@6.0%gcc@9.4.0+libyaml build_system=python_pip arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   vnea4f5          ^libyaml@0.2.5%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   dg4u4b6          ^py-cython@0.29.36%gcc@9.4.0 build_system=python_pip patches=c4369ad arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   gufo2fk      ^python@3.10.13%gcc@9.4.0+bz2+crypt+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tkinter+uuid+zlib build_system=generic patches=0d98e93,7d40923,ebdca64,f2fd060 arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   weoje7u          ^expat@2.5.0%gcc@9.4.0+libbsd build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   jxo3hvq          ^gdbm@1.23%gcc@9.4.0 build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   mex2vf2          ^gettext@0.21.1%gcc@9.4.0+bzip2+curses+git~libunistring+libxml2+pic+shared+tar+xz build_system=autotools arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   azrlbug              ^tar@1.34%gcc@9.4.0 build_system=autotools zip=pigz arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   5m5jlec                  ^pigz@2.7%gcc@9.4.0 build_system=makefile arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   zvk5ifk          ^libxcrypt@4.4.35%gcc@9.4.0~obsolete_api build_system=autotools patches=4885da3 arch=linux-ubuntu20.04-neoverse_n1
#14 43.23  -   x3e3ptm      ^sqlite@3.43.2%gcc@9.4.0+column_metadata+dynamic_extensions+fts~functions+rtree build_system=autotools arch=linux-ubuntu20.04-neoverse_n1

[grondo]

@vsoch may be able to help here, and also I wonder if you'd get more responses opening a spack issue?

[vsoch]

I have hit this error countless times with flux and spack, opened issues several times (latest here spack/spack#38037) and what happens every time (in our automated nightly builds) is that it mysteriously resolves. I wish I could offer insight about why it happens (and why it gets fixed and comes back), but I have no idea.

[christopherwharrop-noaa]

This is becoming a serious problem for me. I am doing my Spack install inside a container build. I had the same luck as you before where I had success by emptying the build cache mirror. This time, I had removed the binary cache settings from the build so that Spack had no other mirrors available and it still gave the error. I removed all packages from my environment yaml file, leaving only flux-core and flux-sched and tried again. It worked. I guess the only idea I have now is to add packages back in, half at a time, and see if I can find a minimal reproducer. If it was only an occasional thing where I could fix it by building without the binary cache and then adding the mirror afterwards and pushing manually, I could work around it. But the current situation is a show-stopping bug. I don't know what the Spack folks will need to help them investigate but I suppose the main thing is for me to try to take my very complex setup and reduce it. Problem is, as you say, the issue often disappears for no explainable reason.

[vsoch]

I'm really sorry @christopherwharrop-noaa - all I can say is that I hear your frustration, and commonly share it with spack. I hope the developers can help resolve it soon.

[christopherwharrop-noaa]

Thank you for sharing your frustration. I've added some information that may be relevant over on your Spack issue. It sounds like there is an ongoing issue with libarchive + iconv in general and that the problem stems from iconv's lack of a *.pc file. I'll post further information (if I have any) in your open Spack issue. Thank you for pointing me to it again. Too much has happened since the last time I was there and I had forgotten about it. So, sorry about that.